Cloudflare Crawl API '/crawl' Explained with Examples

Example with all optional parameters

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://www.exampledocs.com/docs/",
    "limit": 50,
    "depth": 2,
    "formats": ["markdown"],
    "render": false,
    "maxAge": 7200,
    "modifiedSince": 1704067200,
    "source": "all",
    "options": {
      "includeExternalLinks": true,
      "includeSubdomains": true,
      "includePatterns": [
        "**/api/v1/*"
      ],
      "excludePatterns": [
        "*/learning-paths/*"
      ]
    }
}'

Real‑World Use Cases

1. Documentation Site Crawl With Filters

When you need to crawl only the documentation section of a website and deliberately skip unnecessary sections like changelog or archive you can use includePatterns and excludePatterns in your crawl request. These pattern filters let you precisely control which parts of a site are indexed, making the crawl more efficient and targeted.

This is particularly useful when building:

• Technical knowledge bases
• AI training datasets (e.g., for RAG systems)
• Documentation monitoring tools

The example below shows how to start a crawl that scans only the site's docs and omits irrelevant sections.

Example: Crawl Only Documentation Pages

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com/docs",
    "limit": 200,
    "depth": 5,
    "formats": ["markdown"],
    "options": {
      "includePatterns": [
        "https://example.com/docs/**"
      ],
      "excludePatterns": [
        "https://example.com/docs/changelog/**",
        "https://example.com/docs/archive/**"
      ]
    }
  }'

What This Script Does

• Starting URL: Begins crawling at the documentation home (https://example.com/docs).
• limit: Sets a maximum of 200 pages to crawl so you don't exceed quota or get unnecessary content.
• depth: Limits how many levels deep the crawler will follow links (here up to 5).
• formats: Requests results in Markdown, which is ideal for docs and AI training.
• includePatterns: Ensures only URLs under the /docs/ path get crawled.
• excludePatterns: Prevents crawling of specific sub‑sections that you don't need (like “changelog” and “archive”).

Why Use Pattern Rules

Pattern matching helps you avoid crawling irrelevant or sensitive parts of a site. Even if a page is linked from the docs, exclude patterns ensure it's skipped if it matches one of the defined rules. This keeps your crawl results focused and reduces unnecessary processing.

2. Product Catalog Extraction with AI

When building e‑commerce tools, price trackers, catalogs or AI‑driven product analytics systems, developers often need structured product data (such as product names, prices, descriptions, availability and currency). Cloudflare's Crawl API makes this easier by combining crawling, rendering and AI‑powered JSON extraction into one call.

Instead of crawling the site and then running a separate extraction step, you can tell the Crawl API exactly what fields you want to extract and it will return structured JSON using Workers AI.

Crawl Request: Extract Product Data

Below is an example curl command that starts a crawl on a hypothetical shop's products page and extracts key product information using a JSON schema:

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
  -H 'Authorization: Bearer <apiToken>' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://shop.example.com/products",
    "limit": 50,
    "formats": ["json"],
    "jsonOptions": {
      "prompt": "Extract product name, price, description, and availability",
      "response_format": {
        "type": "json_schema",
        "json_schema": {
          "name": "product",
          "properties": {
            "name": "string",
            "price": "number",
            "currency": "string",
            "description": "string",
            "inStock": "boolean"
          }
        }
      }
    },
    "options": {
      "includePatterns": [
        "https://shop.example.com/products/*"
      ]
    }
  }'

What This Does

• Starting URL: Begins crawling at the products listing page.
• limit: Limits the crawl to 50 pages, enough to cover most basic product catalogs.
• formats: Specifies ["json"], so results are returned as structured JSON rather than plain HTML or Markdown.
• jsonOptions: Provides a prompt and a JSON schema to guide Workers AI in extracting exactly the fields you care about: product name, price, description, currency and availability.
• includePatterns: Ensures only URLs that match the “products” path pattern are included in the crawl.

Why This Matters

In traditional web scraping setups, you would need to:

• Crawl pages separately.
• Parse HTML with custom extraction logic.
• Normalize the data into a structured format.

With Cloudflare's Crawl API, you combine crawling, JavaScript rendering, link discovery and AI‑powered extraction in one API call, drastically simplifying your workflow. It's ideal for:

• E‑commerce analytics dashboards
• Price comparison tools
• AI‑powered product search & recommendations
• Automated catalog generation pipelines

By providing a prompt and schema, the Crawl API returns clean, typed JSON without custom scraping logic, which saves time and reduces engineering complexity.

3. Fast Static Content Fetch : Crawl Static Sites without Rendering

When you know that the content you need is already present in the initial HTML, such as on blogs, brochure sites or static documentation. you can tell the Crawl API to skip JavaScript rendering. This makes the crawl much faster and more efficient because Cloudflare doesn't launch a headless browser for each page.

Here's how you perform a fast static content fetch using the /crawl endpoint:

Example : Crawl Static HTML Pages

curl -X POST "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl" \
  -H "Authorization: Bearer <apiToken>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "limit": 100,
    "render": false,
    "formats": ["html", "markdown"]
  }'

What This Request Does

• url: The starting page for the crawl.
• limit: Tells the API to crawl up to 100 pages.
• render: false: Disables JavaScript execution and performs a simple HTML fetch instead of spinning up a browser.
• formats: Requests results in both raw HTML and Markdown for easier content consumption or analysis.

By disabling rendering (render: false), the crawl becomes faster and more cost‑efficient, especially for static sites where the HTML doesn't depend on client‑side JavaScript to load content.

When to Use render: false

• Your pages are fully rendered on the server
• You don't need the crawler to execute JavaScript
• You want quicker results and lower compute usage
• You're crawling simple static content like blogs or documentation

Cloudflare's /crawl endpoint will still follow links and extract content. it simply skips the headless browser step unless you specifically enable it.

4. Crawl with Authentication (Cloudflare /crawl API)

When you need to crawl content that is behind HTTP authentication or requires custom headers (for example, API‑key access), Cloudflare's Crawl endpoint lets you include authentication credentials directly in your crawl request. This makes it possible to crawl protected pages (like internal docs or API endpoints) that would otherwise be inaccessible with a simple anonymous crawl.

I. Crawl with Basic HTTP Authentication

If the site you're crawling requires a username and password (standard HTTP basic auth), you can include an authenticate object in your request body. This tells the crawler to send the credentials as part of the request to the target site.

Example : Basic Auth Crawl

curl -X POST "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl" \
  -H "Authorization: Bearer <apiToken>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://secure.example.com",
    "limit": 50,
    "authenticate": {
      "username": "user",
      "password": "pass"
    }
  }'

In this example:

• url is the protected page you want to crawl.
• limit determines how many pages are crawled.
• authenticate contains the basic authentication credentials that Cloudflare will send when requesting the page.

This allows the crawler to access content that is normally blocked by a login prompt or HTTP basic authentication.

II. Crawl with Token‑Based or Custom Header Authentication

Some APIs or services require a token or custom HTTP header instead of basic auth. You can include these headers in the crawl request with the setExtraHTTPHeaders option.

Example : Token‑Based Auth Crawl

curl -X POST "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl" \
  -H "Authorization: Bearer <apiToken>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://api.example.com/docs",
    "limit": 100,
    "setExtraHTTPHeaders": {
      "X-API-Key": "your-api-key"
    }
  }'

How this works:

• The crawler includes the X-API-Key header in every request it makes during the crawl.
• This allows it to authenticate with APIs or endpoints that expect token‑based authentication.

This technique is especially useful for crawling internal documentation portals, private APIs or content that returns data only when an API token or session token is present.

Why Authentication Options Matter

Many real‑world websites and APIs are protected behind authentication mechanisms. Without supporting credentials or headers, crawlers only return the public content. By providing credentials or custom headers:

• You can crawl login‑protected documentation
• You can index internal knowledge bases for AI systems
• You can extract data from API‑protected endpoints

These features make the Crawl API much more flexible and suitable for enterprise workflows where most content isn't publicly accessible.

5. Wait for Dynamic Content

Some modern websites, especially single‑page applications (SPAs) load their content after the initial HTML response through JavaScript. To ensure the Cloudflare crawler captures the fully rendered page instead of just an empty shell, you can use the gotoOptions and waitForSelector options in your crawl request.

Example : Crawl with Dynamic Content Wait

curl -X POST "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl" \
  -H "Authorization: Bearer <apiToken>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://app.example.com",
    "limit": 50,
    "gotoOptions": {
      "waitUntil": "networkidle2",
      "timeout": 60000
    },
    "waitForSelector": {
      "selector": "[data-content-loaded]",
      "timeout": 30000,
      "visible": true
    }
  }'

What This Does

• gotoOptions.waitUntil: "networkidle2" - Tells the crawler to wait until network activity is mostly finished (no more than two pending requests). This helps ensure content loaded via JavaScript is fully available before extraction.
• gotoOptions.timeout: 60000 - Allows up to 60 seconds for the page to finish loading dynamic content.
• waitForSelector - Instructs the crawler to wait until a specific DOM element appears (in this case, one marked by data-content-loaded) before proceeding. This helps avoid capturing incomplete pages.

This approach is especially useful when crawling JavaScript heavy applications where important content is loaded asynchronously and not available in the initial HTML.

6. Block Unnecessary Resources (Speed Up Crawl)

When crawling a site where you only need text content (not images, videos, fonts,or CSS), you can instruct Cloudflare's Crawl API to block specific resource types. This reduces network requests, speeds up crawling and lowers resource usage, especially helpful for large crawls focused on text extraction or structured data. (developers.cloudflare.com)

Example : Crawl While Blocking Images & Media

curl -X POST "https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl" \
  -H "Authorization: Bearer <apiToken>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "limit": 100,
    "rejectResourceTypes": [
      "image",
      "media",
      "font",
      "stylesheet"
    ]
  }'

What This Does

rejectResourceTypes tells the crawler to block requests for specific kinds of resources like:

• image : prevents loading JPG, PNG, GIF, SVG, etc.
• media : blocks audio and video files.
• font : stops web fonts from downloading.
• stylesheet : skips CSS files, which can speed up the crawl.

By blocking these, the crawler only requests text and essential document content, speeding up crawling and reducing processing costs. Blocking unnecessary resources is particularly useful when you only care about HTML structure, text or JSON output and not how the page looks visually. That way, the crawl focuses on what matters the content without wasting time fetching images or style files.