Apify Website Content Crawler: what it costs and how it fails
The Website Content Crawler is an Apify Actor that crawls a site and writes one dataset row per page, containing the article text as Markdown, plain text or HTML, plus page metadata. The developer charges nothing for it. You pay only Apify platform compute, billed in compute units, so a crawl of static documentation costs cents and a crawl of a heavy JavaScript site costs dollars.
Everything below was verified against Apify's own API and pricing page on 2026-09-09. Where two Apify pages disagree, this page says so instead of picking one.
What does it cost per 1,000 pages?
The Actor's own README quotes its cost estimates against a $0.25/CU baseline. Apify's live pricing page lists $0.2/CU subject to change · verified 2026-09-09 for the Free, Starter and pay-as-you-go tiers. Same compute, a fifth cheaper than the README's arithmetic assumes.
| Crawler type | README estimate, at $0.25/CU | Same crawl at today's $0.2/CU |
|---|---|---|
| Raw HTTP client (Cheerio) | $0.20 / 1,000 pages | ~$0.16 / 1,000 pages |
| Headless browser | $0.50–$5.00 / 1,000 pages | ~$0.40–$4.00 / 1,000 pages |
Source: apify.com/apify/website-content-crawler and apify.com/pricing, both fetched 2026-09-09. The right-hand column is our arithmetic (0.2 ÷ 0.25 = 0.8), not a figure Apify publishes. The README's own word for its estimates is "approximately", and cost swings with page weight, network speed and how often the adaptive crawler decides it needs a browser.
Worked example, 5,000-page documentation site. Over raw HTTP that is roughly $0.80 of compute. Through a headless browser it is roughly $2 to $20, depending on how heavy the pages are. The free plan includes $5/month subject to change · verified 2026-09-09 of platform credit, so the raw-HTTP version of that crawl fits inside the free tier with room left over. The browser version of a heavy site may not.
The add-on that is not in those numbers
Summarize pages with AI is off by default and is billed separately. It starts extra runs of the OpenRouter Actor against your account, which the README prices at about $2 to $3 per 1,000 summarized pages. Pages with no headings fall back to summarizing the whole body and can reach about $0.007 each. On a 5,000-page crawl that add-on costs more than the crawl. Turn it on only when you need the summary field.
Storage, data transfer and residential proxy bandwidth are billed on top of compute as well. See Apify pricing for how those lines add up.
How often do these crawls actually finish?
The Store page shows 99.2% runs succeeded. That number is real, but it is calculated over succeeded plus failed runs only. Apify's API exposes the full 30-day breakdown, and it includes two more terminal states.
| Run outcome, 30 days to 2026-09-09 | Runs | Share |
|---|---|---|
| SUCCEEDED | 3,005,093 | 96.60% |
| TIMED-OUT | 68,116 | 2.19% |
| FAILED | 25,657 | 0.82% |
| ABORTED | 11,855 | 0.38% |
| Total | 3,110,721 | 100% |
Source: stats.publicActorRunStats30Days from api.apify.com/v2/acts/apify~website-content-crawler, fetched 2026-09-09.
Read the third row against the second. Runs time out 2.7 times more often than they fail. So the setting that decides whether you get your data is the crawl cap and the run timeout, not the retry count. A timed-out run is not empty either: the rows written before the clock ran out are already in the dataset, and you can resume the remainder with a narrower start URL.
The Actor ships with maxCrawlPages set to 9,999,999 and maxCrawlDepth set to 20. On a site with faceted URLs that is an unbounded crawl. Set both before your first run.
How do you set up a crawl that will not run away?
- Enter start URLs. Sub-pages of each start URL are crawled by default.
https://example.com/blogalso picks uphttps://example.com/blog/post. Use Include URL patterns (globs) to reach beyond that, Exclude URL patterns to cut a section out. - Set
maxCrawlPagesandmaxCrawlDepth. Depth0crawls only the start URLs. Pages skipped as canonical duplicates still count against the page cap, so budget above the page count you expect. - Choose a crawler type. See the table below.
- Run 20 pages first. Open the dataset and read the
markdownfield on two or three rows. If it contains the nav bar, fixremoveElementsCssSelectoror the HTML transformer now, not after 5,000 pages. - Re-run at full scope, then export. Download JSON, CSV or Excel from the Output tab, or read the dataset through the Apify API for a pipeline that runs on a schedule.
Two switches are worth knowing about and are both off by default. useSitemaps enqueues URLs from the site's sitemap, which reaches pages no internal link points at, at the cost of a slow start on large sites. useLlmsTxt looks for /llms.txt at the domain root and follows the Markdown files it lists, which is usually the cleanest possible source on a site that publishes one.
Which crawler type should you pick?
| Crawler type | Renders JavaScript | Use it when |
|---|---|---|
| Adaptive switching (default) | Per page, automatically | You do not know the site, or it mixes static and dynamic pages |
| Raw HTTP client (Cheerio) | No | Server-rendered documentation, blogs, and anything you can read with JavaScript disabled |
| Headless browser (Firefox+Playwright) | Yes | JavaScript-only content, or a site that blocks you. Slower and more expensive |
| Raw HTTP with JS (JSDOM) | Partially | Deprecated by Apify. Use Cheerio |
| Headless browser (Chrome+Playwright) | Yes | Deprecated by Apify. Use Firefox+Playwright |
Adaptive switching is the default and the right first choice. It is also the reason cost estimates have a tenfold spread: the price of a crawl depends on the fraction of pages it decides to open in Firefox.
About robots.txt
respectRobotsTxtFile is on by default, which is the right default. Two limits are documented on the Actor page and worth knowing before you rely on it: the crawler sends no specific user-agent identifier, and the crawl-delay directive is not supported yet. If you need to be a well-behaved crawler on someone else's infrastructure, cap concurrency yourself. On the legal side, see is Apify legal? for the general position on public data and copyright.
What does one dataset row look like?
| Field | Description | Example |
|---|---|---|
| url | The URL the crawler enqueued | https://docs.example.com/academy/web-scraping |
| markdown | Article body as Markdown. Written when saveMarkdown is on, which is the default | # Web scraping for beginners\n\nLearn how to... |
| text | Same body as plain text, no formatting | Web scraping for beginners Learn how to... |
| metadata.title | Page title tag | Web scraping basics | Documentation |
| metadata.description | Meta description | Learn how to extract information from websites. |
| metadata.canonicalUrl | Canonical URL, used to skip duplicates | https://docs.example.com/academy/web-scraping |
| metadata.languageCode | Detected language | en |
| crawl.depth | Link hops from the start URL | 0 |
| crawl.loadedTime | When the page was fetched | 2026-09-09T16:26:51.030Z |
| crawl.referrerUrl | The page the link was found on | https://docs.example.com/academy |
For a RAG pipeline, markdown is the field you chunk and url plus metadata.title are the citation fields you keep. There is a worked version of that in data for AI and RAG.
How many crawls can you run at once?
Here two Apify pages disagree, and both are first-party, so we are not going to guess.
| Source, checked 2026-09-09 | Free-plan concurrent Actor runs |
|---|---|
| apify.com/pricing | 5 |
| docs.apify.com/account/limits | 25 |
Both pages agree on the paid plans: 32 concurrent runs on Starter, 128 on Scale, 256 on Business. The safe assumption on the free plan is 5, the lower of the two. Design around that and a higher real limit costs you nothing; design around 25 and your queued crawls may simply sit there. If concurrency is load-bearing for you, run five crawls at once and count.
When is something else the better tool?
| You need | Better choice | Why |
|---|---|---|
| Markdown from a single URL, fast | RAG Web Browser in Standby mode | The Website Content Crawler README says so itself. A crawler is the wrong shape for one page |
| Prices, SKUs, ratings, structured fields | A targeted Store Actor, or Web Scraper | This Actor returns article text. It does not do field extraction |
| A scrape API with no platform account behind it | A standalone scrape API | This Actor assumes an Apify account and platform storage. If you want one HTTP endpoint and nothing else, compare the options on our Apify alternatives page rather than here. |
Be honest about that last row. A single-endpoint API is genuinely simpler than an Apify account when you will never crawl more than a few thousand pages a month, and a flat monthly number is easier to plan against than compute units. Apify wins the moment volume grows: at today's raw-HTTP figure, the free plan's $5 credit covers on the order of 30,000 pages, and there is no per-page credit to run out of. The full comparison lives in Apify vs Firecrawl.
Set maxCrawlPages to 20, run it against the site you care about, and read the markdown field. That test costs a fraction of a cent and tells you within two minutes whether the extraction is clean on your target site.
The free plan needs no credit card and includes $5 of platform credit a month, which is enough to answer that question many times over.
The Actor itself is free. You pay Apify platform compute, billed in compute units at $0.2/CU on the Free, Starter and pay-as-you-go tiers as of 2026-09-09. The Actor's README estimates $0.2 per 1,000 pages over raw HTTP and $0.5 to $5 per 1,000 pages through a headless browser, quoted against a $0.25/CU baseline, so at today's list rate expect roughly $0.16 and $0.40 to $4.00 respectively.
Yes, within the free plan's $5 of monthly platform credit and no credit card. At the raw-HTTP rate that covers on the order of tens of thousands of pages; through a headless browser on a heavy site it can be closer to one or two thousand. Run a 20-page test first and read your usage chart before scaling.
Because the crawl was larger than the run timeout allowed, which is the most common non-success outcome for this Actor. Across 3,110,721 public runs in the 30 days to 2026-09-09, 2.19% timed out against 0.82% that failed. Lower maxCrawlPages, raise the run timeout, or split the crawl by section. Rows written before the timeout are already in the dataset.
Yes. The default Adaptive switching crawler type uses raw HTTP for static pages and switches to Firefox via Playwright when a page needs JavaScript to expose its content. You can force the browser for every page by choosing Headless browser (Firefox+Playwright), which is more reliable on protected sites and costs more.
The Website Content Crawler follows links and is built for many URLs at once. RAG Web Browser in Standby mode returns Markdown for a single URL and is faster and more efficient for that job. The Actor's own README recommends RAG Web Browser for single-URL extraction.
The respectRobotsTxtFile input is enabled by default, so the crawler consults robots.txt before each page. Two documented limits: it sends no specific user-agent identifier, and the crawl-delay directive is not supported yet. If crawl rate matters to the site owner, cap concurrency yourself.
Markdown by default, plus plain text, and optionally full HTML and page screenshots. Each dataset row also carries url, metadata such as title, description, canonicalUrl and languageCode, and crawl fields including depth, loadedTime and referrerUrl. Export as JSON, CSV or Excel, or read the dataset through the Apify API.
Common mistakes and fixes
The extracted text is missing the page content.
The adaptive crawler guessed wrong and fetched over raw HTTP. Switch Crawler type to Headless browser (Firefox+Playwright) and re-run one URL to confirm before re-crawling the whole site.
The text contains navigation, cookie banners, and footers.
Change the HTML transformer, or add selectors to Remove HTML elements. The default removeElementsCssSelector already strips nav, footer, script, style and common dialog roles, so anything left over needs its own selector.
The run ends with status TIMED-OUT and a partial dataset.
This is the most common non-success outcome, not a crash. Lower maxCrawlPages, raise the run timeout in the run options, or split the crawl by section using Include URL patterns. The dataset written before the timeout is still readable.
The run fails after restarting several times.
The crawler restarts on crash and resumes, but the platform fails the run if it crashes more than three times per minute. Usually this is memory: lower Initial concurrency or raise the Actor memory.
The crawler is blocked by the target site.
Use the Stealthy web browser (Firefox+Playwright) crawler type together with residential proxies. Residential bandwidth is billed at $8/GB on the Free and Starter plans, so cap the crawl first.