Which AI sites can you actually scrape?
Hugging Face allows every crawler on every path. chatgpt.com disallows everything except a named allow-list that includes /share/. claude.ai leaves share pages open in robots.txt but bans automated access in its terms. gemini.google.com disallows /chat/. Each line below was read from the vendor's own file on 2026-09-09.
Two different jobs get searched with the same words. Scraping an AI product means pulling its public HTML. MCP means letting an assistant run Apify Actors as tools, which is a different setup entirely — that one lives in Apify MCP vs API. This page is about the first job.
What does each AI platform's robots.txt actually allow?
Every row was fetched on 2026-09-09 and reflects the User-agent: * group unless noted. These files change without a changelog, so re-read yours before a production run.
| Platform | User-agent: * rules | What that leaves open | Source |
|---|---|---|---|
| Hugging Face | Allow: / and nothing else | The whole site: model cards, dataset pages, Spaces, docs | huggingface.co/robots.txt |
| ChatGPT | A ~160-line allow-list, then Disallow: / | Only allow-listed paths, including /share/, /g/, /s/, /api/public_content/, /pricing, /features | chatgpt.com/robots.txt |
| Claude | Allow: /api/desktop/, Disallow: /api/*, Disallow: /lti/* | Everything outside /api/ and /lti/, including share pages — but see the terms below | claude.ai/robots.txt |
| Gemini | Allow: /app/download, Disallow: /app/, Disallow: /chat/ | The marketing surface. The conversation app is closed | gemini.google.com/robots.txt |
| Perplexity | Disallow: /*?*q=, /search/new, /search?*/, /marketing/prerelease/, /onboarding/, /join/ | Most of the site. Googlebot gets a stricter group that adds Disallow: /search* | perplexity.ai/robots.txt |
Two details in that table are worth more than the rest.
ChatGPT's file ends in a blanket Disallow: /, so the allow-list is the entire permitted surface. /share/ sits inside it. That is the same path a user gets when they click Share on a conversation, and it is why a share-link scraper is a legitimate build while a "scrape ChatGPT" scraper is not. The same file gives CCBot, Google-Extended, anthropic-ai, Claude-Web, PerplexityBot, FacebookBot and Bytespider their own blanket Disallow: / — OpenAI blocks rival AI crawlers by name while leaving generic ones on the allow-list.
Perplexity's Googlebot group disallows /search* while the wildcard group does not. So a shared Perplexity thread under /search/<slug> is outside Google's index but is not disallowed for a generic crawler. That asymmetry is deliberate on their side, and it is the kind of thing that flips between reads.
Does robots.txt mean you have permission?
No. Robots.txt is an instruction to crawlers. The contract is the terms of service, and on AI products the two disagree more often than they agree.
Claude is the clearest example. Its robots.txt does not disallow share pages, but Anthropic's Consumer Terms (effective 2025-10-08) list, under section 3, "To crawl, scrape, or otherwise harvest data or information from our Services other than as permitted under these Terms," and separately "Except when you are accessing our Services via an Anthropic API Key or where we otherwise explicitly permit it, to access the Services through automated or non-human means, whether through a bot, script, or otherwise" (anthropic.com/legal/consumer-terms). An open path plus a closed contract is still closed.
OpenAI's terms could not be read the same way. openai.com/policies/terms-of-use returned HTTP 403 to an automated fetch on 2026-09-09, so nothing from it is quoted here. Open it in a browser before you build against chatgpt.com.
Hugging Face is the outlier in the other direction. Its Terms of Service, effective 2022-09-15, contain no clause on scraping, crawling, automated access or rate limits — the permissive robots.txt is not contradicted by anything.
The general legal picture, including hiQ v. LinkedIn and the GDPR exposure that comes with collecting anything personal, is covered in is Apify legal and safe. Read it before you scale a job past a few thousand pages.
Which Apify Actor fits which surface?
| Surface | Actor | Pricing model |
|---|---|---|
| Public docs, model cards, marketing pages | Website Content Crawler | Pay per platform usage (compute units) |
ChatGPT shared links (chatgpt.com/share/...) | ChatGPT Conversation Scraper | Pay per event, priced by message count |
| Google AI Overviews in SERPs | See scrape Google SERP | Per result |
| A streamed chat UI you own or are authorised on | Web Scraper or your own Crawlee Playwright Actor | Compute units |
Prices, read from the Actor pages on 2026-09-09. Website Content Crawler estimates $0.5/1,000 pages subject to change · verified 2026-09-09 to $5/1,000 pages subject to change · verified 2026-09-09 in headless-browser mode, and $0.2/1,000 pages subject to change · verified 2026-09-09 with the raw HTTP crawler. Hugging Face model cards are static enough for the HTTP mode, so the twenty-five-fold gap is real money on a large crawl — check whether you need the browser before you turn it on.
The ChatGPT Conversation Scraper bills per conversation by size: $0.05 for 1–10 messages, $0.10 for 11–30, $0.20 for 31–100, $0.35 above 100, plus $0.02 to store the HTML and a $0.003 base fee per run. Its own listing states: "We only extract publicly shared conversations that users have explicitly made available via share links," and that it "cannot access private conversations or your ChatGPT account." That constraint is the point, not a limitation.
Start on the free plan before you commit: the crawler's own page notes Apify's free tier includes $5 of credit a month, which is enough to crawl roughly a thousand pages in HTTP mode and see the output shape for yourself.
Run Website Content Crawler on a public docs site
The obvious objection first: if the pages you want are behind a login, none of this applies and no Actor makes it apply. Automating a logged-in AI product is the case every one of those terms of service is written to stop.
What does the output look like?
Website Content Crawler's dataset fields, from its documented example output on 2026-09-09:
| Field | Description | Example |
|---|---|---|
| url | Requested page URL | https://huggingface.co/meta-llama/Llama-3.1-8B |
| text | Plain-text extraction of the main content | Llama 3.1 is a large language model... |
| markdown | Markdown version of the same content | # Llama 3.1 8B ... |
| html | Transformed HTML, only when saveHtml is on | <div>...</div> |
| summary | One-sentence AI summary, only when summarize is on | Model card for Llama 3.1 8B. |
| metadata.title | Page title from the rendered head | meta-llama/Llama-3.1-8B · Hugging Face |
| crawl.loadedTime | When this page was fetched | 2026-09-09T12:00:00.000Z |
| crawl.depth | Link depth from the start URL | 1 |
Keep crawl.loadedTime in whatever you build downstream. It is the only field that tells you how old a row is, and a RAG corpus without fetch timestamps cannot be refreshed selectively.
How do you wait for a streamed answer to finish?
This is the one problem AI surfaces have that ordinary dynamic pages do not. A chat UI appends tokens to the same DOM node over several seconds. waitForSelector fires on the first token, so a naive scraper captures half a sentence. A flat setTimeout either truncates a long answer or burns compute on a short one.
Poll the node's innerText length instead, and read it once the length has stopped changing for a fixed quiet period.
import { Actor } from 'apify';
import { PlaywrightCrawler } from 'crawlee';
// Read a node's innerText once its length has been stable for `quietMs`.
async function readWhenSettled(page, selector, { quietMs = 1500, timeoutMs = 90_000 } = {}) {
const deadline = Date.now() + timeoutMs;
let lastLength = -1;
let lastChangeAt = Date.now();
while (Date.now() < deadline) {
const length = await page
.$eval(selector, (el) => el.innerText.length)
.catch(() => -1);
if (length !== lastLength) {
lastLength = length;
lastChangeAt = Date.now();
} else if (length > 0 && Date.now() - lastChangeAt >= quietMs) {
return page.$eval(selector, (el) => el.innerText);
}
await new Promise((resolve) => setTimeout(resolve, 250));
}
throw new Error(`Text under ${selector} never settled within ${timeoutMs} ms`);
}
await Actor.init();
const crawler = new PlaywrightCrawler({
async requestHandler({ page, request, log }) {
// PlaywrightCrawler has already navigated to request.url. Do not call page.goto here.
const selector = '<the response container you found in DevTools>';
await page.waitForSelector(selector, { timeout: 30_000 });
const text = await readWhenSettled(page, selector);
log.info(`Settled at ${text.length} characters`);
await Actor.pushData({ url: request.url, text, fetchedAt: new Date().toISOString() });
},
});
await crawler.run(['https://example.com/your-authorised-target']);
await Actor.exit();
Four things about that code that matter more than the loop itself:
- The selector is yours to find. There is no portable one. Open the target in DevTools, find the element whose text grows while the answer streams, and prefer a stable
data-*attribute over a class name that ships from a build hash. $evalreturns the first match. A transcript with several assistant turns needs$$evaland a join, or a selector scoped to the turn you want.- 1500 ms is a starting value, not a measurement. A UI that pauses mid-answer while a tool call runs will look settled and is not. Log
lastLengthon every poll during your first run and pick the quiet period from the plateau you actually see. - The timeout must throw. Returning partial text silently is worse than failing, because a truncated answer looks like a valid row in the dataset and nothing downstream will flag it.
Rate limits, proxy escalation and the blocking behaviour you will meet on a real target are covered in web scraping challenges.
When is a scraper the wrong tool here?
Often, and this page would be dishonest not to say so.
If the platform ships a public API, use the API. Hugging Face publishes open Hub endpoints with an OpenAPI specification at huggingface.co/.well-known/openapi.json, subject to account-level rate limits. Structured JSON that the vendor maintains beats a crawler that breaks the next time they change a CSS class. Reach for Website Content Crawler on Hugging Face only for the prose in model cards and docs, which the API does not return as clean text.
If the data lives behind a login, stop. Anthropic's terms above are explicit, Google closes /chat/ in robots.txt, and OpenAI disallows everything outside its allow-list. There is no configuration that turns that into a compliant job.
If you only need a handful of shared conversations, the per-event pricing above is cheaper than the hour you would spend writing selectors. If you need a hundred thousand public documentation pages, write the crawler.
What changed on this page, and when
- 2026-09-09 — Rebuilt around per-vendor robots.txt and terms, all re-read on the day. Replaced a flat two-second sleep in the code sample with
innerText-length stabilisation. Corrected the Website Content Crawler output fieldcrawl.loadedAttocrawl.loadedTime. Added the ChatGPT Conversation Scraper's pay-per-event prices. Recorded that OpenAI's terms page returned 403 to an automated fetch, rather than quoting it from memory.
FAQ
Only the paths chatgpt.com allows. Its robots.txt ends in a blanket Disallow: / after an explicit allow-list that includes /share/, so publicly shared conversation links are open and the rest of the product is not. Read OpenAI's terms of use in a browser before you build.
Its robots.txt is a two-line Allow: / for all user agents, and its Terms of Service, effective 2022-09-15, contain no clause on scraping or automated access. Both were checked on 2026-09-09. For model and dataset listings the public Hub API is still the better source than a crawler.
Robots.txt does not disallow them, but Anthropic's Consumer Terms effective 2025-10-08 prohibit crawling or scraping the Services and accessing them through automated means without an API key. The contract governs, so treat share pages as off-limits.
Because waitForSelector fires on the first token. Streamed UIs append text to the same node for several seconds. Poll the node's innerText length and only read it once the length has stopped growing for a fixed quiet period.
Website Content Crawler's own page quoted $0.5 to $5 per 1,000 pages in headless-browser mode and $0.2 per 1,000 with the raw HTTP crawler on 2026-09-09. Static model cards and docs usually work in HTTP mode, which is where the difference is.
No. Scraping pulls public HTML from a site. The Model Context Protocol lets an assistant call Apify Actors as tools through the hosted endpoint at mcp.apify.com, which is not an Actor you run. See the Apify MCP guide for that setup.