How to scrape AI-powered sites, chat UIs, and AI platforms

Contains affiliate links. Disclosure.

Quick answer

Apify can scrape AI-generated websites, chatbot interfaces, and dynamic AI apps using Playwright-based Actors that fully execute JavaScript and wait for dynamic content. You still need explicit waits (selectors, timeouts, or custom logic) for streamed model output, and you must stay within each platform’s terms and public-data rules.

Browse Apify Actors · Website Content Crawler

Two workflows are often confused: (1) scraping public AI-related pages (model cards, marketing sites, shared conversation links) versus (2) connecting ChatGPT to Apify via MCP so an assistant can run Actors. This guide focuses on scraping web UIs; for MCP, see Use MCP servers.

⏱️Time~10–30 min

CostActor-dependent

DifficultyIntermediate

OutputCSV / JSON / JSONL

Challenges of AI-powered sites

• Client-rendered shells: HTML arrives empty and content appears after hydration. Plain HTTP clients miss the final DOM.
• Streaming responses: Chat UIs append tokens over time, so scraping too early captures partial text.
• Infinite scroll and lazy loading: Model galleries and feeds load assets on demand.
• Bot mitigation: AI vendors and CDNs may challenge automated browsers. Residential or datacenter proxies and realistic fingerprints help but are not guaranteed.
• Legal and contractual risk: Many AI products forbid automated access to logged-in experiences. Prefer public pages and document your use case (legality overview).

Recommended approach

1. Use a headless browser Actor: Website Content Crawler, Web Scraper, or Store Actors built on Playwright / Puppeteer.
2. Stabilize the DOM: Wait for selectors that indicate “response complete,” or add a short page.waitForTimeout / networkidle-style wait when the site has no stable selector (last resort).
3. Scope data extraction: Prefer structured fields (JSON-LD, API responses in HAR) when available, and fall back to CSS selectors on the rendered tree.
4. Rate-limit and rotate: Throttle concurrency and add proxies for difficult targets (scraping challenges).

Code example: Playwright-style wait in an Actor

Below is a minimal pattern you might use inside a custom Actor (Node) after Actor.launchPlaywright() or equivalent. Adapt to your SDK version and Crawlee class:

import { Actor } from "apify";
import { PlaywrightCrawler } from "crawlee";

await Actor.init();

const crawler = new PlaywrightCrawler({
  async requestHandler({ page, request, log }) {
    await page.goto(request.url, { waitUntil: "networkidle" });

    // Example: wait for a chat bubble or result container—replace with real selectors
    const selector = "[data-testid='ai-message'], .chat-message, .prose";
    await page.waitForSelector(selector, { timeout: 60_000 });

    // Optional: short pause while tokens stream in (avoid deprecated waitForTimeout)
    await new Promise((resolve) => setTimeout(resolve, 2000));

    const text = await page.$eval(selector, (el) => el.innerText);
    await Actor.pushData({ url: request.url, text });
  },
});

await crawler.run(["https://example.com/ai-demo"]);
await Actor.exit();

For token streaming, you may need longer waits, mutation observers, or capturing final innerText after innerText length stops changing. There is no universal selector, so inspect the target app in DevTools.

No-code: popular AI platform pages

These patterns target public pages, not private in-app chats.

ChatGPT (shared conversations)

Extract publicly shared conversation links only. Never scrape private sessions you are not authorized to access.

ChatGPT Conversation Scraper →

Hugging Face (model cards, datasets, docs)

Use a crawler Actor with start URLs on huggingface.co.

Website Content Crawler →

AI overviews in SERPs

To capture AI-style summaries in Google results, use the SERP scraper flow from Scrape Google SERP.

Example output shape (crawler)

Illustrative fields from a content crawler on a public model page:

📊Output Data Fields

Field	Description	Example
url	Source page URL	https://huggingface.co/meta-llama/Llama-3.1-8B
title	Rendered title	meta-llama/Llama-3.1-8B · Hugging Face
text	Main visible text	Llama 3.1 is a large language model...
markdown	Markdown extraction when enabled	# Llama 3.1 8B ...
metadata.description	Meta description	We're on a journey to advance AI...
crawl.loadedAt	Fetch timestamp	2026-03-21T12:00:00.000Z

💾Sample output (illustrative)

Fields vary by Actor and page.

{
  "url": "https://huggingface.co/meta-llama/Llama-3.1-8B",
  "title": "meta-llama/Llama-3.1-8B · Hugging Face",
  "text": "Llama 3.1 is a large language model developed by Meta AI...",
  "markdown": "# Llama 3.1 8B\n\n...",
  "metadata": {
    "description": "We're on a journey to advance and democratize artificial intelligence through open source and open science.",
    "languageCode": "en"
  },
  "crawl": {
    "depth": 1,
    "loadedAt": "2026-03-21T12:00:00.000Z",
    "httpStatusCode": 200
  }
}

What you can do with the data

• Research and benchmarking: Compare public model documentation and capabilities.
• RAG corpora: Ingest allowed public text into vector stores (data for AI & RAG).
• Monitoring: Track how AI landing pages and docs change over time.

More Actors

Search the Store for your specific target; new AI scrapers appear regularly.

Apify Store: AI scrapers

FAQ

Frequently Asked Questions

Yes, when you use browser Actors that execute JavaScript (typically Playwright or Puppeteer) and add waits suited to streaming or client-rendered content. Effectiveness depends on the site’s structure and protections.

They often stream tokens, load content lazily, and hide state inside closed WebSocket or fetch flows. You need stable selectors or custom wait logic, and sometimes proxies, to get complete text.

It depends on jurisdiction, contract terms, and whether data is public. Review each platform’s terms, avoid logged-in areas you are not allowed to automate, and read our scraping legality guide.

Scraping targets public web pages (for example shared links). MCP lets ChatGPT invoke Apify Actors as tools inside the chat product. See Use MCP servers for the latter.

Use browser Actors with realistic settings, reduce concurrency, and consider proxy plans. Success is target-specific, so monitor runs and adjust.

For general public pages, start with Website Content Crawler or Web Scraper. For specialized platforms, search the Store for the site name plus scraper.