Web Scraping with AI: How LLMs Are Transforming Data Extraction in 2026

Contains affiliate links. Disclosure.

LLMs did not replace CSS overnight, but they did change where teams spend time: instead of nursing a selector file per theme, you can often send cleaned HTML (or markdown) to a model and ask for JSON against a schema—handy when layouts drift or you touch many domains. Large language models reason about semantics, so fields like price, rating, and stock can come back without hand-built paths—at the cost of latency, tokens, and occasional guesses. The sections that follow spell out four common roles for LLMs in scraping (selector help, structured extraction, classification, validation), where that breaks down, and when traditional selectors or pre-built Actors still win. Try Firecrawl extract · Apify Store

The Shift: From Selectors to Semantics

Traditional scraping relies on CSS or XPath selectors. When a site changes its HTML structure, selectors break. AI extraction works differently: send cleaned HTML (or markdown) to an LLM with a schema or prompt, and the model returns structured JSON. Layout changes matter less because the model infers meaning from content.

Trade-off: AI extraction is more flexible but slower and costlier than CSS. Use it where layout varies or maintenance burden is high.

Four Ways LLMs Are Used in Scraping

Use Case	How it works	Example
1. Selector generation	LLM analyzes page and suggests CSS/XPath	"Extract product title" → .product-name or //h1[@class='title']
2. Structured extraction	HTML → LLM + schema → JSON	Send HTML, get {price, rating, availability}
3. Content classification	LLM labels pages or snippets	"Is this a product page or category?"
4. Quality validation	LLM checks extraction completeness	"Does this record have all required fields?"

Most practical today: structured extraction from unstructured HTML. Tools like Firecrawl, Apify AI extract, and Diffbot NLP expose this as API endpoints.

LLM-Powered Extraction Tools

Tool	Approach	Best For
Firecrawl /extract	Crawl + LLM extraction with schema or prompt	Product data, articles, leads
Apify AI extract	Actor-based; integrates with Crawlee	Custom pipelines, scheduled runs
Diffbot NLP	Pre-trained extractors + custom	Entities, products, articles

Firecrawl's extract endpoint crawls URLs (with optional wildcards), cleans content, and returns JSON matching your schema. Firecrawl extract docs. Apify's AI extraction integrates with the Actor ecosystem for scheduling and storage. Apify Store has pre-built extraction Actors.

How It Works: End-to-End

1. Fetch: Crawl or scrape the page; get HTML or markdown.
2. Clean: Strip ads, nav, footer. Pass main content to the LLM.
3. Prompt: "Extract from this HTML: product name, price, rating, availability. Return JSON matching this schema."
4. Validate: Check required fields, types. Retry or flag on failure.
5. Store: Push to dataset, database, or downstream pipeline.

Example schema:

{
  "type": "object",
  "properties": {
    "name": {"type": "string"},
    "price": {"type": "number"},
    "currency": {"type": "string"},
    "rating": {"type": "number"},
    "availability": {"type": "string"}
  },
  "required": ["name", "price"]
}

The LLM reads the HTML and populates the schema. No selectors—works across different site layouts.

Example: E-Commerce Without Custom Selectors

Extracting product data from a generic e-commerce page:

Input: HTML from https://example-shop.com/product/123

Output (LLM):

{
  "name": "Wireless Headphones Pro",
  "price": 149.99,
  "currency": "USD",
  "rating": 4.5,
  "availability": "In Stock"
}

Same prompt works for different sites. Layout changes rarely break extraction—the model adapts to new structures.

Limitations

Issue	Impact	Mitigation
Cost	LLM tokens per page; scales with volume	Use AI only where needed; pre-filter pages
Latency	Slower than CSS (seconds vs milliseconds)	Batch requests; use async
Hallucination	Model may fabricate or guess values	Strict schema; validation; confidence thresholds
Rate limits	API caps on LLM providers	Queue; backoff; tier upgrades

For high-volume production (millions of pages), traditional selectors or pre-built Actors are usually more cost-effective. AI shines for variable layouts, one-off jobs, and research.

When to Use AI Extraction

• Highly variable page layouts (many sites, one extractor)
• One-off scraping (no time to write selectors)
• Research / exploration (schema not yet fixed)
• Content that resists selectors (nested, non-semantic HTML)

When NOT to Use

• High-volume production (cost per page too high)
• Simple, stable structure (CSS is faster and cheaper)
• Real-time requirements (LLM latency too high)
• Strict accuracy needs (hallucination risk)

Comparison: CSS vs AI vs Pre-Built Actors

Approach	Maintenance	Cost	Latency	Flexibility	Best For
CSS / XPath	High (selectors break)	Low	Low	Site-specific	Stable, high-volume
AI extraction	Low	High (tokens)	Medium	High	Variable layouts, one-offs
Pre-built Actors	Low	Per-run	Low–medium	Template-defined	Common sites (Amazon, Google)

Hybrid strategy: use pre-built Apify Actors where they exist. For custom, variable sites, use Firecrawl extract or Apify AI. Reserve CSS for stable, high-volume targets where cost matters.

Hybrid Approaches: AI + Selectors

For many production systems, a hybrid works best:

• Primary: CSS or XPath for stable, high-volume fields (product ID, URL).
• Fallback: LLM extraction when selectors fail or layout varies.
• Validation: LLM checks completeness (e.g., "Does this product have price and name?") and flags gaps.

This reduces cost (AI only when needed) while improving robustness. Implement a selector-first pass; if extraction fails or returns null for critical fields, trigger an LLM extraction for that page.

Schema Design for AI Extraction

Effective schemas constrain the model and reduce hallucination:

• Use required — Models tend to fill required fields; omit optional ones if absent.
• Narrow types — "type": "number" for price; "enum": ["In Stock", "Out of Stock"] for availability.
• Provide examples — Few-shot examples in the prompt improve consistency.
• Validate output — Run JSON against the schema; reject and retry on validation failure.

Tools like Firecrawl accept JSON Schema directly. Apify AI extractors often use similar schema definitions. Test with diverse pages to catch edge cases.

Cost Control Strategies

• Page pre-filtering: Only send pages that match your target (e.g., product pages) to the LLM. Reduces token usage.
• Chunk batching: For long articles, split into sections. Extract per section and merge. Avoid sending entire 10K-word pages.
• Caching: If the same page is re-scraped, cache extraction results. Re-run LLM only when content hash changes.
• Tiered extraction: Use smaller models for simple pages; reserve larger models for complex layouts.

Vendor Comparison: Firecrawl vs Apify AI vs Diffbot

Vendor	Extraction Model	Schema Support	Best Fit
Firecrawl	Configurable (schema or prompt)	JSON Schema	API-first, variable layouts
Apify	Actor-dependent	Via Actor config	Scheduled, storage, integrations
Diffbot	Pre-trained NLP	Product, Article, etc.	Entity extraction, knowledge graphs

Real-World Example: Multi-Retailer Price Monitor

A common use case is monitoring product prices across retailers with different HTML. With AI extraction: (1) scrape product URLs—lightweight, no LLM; (2) for each URL, call Firecrawl extract or Apify AI with schema {name, price, currency, availability}; (3) store results and compare to previous runs; (4) alert when price drops. Without AI you maintain separate selectors per retailer. With AI, one schema handles all. At 1,000 products across 10 sites, AI is often justified; at 100,000 products, use hybrid (selectors where possible).

Implementation Tips

1. Narrow content: Send only main content div to the LLM; strip boilerplate.
2. Strict schema: Required fields, types, and enums reduce hallucination.
3. Retry logic: On validation failure, retry with a refined prompt or skip.
4. Fallback: Use AI when selectors fail; keep selectors as primary where possible.
5. Monitor cost: Track tokens per page; set alerts for budget overruns.

For full pipelines that combine scraping with LLM transformation, see LangChain Apify Content Pipeline.

Start with Firecrawl extract

For quick AI extraction without custom code, use Firecrawl's extract endpoint. Define a schema, pass URLs, get JSON. Scale to Apify when you need scheduling and storage.

Frequently Asked Questions

The LLM receives cleaned HTML (or markdown) plus a schema or prompt. It infers structure from content and returns JSON. No CSS selectors—the model understands semantics.

Not always. AI can hallucinate. For stable, well-structured pages, CSS is often more reliable. AI excels when layout varies across sites or changes frequently.

Varies by provider. Firecrawl and similar services charge per credit; extraction uses more credits than plain scrape. Expect roughly 10–50× the cost of CSS parsing for the same page.

Use pre-built Apify Actors for common targets (Amazon, Google, etc.). Use AI extraction for custom or variable layouts where no Actor exists or maintenance is high.

Use strict JSON schemas with required fields and types. Validate output and retry on failure. Send only relevant content, not full pages. Add confidence thresholds where supported.

Yes. Apify offers AI-powered extraction in some Actors and integrations. Check the Apify Store for AI extractors. Firecrawl's extract endpoint is another option for schema-based extraction.