Handling Dynamic Websites: AJAX, Infinite Scroll, and Single-Page Apps (2026)

Contains affiliate links. Disclosure.

Static HTML scrapers fail on modern sites: content loaded via AJAX, infinite scroll feeds, and single-page apps (SPAs) render in the browser, not the initial response. View Source shows a near-empty shell; Inspect Element reveals the full DOM. This guide covers three approaches—API interception (fastest), Playwright waits (reliable), and scroll/click triggers (for infinite scroll)—plus SPA-specific techniques and a Playwright vs httpx vs Crawlee comparison. Deploy on Apify for managed execution.

The Challenge: Why Static Scrapers Fail

Scenario	What You See	Why It Fails
AJAX-loaded list	Empty <div> in HTML, full list in browser	requests/BeautifulSoup gets initial HTML only
Infinite scroll	Load more on scroll	No scroll event; no new HTML without interaction
SPA (React/Vue)	Skeleton or loading state	Content appears after JS bundle runs
Lazy-loaded images	Placeholder until scroll	src in HTML may be data-src or lazy attribute

Detection test: Right-click → View Page Source. If the content you want is missing but visible in the rendered page, it's dynamic.

Approach 1: Intercept the AJAX Request (Best Performance)

Many dynamic sites fetch data from a JSON API. The browser requests it; you can too—without loading the full page.

Steps:

1. Open Chrome DevTools → Network tab → filter by XHR or Fetch.
2. Reload the page or trigger the load (scroll, click).
3. Find the request that returns the data you need. Note URL, query params, headers.
4. Call it directly with httpx or requests:

import httpx

# Example: product API
resp = httpx.get(
    "https://example.com/api/products?page=1&limit=50",
    headers={"User-Agent": "Mozilla/5.0 (compatible; Bot/1.0)", "Accept": "application/json"}
)
data = resp.json()
# Extract directly from JSON—no HTML parsing

Advantages: 10x faster than browser, minimal resources, easy to parallelize. No JavaScript execution.

When it works: The site uses a clean REST/GraphQL API. No cookies or tokens required (or you can obtain them once).

When it doesn't: Obfuscated endpoints, HMAC signatures, heavy session validation, or data embedded only in rendered HTML.

Approach 2: Playwright Wait Strategies

When API interception isn't feasible, use a real browser and wait for content to appear.

page.waitForSelector() — Wait for a specific element:

await page.waitForSelector('.product-listing', { timeout: 15000 });
const products = await page.locator('.product-card').all();

page.waitForResponse() — Wait for an API response:

const [response] = await Promise.all([
  page.waitForResponse(res => res.url().includes('/api/products') && res.status() === 200),
  page.goto('https://example.com/products')
]);
const data = await response.json();

page.waitForLoadState('networkidle') — Wait until no network activity for 500ms:

await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForLoadState('networkidle');

Best practice: Prefer waitForSelector or waitForResponse over networkidle—they're more predictable. networkidle can hang on sites with persistent WebSocket or analytics.

Approach 3: Infinite Scroll and Click Triggers

Some sites load more content only when you scroll or click "Load more."

Infinite scroll loop (Playwright):

let previousCount = 0;
let stableCount = 0;

while (stableCount < 3) {  // Stop after 3 scrolls with no new items
  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await page.waitForTimeout(2000);
  
  const items = await page.locator('.item').count();
  if (items === previousCount) {
    stableCount++;
  } else {
    stableCount = 0;
  }
  previousCount = items;
  
  if (items >= 100) break;  // Safety limit
}

Click "Load more":

while (await page.locator('button:has-text("Load more")').isVisible()) {
  await page.click('button:has-text("Load more")');
  await page.waitForLoadState('networkidle');
}

SPA Scraping: React, Vue, Client-Side Routing

SPAs serve a minimal HTML shell; the app hydrates and renders client-side. Key considerations:

• Hydration delay: Wait for the framework to render. waitForSelector on a content element usually suffices.
• Client-side routing: Navigating to /products may not trigger a full reload. Use page.goto() for each URL or page.click() for in-app navigation.
• Lazy routes: Some routes load code on demand. Wait for the route's content, not just domcontentloaded.

Example:

await page.goto('https://spa.example.com/products');
await page.waitForSelector('[data-testid="product-grid"]', { timeout: 15000 });
// Content is now rendered

Performance: Intercept vs Full Render

Approach	Speed	Resource Use	When to Use
httpx (API interception)	Fastest	Low	Clean API, no auth complexity
Playwright (full render)	Slower	High (browser)	No API, heavy JS, anti-bot
Crawlee	Medium	Medium	Apify deployment, queue + retries

Rule of thumb: If you can get the data from an API, do it. Reserve Playwright for when the site has no usable API or requires cookies, captchas, or human-like behavior. See Playwright vs Puppeteer vs Selenium 2026 for tool choice.

Comparison: Tools for Dynamic Sites

Tool	AJAX/API	Infinite Scroll	SPA	Best For
httpx/requests	✅ Direct API calls	❌	❌	API interception
Playwright	✅ waitForResponse	✅ Scroll/click	✅ Full render	Complex dynamic sites
Crawlee	✅ Via Playwright	✅ Via Playwright	✅ Via Playwright	Apify, production pipelines
Puppeteer	✅ Similar to Playwright	✅ Similar	✅ Similar	Node.js, Chrome-only
Selenium	✅ WebDriverWait	✅	✅	Cross-browser, legacy

For Python, use web scraping Python guide: httpx for API, Playwright for Python for browser. For Node/Apify, Crawlee with PlaywrightCrawler is the standard.

Prefer API interception

Before writing Playwright code, spend 5 minutes in DevTools → Network → XHR. If the data comes from a clean JSON endpoint, call it with httpx. You'll save time and compute.

Try Apify | Playwright vs Puppeteer

Frequently Asked Questions

View Source (Ctrl+U). If the content you want is missing, it's loaded dynamically. Check Network → XHR for API requests that return the data.

Intercept the underlying API. Use Chrome DevTools to find the XHR/fetch request, then call that URL directly with httpx or requests. Skip browser rendering entirely.

When there's no clean API, the endpoint uses complex auth, or the site requires cookies/captcha. Also for visual verification or when structure is only in rendered HTML.

Loop: scroll to bottom, wait 1–2 seconds, check if new items appeared. Repeat until no new items for N scrolls or you hit a safety limit.

Yes. Crawlee's PlaywrightCrawler uses Playwright under the hood. Use it for Apify deployment with built-in queue, retries, and proxy support.