E-Commerce Data Collection Guide 2026: Prices, Products, Reviews, and Inventory

Contains affiliate links. Disclosure.

E-commerce data collection in 2026 spans four core types: product catalog, pricing, reviews and ratings, and inventory. Each has different crawl frequency needs, anti-bot considerations, and target-specific approaches. This guide covers data types, price monitoring architecture, target breakdowns (Amazon, Shopify, WooCommerce), review scraping strategies, inventory tracking, Apify Store Actors, data schemas, and a price alert system. For pre-built scrapers, the Apify Store offers Amazon, Shopify, and Walmart Actors. For anti-bot heavy targets, Bright Data provides Scraping Browser and datasets.

Four Types of E-Commerce Data

Type	Typical Fields	Crawl Frequency	Anti-Bot
Product catalog	title, description, images, category	Daily–weekly	Moderate
Pricing	price, currency, sale_price, timestamp	Hourly (competitive) / daily	High (dynamic pricing)
Reviews/ratings	rating, review_count, review_text	Daily–weekly	Varies
Inventory	in_stock, quantity (if exposed)	Hourly–daily	Moderate

Product catalog changes slowly. Pricing can change hourly during promotions. Reviews accumulate over time. Inventory status matters for availability alerts. Design your pipeline around these cadences—see web scraping architecture patterns for queue-based and managed setups.

Price Monitoring Architecture

Crawl frequency — For competitive price monitoring: hourly or every few hours. For catalog-only: daily is sufficient.

Delta detection — Store each run's output. Compare with previous run to detect price changes. Only alert or persist when delta exceeds threshold (e.g., >5% drop).

Data flow:

[Cron / Scheduler] → [Scraper] → [Staging DB] → [Delta Comparison] → [Alert / Update]

Use idempotent writes (upsert by product_id). Log timestamps for lineage. For retry logic, see Apify error handling.

Target-by-Target Breakdown

Amazon

• Apify Actor — Use a pre-built Amazon scraper from the Apify Store. Handles pagination, product detail, reviews.
• Anti-bot — Amazon blocks aggressively. Residential proxies or Bright Data Scraping Browser recommended.
• Data — ASIN, title, price, rating, review count, availability. Some Actors extract bullet points and images.

Shopify Stores

• Sitemap + product JSON — Many Shopify stores expose https://store.com/products.json and a sitemap. Use sitemap for URL discovery; products.json for structured data (often no browser needed).
• Custom themes — Some themes render products via JavaScript. Use Playwright or Cheerio if HTML is static.
• Apify Store — Shopify store scrapers available. Configure start URLs or sitemap.

WooCommerce

• REST API — WooCommerce exposes /wp-json/wc/v3/products with authentication. If you have API keys, prefer the API over scraping.
• HTML fallback — For public stores without API access, scrape product listing and detail pages. Structure similar to Shopify.

Generic E-Commerce Sites

• Structured data — Many sites use JSON-LD Product schema. Parse <script type="application/ld+json"> for name, price, availability.
• HTML parsing — Fallback to CSS selectors for product cards, prices, and "Add to cart" for stock inference.
• Firecrawl — Use Firecrawl e-commerce product scraping with extraction schemas for LLM-friendly output.

Review Scraping: JSON-LD vs HTML

Structured data (JSON-LD) — Sites may embed AggregateRating or Review in <script type="application/ld+json">. Parse this first—structured, reliable.

HTML parsing — Reviews in DOM require selectors for rating stars, review text, date. Layouts vary; use robust selectors.

Third-party (Trustpilot, G2) — Aggregate review platforms. Separate scrapers or APIs. Some Apify Actors target Trustpilot and G2.

Inventory Tracking

In-stock / out-of-stock — Look for:

• "Add to Cart" button presence (in stock) vs "Out of Stock" / disabled button
• availability in JSON-LD
• Dedicated stock indicators in HTML

Quantity — Rarely exposed. When it is, use with care (sites may hide it to prevent scraping).

Delta detection — Track (product_id, in_stock, timestamp). Alert when status flips from in-stock to out-of-stock (or vice versa) for high-priority SKUs.

E-Commerce Anti-Bot Considerations

• Dynamic pricing — Prices can vary by session, geo, or A/B test. Use consistent proxy location. Browser rendering may be needed if price loads via JavaScript.
• JavaScript rendering — Product grids and prices often render client-side. Use Playwright or Crawlee PlaywrightCrawler.
• Rate limits — E-commerce sites throttle. Add delays, rotate IPs, use residential proxies.

Apify Store E-Commerce Actors

Actor Type	Use Case
Amazon Product Scraper	Products, prices, reviews, ASIN
Amazon Search Scraper	Search results, compare sellers
Shopify Store Scraper	Products from Shopify stores
Walmart Product Scraper	Walmart catalog and pricing

Search the Apify Store for "Amazon," "Shopify," "Walmart" to find maintained Actors. Configure proxy (e.g., Bright Data) in Actor input for blocked targets.

Data Schema Example

{
  "product_id": "B08N5WRWNW",
  "title": "Product Name",
  "price": 29.99,
  "currency": "USD",
  "sale_price": null,
  "in_stock": true,
  "rating": 4.5,
  "review_count": 1234,
  "url": "https://...",
  "timestamp": "2026-03-19T10:00:00Z"
}

Use product_id (or URL) as primary key for upserts. timestamp supports freshness checks and time-series analysis.

Price Alert System

1. Store to DB — Scraper writes to PostgreSQL/SQLite with (product_id, price, timestamp).
2. Cron comparison — Run a job every hour: fetch latest run, compare with previous. If price dropped >5%, trigger alert.
3. Slack webhook — Post to Slack: "Product X dropped from $50 to $42 (16% off)."
4. Optional — Add email or in-app notification. Use idempotent alerting (don't re-alert for same drop).

Comparison: DIY vs Apify vs Bright Data Datasets

Approach	Effort	Maintenance	Anti-Bot	Best For
DIY (Crawlee/Scrapy)	High	You	You provide proxies	Custom logic, full control
Apify Actors	Low	Community/Apify	Add proxy in input	Pre-built scrapers, scheduling
Bright Data Datasets	Very low	Bright Data	Built-in	Pre-collected e-commerce data

DIY — Full control over schema and logic. You handle proxies and retries. See web scraping architecture patterns.

Apify — Pick an Actor, set start URLs, add proxy. Scheduling and API included. Try Apify.

Bright Data Datasets — Pre-collected product and price data. No scraper to run. Best when you need broad coverage without building.

Handling Pagination and Scale

E-commerce listings paginate. Strategy: discover URLs from sitemaps or category pages, then scrape each product URL. For search results, handle "Load More" or page=2,3,… parameters. Crawlee and Scrapy both support adding new URLs to the queue from within the request handler—follow pagination links and enqueue them. Set maxRequestsPerCrawl to avoid runaway crawls. For millions of products, consider distributed workers (see web scraping architecture patterns) or managed platforms that scale automatically.

Legal and Terms of Service

Respect robots.txt and rate limits. E-commerce ToS often prohibit automated scraping—evaluate legal risk per jurisdiction and use case. For pricing intelligence, many businesses use licensed data feeds where available (e.g., manufacturer feeds, retailer APIs). When scraping is necessary, minimize impact: polite delays, identifiable User-Agent for contact, and avoid scraping personally identifiable information. See our legal compliance framework for detailed guidance.

Normalization and Deduplication

Product data varies by source. Normalize: map "Currently unavailable" and "Out of stock" to a consistent in_stock: false. Standardize currencies and units (e.g., always store price in cents or base currency). Deduplicate by product_id or canonical URL—same product may appear in multiple categories. Use upsert (ON CONFLICT) so each run refreshes the record instead of creating duplicates.

Choosing Crawl Targets

Prioritize high-impact SKUs. For price monitoring, focus on competitors' bestsellers and your own catalog. For review aggregation, start with top products by sales. Use sitemaps and category pages to discover URLs; avoid blind crawling. Set maxRequestsPerCrawl and schedule runs to balance freshness with cost. For 10K products, daily might suffice; for 1M, consider sharding by category or brand and staggering runs. The Firecrawl e-commerce guide shows schema-based extraction for product data with LLM-friendly output.

Start with Apify Store

For Amazon, Shopify, Walmart, use an Apify Actor first. Add Bright Data proxy if blocked. Build custom only when pre-built doesn't fit.

Apify E-Commerce Actors | Bright Data Datasets

Frequently Asked Questions

For competitive monitoring: hourly. For catalog and reviews: daily. Balance freshness with rate limits and cost.

Use residential proxies and browser rendering. Apify Amazon Actors and Bright Data Scraping Browser are built for this. Avoid high concurrency from few IPs.

Many expose /products.json and a sitemap. Try that first (no browser). For custom themes, use Playwright or an Apify Shopify scraper.

Include product_id, title, price, currency, in_stock, rating, review_count, url, timestamp. Use product_id for idempotent upserts.

Scrape regularly, store in DB. Run a cron that compares latest vs previous run. If price drops beyond threshold, send Slack webhook or email. See the price alert section above.