Retail Extraction: Architecting Google Shopping Pipelines (2026)
Google Shopping pulls together a wide mix of sellers—Shopify stores, big-box retailers, and smaller shops—in one results surface.
That makes it useful for price monitoring and competitive checks: you can compare how a product shows up across sellers and regions. The catch is the page changes often, a lot of markup is loaded in the browser, and Google rate-limits aggressive scraping.
Here’s a practical way to get structured JSON out of Google Shopping with the Apify platform.
WAFs and geography
Google invests heavily in bot detection. For Shopping, you generally need residential-style IPs and careful input parameters—not a single datacenter IP hammering the same endpoint.
What goes wrong (geo mismatch): The SERP you see depends on where the request looks like it’s coming from. If you query "Playstation 5" from a Frankfurt datacenter IP, you’ll mostly get German sellers and prices in euros. If your pipeline assumes US dollars, you either break on parse or store wrong numbers without noticing.
The Google Shopping Scraper on Apify lets you set inputs such as country so proxy routing lines up with the market you actually want.
Pagination stops around ~120 items
Google Shopping doesn’t scroll forever. For broad queries it typically caps you around 10–12 pages—on the order of 120 items. Pushing past that with &start=150 often gives you empty HTML or repeated earlier pages.
Workaround: split the query
To cover a large catalog (say thousands of SKUs), break one vague search into many specific ones so each run stays under the cap.
Instead of only "Running Shoes", use tighter strings, for example:
"Nike pegasus running shoes size 10 black""Hoka clifton 9 running shoes size 11 red"
Run those in parallel (within rate limits), merge the JSON, and dedupe in your database (e.g. Postgres). That’s how teams cover more than a single SERP page set allows.
Example input and what you get back
Typical Actor input looks like:
{
"queries": ["wireless noise cancelling headphones"],
"countryCode": "US",
"maxItems": 100
}
Apify runs the browser workload (including handling many of the usual anti-bot hurdles) and returns structured rows. Useful fields often include:
| Field | Why it matters |
|---|---|
price & originalPrice | Spot discounts and how aggressive pricing is across sellers. |
seller | See who is listing low—useful for MAP or channel checks. |
reviews & rating | Rough signal of listing quality vs. price. |
productLink | Jump-off URL for a deeper scrape on the merchant site. |
Follow-up scrapes for “real” landed cost
Shopping results are a strong discovery layer: you get a headline price (e.g. $299.99) and a link. They usually won’t show shipping, cart promos, or live stock the way the merchant’s checkout does.
A common two-step flow:
- Discovery: Run the Google Shopping Scraper and collect product URLs (e.g. 50 competitor Shopify product pages).
- Deep pass: Feed those URLs into a
PlaywrightCrawler(or similar) that can walk the merchant site—sometimes simulating cart or shipping—to approximate total cost.
Why teams offload this to Apify
Hand-rolled Python against raw SERP HTML breaks often: selectors and layout shift, and you’re on the hook for proxies and retries.
Running extraction on Apify moves browser + anti-bot maintenance to maintained actors so your team spends more time on pricing logic, alerts, and storage—not chasing DOM diffs.
Not for competitive market-wide pricing. Google’s Shopping Content API is for merchants uploading feeds—not for querying everyone else’s prices. Scraping the public Shopping UI is still how most teams read the market.
From many EU IPs, Google shows a cookie consent wall first. A dumb parser may grab button text instead of prices. Configure your Actor or workflow to accept or dismiss consent where allowed, or use tooling that already handles it.