Competitor Analysis with Web Scraping: The Complete Guide
Quick Answer
Competitor price scraping means running scheduled Actors that pull a rival's prices, catalog, reviews, and hiring pages automatically, then alert you on the diff instead of you checking by hand. On Apify, you run Actors on schedules, store structured rows in datasets, and feed Sheets, BI tools, or Slack via API and webhooks.
This guide maps each signal to a specific Actor, cron, and downstream destination, plus a working change-detection workflow for competitor pricing and messaging pages. If competitor tracking isn't quite the workflow you need, the full list of Apify use cases covers lead generation, RAG ingestion, and the other jobs Apify does well.
Updated 2026-09-09: re-verified every Actor link on this page against Apify's own API, added a named Actor for review scraping (previously just a bare Store-search link), and removed a monthly cost estimate that had no methodology behind it — see Cost ballpark for what replaced it.
Why scrape instead of checking manually
Manual competitor research (opening tabs, copying prices into a sheet, skimming reviews) is fine for a one-off snapshot. It breaks down the moment you need it weekly across several rivals.
| Approach | Manual checking | Scraping with Apify |
|---|---|---|
| Coverage | A handful of pages you remember to revisit | Hundreds of URLs across every competitor |
| Frequency | Whenever someone has time | Scheduled cron, daily or hourly |
| History | Lost unless you screenshot | Every row keyed and timestamped for diffs |
| Alerts | None | Webhook to Slack, email, or BI on threshold breach |
| Consistency | Subjective, easy to miss | Same fields captured every run |
The payoff is not just speed. Structured, timestamped rows let you see trends (discount cadence, review-sentiment drift, hiring spikes) that a manual glance never surfaces. For a wider view across a whole category rather than named rivals, pair this with market intelligence.
Types of data to track
| Signal | What to collect | What it tells you |
|---|---|---|
| Pricing | Product and PLP prices, promos | Margin pressure, discount cadence |
| Inventory | In stock, quantity hints | Velocity, supply stress |
| Reviews & Q&A | Stars, text, volume | Quality gaps, unmet needs |
| Catalog | New SKUs, categories | Expansion and seasonality |
| Content & SEO | Titles, H1s, meta, blog cadence | Messaging and keyword pivots |
| Jobs | Open roles, locations, seniority | Roadmap hints 6–12 months early |
| SERP | Rankings for target keywords | Organic share of voice |
| Social | Followers, posts, engagement (public) | Brand momentum and campaigns |
Prioritize 3–5 competitors and specific URLs (bestsellers, pricing page, careers, review profiles) so runs stay cheap and interpretable.
Recommended Actors by data type
| Data type | Starting point | What you learn | Guide |
|---|---|---|---|
| Amazon listings | Amazon Product Scraper | ASIN-level price, Buy Box, rating shifts | Scrape Amazon products |
| General e-commerce | E-commerce Scraping Tool | Cross-storefront price and stock | Scrape e-commerce prices · Best e-commerce scrapers |
| Site copy / blogs | Website Content Crawler | Messaging and content cadence over time | Scrape website content |
| Google SERP | Google Search Results Scraper | Organic share of voice for your keywords | Scrape Google SERP |
| Local presence | Google Maps Scraper | Locations, ratings, review velocity | Scrape Google Maps · Best Google Maps scrapers |
| Reviews (platform-specific) | Trustpilot All Reviews Scraper (G2, Capterra: Apify Store search) | Quality gaps and unmet needs | Browse Apify Store |
| Instagram / social | Instagram Scraper | Posting cadence, campaigns, engagement | Scrape Instagram · Best social media scrapers |
Verify input schema, pricing model, and rate limits on each Actor page before production schedules. Not sure which Actor fits a niche site? The most popular Actors list is a reliable starting shortlist.
Workflow: from scrape to decision
Step 1: Scope and URL list
Per competitor, capture:
- Hero product or category URLs you must not lose sight of
- Public pricing or plans page (SaaS)
- Careers landing page
- Third-party review profile URLs where buyers compare you
Store these in a sheet or JSON config your runs read via task input, one source of truth prevents drift.
Step 2: Run once, then schedule
- Run each Actor manually once from the Console. Confirm the dataset schema matches what you expect before you automate anything on top of it.
- In the Apify Console, open Schedules in the navigation menu and click Create new.
- Attach the Actor (or task) and set a cron expression (Apify schedules docs).
| Signal | Suggested cadence | Example cron |
|---|---|---|
| Pricing | Daily | 0 6 * * * |
| Reviews | Daily | 0 7 * * * |
| Content / SEO snapshot | Weekly | 0 8 * * 1 |
| Job postings | Weekly | 0 8 * * 3 |
| SERP | Weekly | 0 9 * * 1 |
Step 3: Normalize and key data
Use a stable key per entity: product URL, ASIN, job ID, or (competitor, keyword) for SERP. Add scrapedAt on every row so BI can time-series and diff.
Step 4: Alerts
Webhook on run succeeded → Make, n8n, or Zapier → rules such as:
- Price drops beyond X%
- Review average below threshold
- Spike in new roles in a new department
- Homepage H1 or pricing table text changed (diff against prior crawl)
Integration with analytics tools
Apify datasets are tabular JSON, so they map cleanly to the tools teams already use:
| Destination | Pattern |
|---|---|
| Google Sheets / Airtable | Scheduled append or upsert via integration or low-code workflow; good for shared scorecards. |
| Spreadsheet + Looker Studio | Sheet as a lightweight semantic layer for exec views. |
| Warehouse (BigQuery, Snowflake, Redshift) | API or orchestrator loads JSONL batches; join with CRM and product data. |
| BI (Power BI, Tableau, Looker) | Point at warehouse tables or synced CSV drops in object storage. |
| Slack / email | Webhook → workflow → formatted message with deep links to product or job URLs. |
For programmatic pulls, use the Apify API to list run datasets and page through items, ideal for nightly ETL owned by data engineering.
Working playbook: pricing + messaging change detection
No custom Actor required. Use apify/website-content-crawler in Markdown mode against a short list of high-signal competitor URLs (pricing, features, homepage hero, changelog), schedule weekly, hash each page's Markdown, diff against the prior run.
Schedule input (5 competitors × ~3 pages each):
{
"startUrls": [
{ "url": "https://competitor-a.com/pricing" },
{ "url": "https://competitor-a.com/changelog" },
{ "url": "https://competitor-b.com/pricing" }
],
"maxCrawlDepth": 0,
"saveMarkdown": true,
"crawlerType": "playwright:adaptive"
}
n8n change detector (triggered on Apify run succeeded webhook):
import crypto from 'crypto';
const today = $input.all().map(i => i.json);
const previous = $('Airtable - previous snapshots').all().map(i => i.json);
const changes = [];
for (const page of today) {
const hash = crypto.createHash('sha256').update(page.markdown || '').digest('hex');
const prev = previous.find(p => p.url === page.url);
if (prev && prev.hash !== hash) {
changes.push({
url: page.url,
title: page.metadata?.title,
prevHash: prev.hash.slice(0, 8),
newHash: hash.slice(0, 8),
wordDelta: (page.markdown.split(/\s+/).length - (prev.wordCount || 0)),
});
}
}
return changes.map(c => ({ json: c }));
Route changes to an LLM summarization node (Claude or GPT) with prompt: "Here is the old and new Markdown of {{url}}. In 3 bullets, describe what materially changed. Ignore nav, cookie banners, and dates." Post the summary to Slack #competitive with a link to the live page.
Overwrite the Airtable snapshot table with today so next Monday's run diffs against this week.
Job postings as leading indicators
| Hiring signal | Often implies |
|---|---|
| Many ML / data roles | New intelligent features or data products |
| Enterprise sales in a new geo | Expansion into your territory |
| Infra / SRE cluster | Scale event or reliability push |
How: crawl careers weekly, count roles by department/location, alert on new clusters compared to the prior snapshot.
Content and SEO change detection
Track title tags, H1s, pricing copy, and blog cadence. Workflow: weekly crawl → store text or hash → diff vs previous week → notify on meaningful changes (not trivial nav tweaks). Pair the on-page crawl with a weekly Google SERP scrape to see whether messaging changes move their rankings on the keywords you both target.
Using AI on top of scraped data
Summarize large review exports or price-change CSVs with an LLM. Keep PII out of third-party prompts unless you have a clear lawful basis.
Review batch prompt (template)
You are a competitive intelligence analyst. Summarize these reviews for {Competitor}:
1) Top complaints (with rough frequency)
2) Top praised themes
3) Sentiment trend vs older reviews if timestamps are present
Legal and ethical boundaries
Scrape public pages; do not bypass authentication you are not entitled to automate. Respect terms, robots.txt, and reasonable request rates. Do not use data to misrepresent a competitor. See Is web scraping legal?.
Cost ballpark
Most of the Actors above are priced pay-per-event, not a flat monthly rental, so a single "competitor stack" number would depend entirely on which Actors you run and how many rows you pull. The Amazon Product Scraper above, for example, charges $0.005 per result on the Free plan, falling to $0.003 per result on Business — Apify's Store discount tier tracks your plan (Free = no discount, Starter = Bronze, Scale = Silver, Business = Gold) — plus separate add-on events, which are not all small: on the Free tier the same Actor charges $0.0025 per offer, $0.0025 per seller, and $0.06 per delivery location — twelve times the per-result rate (api.apify.com/v2/acts/junglee~Amazon-crawler, checked 2026-09-09). Multiply each Actor's per-result rate by your expected row count instead of guessing at a monthly figure.
- Google Maps
- Google Maps Reviews
- Google Search
- TikTok
- Amazon
- X / Twitter
- Facebook Posts
- LinkedIn Profile
- Airbnb
- Website Content Crawler (compute-based)
| Plan | Monthly credit | This workload | Covered by credit? |
|---|---|---|---|
| Free$0 | $5.00 | $0.0040/place (no Store discount) → $40.00 | +$35.00 over |
| Starter$29/mo | $29.00 | $0.0030/place (Bronze) → $30.00 | +$1.00 over |
| Scale$199/mo | $199 | $0.0020/place (Silver) → $20.00 | Yes |
| Business$999/mo | $999 | $0.0015/place (Gold) → $15.00 | Yes |
Per-result rates follow the Apify Store discount tier attached to your plan (Free = no discount, Starter = Bronze, Scale = Silver, Business = Gold), read from Apify’s API on 2026-08-22; plan prices were checked 2026-08-22. Excludes proxy costs. Confirm current pricing on the Actor’s page before budgeting.
Per-unit rates for the Actors on this page, by plan (each plan's Apify Store discount tier in brackets):
| Actor (primary charge event) | Free (no discount) | Starter (Bronze) | Scale (Silver) | Business (Gold) |
|---|---|---|---|---|
| Amazon Product Scraper — per result | $0.005 | $0.004 | $0.0035 | $0.003 |
| E-commerce Scraping Tool — per product detail | $0.006 | $0.0015 | $0.00125 | $0.001 |
| Google Search Results Scraper — per search page | $0.0045 | $0.0025 | $0.00195 | $0.0018 |
| Google Maps Scraper — per place | $0.004 | $0.003 | $0.002 | $0.0015 |
| Instagram Scraper — per result | $0.0027 | $0.0023 | $0.0019 | $0.0015 |
| Trustpilot All Reviews Scraper — per review | $0.00075 | $0.00065 | $0.0006 | $0.0005 |
Two exceptions on this list: Website Content Crawler has no per-event price at all — it bills Compute Units ($0.20/CU on Free and Starter, $0.16 on Scale, $0.13 on Business). And the Trustpilot Actor adds a per-run start charge ($0.05 Free, $0.04 Bronze, $0.03 Silver, $0.025 Gold), which matters on a daily cron.
A four-signal competitor stack, per month:
| Line item (monthly volume) | Free | Starter | Scale | Business |
|---|---|---|---|---|
| Amazon Product Scraper — 6,000 results (200 products, daily) | $30.00 | $24.00 | $21.00 | $18.00 |
| Google Search Results Scraper — 200 search pages (50 keywords, weekly) | $0.90 | $0.50 | $0.39 | $0.36 |
| Trustpilot — 1,000 reviews + 30 daily runs | $2.25 | $1.85 | $1.50 | $1.25 |
| Instagram Scraper — 1,500 results | $4.05 | $3.45 | $2.85 | $2.25 |
| Usage total | $37.20 | $29.80 | $25.74 | $21.86 |
Each plan's included usage is consumed by these charges first: $5 on Free, $19 on Starter, $199 on Scale, $999 on Business. Rates checked 2026-09-09 via api.apify.com/v2/acts/<user>~<actor> and apify.com/pricing.
Always check the current rate on each Actor's own Pricing tab before you schedule it — event pricing changes — and run one manual execution (a smoke run) to catch surprises before the cron does.
Related: E-commerce price monitoring · Market intelligence · Social media analytics
Try Apify: free credits to test competitor scrapes →
Common datasets include pricing and availability, product catalog changes, reviews and ratings, marketing and SEO surface text, job postings, SERP positions for your keywords, and public social metrics. Apify’s Store provides Actors for many sites; custom sites can use Crawlee in your own Actor.
Hiring is a leading indicator: new engineering stacks, go-to-market motion, and geographies show up in reqs before press releases. Weekly snapshots of career pages make spikes and new role families easy to detect.
Publicly available factual information is often scrapable under prevailing interpretations in major jurisdictions, but site terms and computer abuse laws still apply. Collect only what you need, avoid circumventing technical barriers improperly, and read our legality overview.
Match cadence to volatility: pricing daily (hourly in promos), reviews and jobs weekly, SERP and marketing copy weekly to monthly. Apify schedules let you set different crons per Actor or task.
Yes. Use run webhooks plus Make, n8n, or Zapier to compare new datasets to baselines and post to Slack or email. You can also load datasets into a warehouse and let BI subscriptions fire when metrics cross thresholds.
Prefer browser-based Actors (Playwright or Puppeteer templates) or a site-specific Store Actor. Add proxies where the README recommends them. See web scraping challenges for mitigation patterns.
Off-the-shelf tools are quick to start but lock you into their schema, sources, and pricing. Scraping with Apify gives you raw, structured ownership of exactly the URLs and fields you choose, plus easy export to your own warehouse or BI. The trade-off is setup time and maintenance when target sites change layout. Many teams run both: a packaged tool for breadth, scraping for the few signals they need precise control over.
Start with 3 to 5 direct rivals and the specific URLs that matter (bestsellers, pricing page, careers, review profiles). This keeps runs cheap and the output interpretable. Add competitors once your alerts and dashboards are working, rather than scraping everything from day one.