Web Scraping for Social Media Analytics: Instagram, Twitter/X, TikTok, Reddit (2026)
This is a hands-on, per-platform scraping walkthrough for Instagram, Twitter/X, TikTok, and Reddit. For each network you get exactly what fields you can pull from public data, the platform-specific block risks and rate limits, and how Apify scrapers compare to the official APIs. For the full evergreen overview of social media analytics with Apify (use cases, the brand monitoring stack, and a normalized schema), see the social media analytics use case.
Social Media Scraping Use Cases
| Use Case | Data Needed | Best Platform |
|---|---|---|
| Brand monitoring | Mentions, sentiment, share of voice | Twitter/X, Reddit |
| Influencer research | Follower count, engagement rate, post frequency | Instagram, TikTok |
| Trend detection | Hashtag volume, viral content | TikTok, Twitter/X |
| Competitor social strategy | Post cadence, top content, audience reaction | All platforms |
| Crisis detection | Spike in negative sentiment | Twitter/X, Reddit |
| Market research | Product feedback, feature requests |
Platform-by-Platform Breakdown
Instagram
What you can scrape (public only): Profile metadata (username, bio, follower count, following), post URLs and captions, hashtag feeds, public Reels metadata, engagement counts (likes, comments). Use Apify Instagram Scraper for profile, hashtag, and post extraction.
What you cannot scrape: DMs, private account content, Stories (unless public and within short window), logged-in feed. Instagram aggressively blocks scrapers; use residential proxies and low concurrency.
Rate limits and ban risks: Expect 100–300 requests/day per IP before blocks. Rotate sessions. Avoid rapid-fire pagination. Run during off-peak hours to reduce detection.
Twitter/X
What you can scrape: Public tweets by keyword, user timeline, follower/following counts (when visible), tweet metadata (likes, retweets, replies). Apify Twitter/X Scraper fills gaps where API v2 limits apply.
API v2 limits: Free tier allows ~1,500 tweets/month. Pro tier is costly. Many research and monitoring use cases exceed API limits—scraping becomes necessary.
Rate limits: Twitter detects and blocks automated access. Residential proxies, realistic delays, and session rotation are essential. Expect higher failure rates than Instagram.
TikTok
What you can scrape: Public creator profiles, video metadata (views, likes, comments, shares), trending hashtag feeds, search results. Apify TikTok Scraper handles these.
Challenges: TikTok uses strong anti-bot measures. Headless browsers are often detected. Apify Actors use managed infrastructure and proxy pools to improve success rates.
Best practices: Target public profiles and hashtags. Avoid scraping logged-in content. Use geo-targeted proxies when testing region-specific trends.
Reddit
What you can scrape: Subreddit posts, comments, upvotes, author, timestamps. Apify Reddit Scraper is well-suited for sentiment analysis, trend tracking, and community monitoring.
API comparison: Reddit's API has free tiers but reduced access post-2023. Scraping public subreddits remains viable. Respect rate limits (Reddit is more lenient than Instagram/TikTok).
Use cases: Brand mentions, product feedback, competitor discussions, crisis monitoring. Reddit's threaded structure makes sentiment and topic extraction straightforward.
What You Can vs Cannot Scrape
| Data Type | Generally Scrapable | Generally Not |
|---|---|---|
| Public profiles | ✅ | — |
| Public posts/tweets/videos | ✅ | — |
| Hashtag feeds | ✅ | — |
| Comments (public) | ✅ | — |
| DMs, private messages | ❌ | Requires auth, ToS violation |
| Logged-in feed | ⚠️ | ToS risk, harder technically |
| Private accounts | ❌ | Not accessible |
| Stories (ephemeral) | ⚠️ | Short window, ToS gray area |
Apify Scrapers vs Official APIs
| Platform | Apify Scraper | Official API | Best For |
|---|---|---|---|
| Profile, hashtag, posts, engagement | Graph API (limited, business accounts) | Broad public data, research | |
| Twitter/X | Tweets by keyword/user, metadata | API v2 (1.5K–10K tweets/mo free) | Scale beyond API limits |
| TikTok | Creator profiles, hashtags, video metadata | Research API (restricted) | Trend detection, influencer research |
| Subreddits, posts, comments | Reddit API (reduced free tier) | Sentiment, community monitoring |
Cost: Apify charges per compute and proxy usage. For moderate volume (1K–10K items/day), often cheaper than API Pro tiers. For very high volume, evaluate API enterprise pricing.
Freshness: Scrapers return live data at request time. APIs may have slight delay. Both are suitable for analytics.
Data fields: APIs sometimes expose more structured fields (e.g., verified status). Scrapers infer from rendered HTML. Validate critical fields.
Data Use Cases: Sentiment, Brand Tracking, Benchmarking
Sentiment analysis: Run Reddit or Twitter data through an LLM or sentiment model. LangChain Apify content pipeline shows how to scrape → chunk → embed → analyze.
Brand tracking: Scrape mentions by keyword across platforms. Deduplicate, score sentiment, alert on spikes. Combine with Apify for SEO for cross-channel visibility.
Competitive benchmarking: Extract competitor post frequency, top-performing content, engagement rates. Compare against your own metrics. Use for content strategy.
Rate Limits and Session Management
- Throttling: Add 2–5 second delays between requests. Burst traffic triggers blocks faster.
- Session persistence: For multi-page flows (e.g., scrolling a feed), use sticky sessions so the same IP handles the full sequence.
- Proxy rotation: Rotate on 429/403. Use residential pools for Instagram, TikTok. Datacenter IPs work for Reddit in many cases.
- Backoff: On block, exponential backoff (1m, 5m, 15m) before retry. Avoid hammering the same endpoint.
Reddit is the most scraping-friendly major platform. Lower block rates, clear structure, excellent for sentiment and trend analysis. Use as a proof-of-concept before tackling Instagram or TikTok.
Scraping public data is generally legal in many jurisdictions (US hiQ v. LinkedIn). Instagram's ToS prohibits it. Enforcement targets large-scale commercial resellers. For internal analytics, use reasonable limits and residential proxies.
Apify Twitter/X Scraper covers tweets by keyword, user, and search. API v2 has strict limits; scrapers fill the gap. Use residential proxies and expect blocks at scale.
TikTok is heavily protected. Use Apify TikTok Scraper with residential proxies. Limit concurrency. Stick to public profiles and hashtags. Some blocks are unavoidable—design for retry and fallback.
Pipe Reddit or Twitter data into a sentiment model (e.g., Hugging Face, or LLM via LangChain). The LangChain Apify content pipeline shows scrape → transform → analyze workflows.
Use APIs when volume fits free/Pro tiers and you need compliance. Use Apify scrapers when you exceed API limits, need data APIs don't offer, or want a unified multi-platform pipeline.