Scraping: guides & tutorials

Describe your scraping target to Claude and get Python Apify Actor code. Deploy with apify push. Claude does not run the scraper for you: it generates code you review, test, and deploy. New to Claude? You can try Claude free for a week to draft and iterate on Actor code before you commit. For pre-built targets, check the Apify Store first. Claude shines when you need a custom target: an internal portal, a niche marketplace, or a site with a unique layout. The Store covers popular sites (Indeed, Amazon, LinkedIn); Claude fills the gaps. Expect 15–30 minutes from prompt to first successful run for a simple target.

Backup scraped data, PostgreSQL databases, configs, and actor code. Use the 3-2-1 rule: three copies, two media types, one offsite. This guide covers rsync, pg_dump, S3 lifecycle, and recovery testing.

Schedule web scrapers with cron: use the 5-field crontab format, set environment variables correctly, and log output. Prefer systemd timers for modern Linux. For managed scheduling with no server, use Apify.

Monitor self-hosted scrapers with Prometheus and Grafana. Track success rate, pages per minute, error rate, latency, and data volume. This guide uses a Docker Compose stack. Grafana Cloud is a managed alternative.

Use Nginx as a reverse proxy for your scraping API: SSL termination, rate limiting, load balancing. Put Nginx in front of FastAPI, Flask, or Node.js scrapers. For managed APIs with no server setup, Apify exposes scrapers via REST API.

Store scraped data in PostgreSQL with a clean schema, bulk inserts via COPY or executemany, deduplication with ON CONFLICT, and full-text search. Use pgBouncer for connection pooling. Apify provides managed datasets if you prefer no DB ops.

Use Redis as a job queue for distributed scraping: LPUSH/BRPOP, Bull/BullMQ (Node.js), or Celery (Python). Deduplicate URLs with Redis SET. Implement rate limiting with a token bucket. For managed scaling without queues, use Apify.

Secure scraping API servers with Let's Encrypt and Certbot. Use Nginx with TLS 1.2/1.3 only, HSTS, and strong ciphers. Auto-renew certificates with a systemd timer. For managed APIs with built-in SSL, use Apify.

Instagram is a heavily protected ecosystem. Teams still need public metrics like follower counts, reel velocity, and comment signals for influencer vetting, but automated collection is actively restricted.

Running an Instagram extraction workflow in 2026 means understanding SPA behavior, GraphQL/mobile endpoints, and IP reputation controls.

This guide outlines common failure modes in basic scripts and shows a more durable path using the Apify Platform.