Store scraped rows with schemas, dedupe, and upserts from crawlers to SQL or NoSQL. Apify outputs structured JSON you can load into your database.
Storing scraped rows well means schemas, deduplication, and upserts that move data cleanly from crawler to SQL or NoSQL. These guides cover loading Apify output into databases without duplicates or drift.
Apify returns structured JSON you can map straight into tables or collections. Below you will find patterns for schema design, dedupe keys, and incremental loads from scrapes into your database.
Supabase is an open-source Firebase alternative: PostgreSQL database, Auth, Storage, Realtime, and Edge Functions in one stack. Supabase Cloud starts at $25/mo for production — self-hosting on a Liquid Web VPS cuts costs and gives you data sovereignty and full control. This guide covers server requirements, installation, configuration, SSL, migration from cloud, and maintenance.
Store scraped data in PostgreSQL with a clean schema, bulk inserts via COPY or executemany, deduplication with ON CONFLICT, and full-text search. Use pgBouncer for connection pooling. Apify provides managed datasets if you prefer no DB ops.
PostgreSQL for structured, queryable data with good indexing. MongoDB for flexible schemas that evolve as target site structures change. TimescaleDB for time series price or metric data. ClickHouse for analytical queries on large volumes. Choice depends on query patterns: OLTP versus analytical access shapes the decision.
Include source_url, scraped_at, and a content_hash on every record for auditing and deduplication. Use nullable columns for fields that may be absent. Keep raw text in a separate column alongside parsed fields so you can re-parse without re-scraping. Version schema changes with migrations rather than silent column additions.
Schedule Apify actor runs at appropriate frequencies per data volatility, compare content_hash before writing to skip unchanged records, and set updated_at timestamps. Archive old records rather than deleting for auditability. Add indices on fields used in WHERE clauses to keep query performance stable as table sizes grow.
Use the Apify dataset API to paginate JSON results and INSERT/UPSERT into your database. Webhook notifications trigger a serverless function or n8n workflow when runs complete. For high-volume flows, export to S3 as JSONL and use COPY INTO or stream loader tools for bulk inserts rather than row-by-row API calls.
