Use PostgreSQL for scraped data: JSONB, constraints, and pgvector for embeddings. Learn ingestion patterns from Apify runs into durable tables.
PostgreSQL is a durable home for scraped data, with JSONB for flexible records, constraints for integrity, and pgvector for embeddings. These guides cover ingestion patterns from Apify runs into reliable tables.
Postgres handles both structured fields and semi-structured JSON well, which suits varied scrape output. Below you will find patterns for loading, deduplicating, and querying scraped data in PostgreSQL.
Supabase is an open-source Firebase alternative: PostgreSQL database, Auth, Storage, Realtime, and Edge Functions in one stack. Supabase Cloud starts at $25/mo for production — self-hosting on a Liquid Web VPS cuts costs and gives you data sovereignty and full control. This guide covers server requirements, installation, configuration, SSL, migration from cloud, and maintenance.
Store scraped data in PostgreSQL with a clean schema, bulk inserts via COPY or executemany, deduplication with ON CONFLICT, and full-text search. Use pgBouncer for connection pooling. Apify provides managed datasets if you prefer no DB ops.
Firecrawl returns clean Markdown, structured JSON, and rich metadata from any URL — but by default it stores nothing. Every crawl result exists only for the duration of your API response. If you want to search past crawls, track changes over time, or build a content pipeline, you need persistent storage.
Supabase provides a managed PostgreSQL instance with a REST API, real-time subscriptions, and a built-in vector extension (pgvector) — making it the most developer-friendly database target for Firecrawl output. This tutorial walks through the complete pipeline: schema design, automated ingestion, full-text search, and incremental re-crawl updates.
PostgreSQL is a natural fit for scraped data: JSONB columns store semi-structured records flexibly, full-text search indexes enable content querying, and time-series partitioning manages historical price data. Apify dataset exports load via COPY for bulk inserts. Postgis extensions add geospatial indexing for location-based scraped data.
Include source_url, scraped_at, content_hash, and raw_json columns on every table. Use materialized views for aggregated reports that would be slow to query from raw records. Add partial indexes on active-record flags to keep query performance stable as tables grow. Store PII in a separate, access-controlled schema.
Export dataset as JSONL, use the PostgreSQL COPY command with json_populate_record for structured inserts, or use pgloader for bulk loading from CSV exports. For streaming inserts during active runs, use webhooks to trigger a serverless function that upserts new items via the standard INSERT ON CONFLICT DO UPDATE pattern.
JSONB for schema-flexible raw records, pg_trgm for fuzzy text search on product names or URLs, TimescaleDB extension for time series price and metric data, PostGIS for geospatial queries on scraped location data, and pgvector for storing embeddings alongside scraped text in a RAG pipeline.
