Firecrawl + Supabase: Store Crawled Data in PostgreSQL

Firecrawl returns clean Markdown, structured JSON, and rich metadata from any URL — but by default it stores nothing. Every crawl result exists only for the duration of your API response. If you want to search past crawls, track changes over time, or build a content pipeline, you need persistent storage.

Supabase provides a managed PostgreSQL instance with a REST API, real-time subscriptions, and a built-in vector extension (pgvector) — making it the most developer-friendly database target for Firecrawl output. This tutorial walks through the complete pipeline: schema design, automated ingestion, full-text search, and incremental re-crawl updates.

Contains affiliate links. Disclosure.

What you will build

By the end of this tutorial you will have:

• A Supabase PostgreSQL table that stores URLs, titles, Markdown content, metadata, and crawl timestamps
• A Node.js ingestion script that calls the Firecrawl API and writes results directly to Supabase
• A webhook handler that auto-ingests Firecrawl batch crawl results as they arrive
• Full-text search using PostgreSQL tsvector to query your content database
• An incremental update strategy that de-duplicates re-crawls by URL

---

Prerequisites

Requirement	Details
Firecrawl API key	Free tier: 500 credits/month
Supabase project	Free tier: 500 MB storage, 2 GB bandwidth/month
Node.js ≥ 18	For the ingestion scripts
@supabase/supabase-js v2	Supabase client library
@mendable/firecrawl-js	Official Firecrawl JS SDK

---

Step 1 — Create a Supabase project

1. Go to supabase.com and create a new project.
2. Select a region close to your infrastructure.
3. From Project Settings → API, copy your Project URL and anon/service_role key.

Store both in a .env file:

SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_KEY=your-service-role-key
FIRECRAWL_API_KEY=fc-your-api-key

Use the service role key (not the anon key) for server-side ingestion scripts. It bypasses Row Level Security and is safe for backend-only code. Never expose it in a browser.

---

Step 2 — Design the storage schema

Open the Supabase SQL Editor and run the following migration:

-- Enable the pg_trgm extension for full-text search helpers
create extension if not exists pg_trgm;

-- Main content table
create table if not exists crawled_pages (
  id            bigint generated always as identity primary key,
  url           text not null,
  title         text,
  markdown      text,
  html          text,
  metadata      jsonb,
  crawl_job_id  text,
  crawled_at    timestamptz not null default now(),
  -- Deduplicate by URL: one canonical row per URL
  constraint crawled_pages_url_unique unique (url)
);

-- Full-text search: index over title + markdown
alter table crawled_pages
  add column if not exists fts tsvector
  generated always as (
    to_tsvector('english', coalesce(title, '') || ' ' || coalesce(markdown, ''))
  ) stored;

create index if not exists crawled_pages_fts_idx
  on crawled_pages using gin (fts);

-- Standard indexes
create index if not exists crawled_pages_url_idx on crawled_pages (url);
create index if not exists crawled_pages_crawled_at_idx on crawled_pages (crawled_at desc);
create index if not exists crawled_pages_crawl_job_idx on crawled_pages (crawl_job_id);

Schema column reference

Column	Type	Purpose
url	text UNIQUE	Canonical URL — deduplication key
title	text	Page title extracted by Firecrawl
markdown	text	LLM-ready Markdown content
html	text	Raw HTML (optional; omit to save space)
metadata	jsonb	Full Firecrawl metadata object (language, og tags, etc.)
crawl_job_id	text	Links rows back to a specific crawl run
crawled_at	timestamptz	Timestamp of the crawl — use for incremental updates
fts	tsvector	Generated column for full-text search

---

Step 3 — Install dependencies

mkdir firecrawl-supabase && cd firecrawl-supabase
npm init -y
npm install @mendable/firecrawl-js @supabase/supabase-js dotenv

---

Step 4 — Single-URL scrape and store

Create scrape-and-store.ts:

import FirecrawlApp from '@mendable/firecrawl-js';
import { createClient } from '@supabase/supabase-js';
import 'dotenv/config';

const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });
const supabase = createClient(
  process.env.SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_KEY!,
);

async function scrapeAndStore(url: string): Promise<void> {
  console.log(`Scraping: ${url}`);

  const result = await firecrawl.scrapeUrl(url, {
    formats: ['markdown', 'html'],
  });

  if (!result.success) {
    throw new Error(`Firecrawl error: ${result.error}`);
  }

  const { error } = await supabase
    .from('crawled_pages')
    .upsert(
      {
        url,
        title: result.metadata?.title ?? null,
        markdown: result.markdown ?? null,
        html: result.html ?? null,
        metadata: result.metadata ?? null,
        crawled_at: new Date().toISOString(),
      },
      { onConflict: 'url' }, // Update existing row if URL already exists
    );

  if (error) throw error;

  console.log(`Stored: ${url}`);
}

// Example: scrape a single page
await scrapeAndStore('https://docs.firecrawl.dev/introduction');

The upsert with onConflict: 'url' is the core of the incremental update strategy: re-crawling a URL overwrites the existing row with fresh content rather than creating a duplicate.

---

Step 5 — Batch crawl an entire site

For multi-page crawls, use the Firecrawl /crawl endpoint with a webhook so Supabase receives pages as they arrive rather than waiting for the entire job to complete.

5a — Start a crawl job

import FirecrawlApp from '@mendable/firecrawl-js';
import 'dotenv/config';

const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });

const job = await firecrawl.asyncCrawlUrl('https://docs.firecrawl.dev', {
  limit: 100,
  scrapeOptions: { formats: ['markdown'] },
  webhook: {
    url: 'https://your-api.example.com/webhooks/firecrawl',
    events: ['page.scraped', 'crawl.completed', 'crawl.failed'],
  },
});

console.log('Crawl job started:', job.id);

5b — Handle the webhook

Create a minimal webhook handler (Express or any HTTP framework):

import express from 'express';
import { createClient } from '@supabase/supabase-js';
import 'dotenv/config';

const app = express();
app.use(express.json({ limit: '10mb' }));

const supabase = createClient(
  process.env.SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_KEY!,
);

app.post('/webhooks/firecrawl', async (req, res) => {
  const event = req.body;

  // Acknowledge immediately — Firecrawl expects a fast 200
  res.sendStatus(200);

  if (event.type === 'page.scraped' && event.data) {
    const page = event.data;

    await supabase.from('crawled_pages').upsert(
      {
        url: page.metadata?.sourceURL ?? page.url,
        title: page.metadata?.title ?? null,
        markdown: page.markdown ?? null,
        metadata: page.metadata ?? null,
        crawl_job_id: event.jobId,
        crawled_at: new Date().toISOString(),
      },
      { onConflict: 'url' },
    );
  }
});

app.listen(3000, () => console.log('Webhook handler running on port 3000'));

Tip: Expose the webhook locally during development using ngrok or a Cloudflare Tunnel. In production, deploy this handler as a serverless function (Vercel, Cloudflare Workers, AWS Lambda).

---

Step 6 — Full-text search

The fts generated column (built in Step 2) enables fast PostgreSQL full-text search over all stored content without any additional tooling.

Query from Supabase JavaScript client

async function searchContent(query: string, limit = 10) {
  const { data, error } = await supabase
    .from('crawled_pages')
    .select('id, url, title, crawled_at')
    .textSearch('fts', query, {
      type: 'websearch', // Parses natural language: "firecrawl AND webhook"
      config: 'english',
    })
    .order('crawled_at', { ascending: false })
    .limit(limit);

  if (error) throw error;
  return data;
}

const results = await searchContent('webhook ingestion postgresql');
console.log(results);

Direct SQL for advanced ranked search

For relevance ranking, drop down to raw SQL using ts_rank:

select
  url,
  title,
  crawled_at,
  ts_rank(fts, websearch_to_tsquery('english', 'webhook ingestion')) as rank
from crawled_pages
where fts @@ websearch_to_tsquery('english', 'webhook ingestion')
order by rank desc
limit 10;

This returns results ordered by relevance score — critical when your database grows beyond a few hundred pages.

---

Step 7 — Incremental re-crawl strategy

Naive re-crawls duplicate data. The pattern below fetches only pages not crawled in the last 7 days:

async function incrementalRecrawl(domain: string): Promise<void> {
  // Find stale pages (not crawled in 7 days)
  const cutoff = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString();

  const { data: stalePages } = await supabase
    .from('crawled_pages')
    .select('url')
    .like('url', `%${domain}%`)
    .lt('crawled_at', cutoff);

  if (!stalePages || stalePages.length === 0) {
    console.log('All pages are fresh.');
    return;
  }

  console.log(`Re-crawling ${stalePages.length} stale pages...`);

  for (const { url } of stalePages) {
    await scrapeAndStore(url); // from Step 4 — upserts automatically
    // Add a short delay to stay within Firecrawl rate limits
    await new Promise((r) => setTimeout(r, 200));
  }
}

Strategy	When to use
upsert on every crawl	Simple pipelines with low volume
Query stale rows, recrawl selectively	Large databases, cost-sensitive operations
Crawl job with webhook + job ID filter	Full-site refreshes on a schedule

---

Building a content search API

Combine the Supabase client with a minimal HTTP handler to expose your crawled content as a searchable API:

// GET /api/search?q=postgresql+webhook&limit=5
app.get('/api/search', async (req, res) => {
  const q = String(req.query.q ?? '');
  const limit = Math.min(Number(req.query.limit ?? 10), 50);

  if (!q) return res.status(400).json({ error: 'Missing query parameter: q' });

  const { data, error } = await supabase
    .from('crawled_pages')
    .select('id, url, title, crawled_at')
    .textSearch('fts', q, { type: 'websearch', config: 'english' })
    .order('crawled_at', { ascending: false })
    .limit(limit);

  if (error) return res.status(500).json({ error: error.message });

  return res.json({ results: data, count: data.length });
});

This is the minimal foundation for a documentation search engine, a competitive intelligence feed, or a RAG knowledge base. To go further with vector search (semantic similarity rather than keyword matching), see the RAG pipeline guide.

---

Supabase vs. other PostgreSQL options for Firecrawl storage

Option	Managed	Free Tier	pgvector	Best for
Supabase	✅ Yes	500 MB / 2 GB bandwidth	✅ Built-in	Developer projects, fast setup
Neon	✅ Yes	0.5 GB storage	✅ Built-in	Serverless, branching workflows
Railway	✅ Yes	$5 credit/month	Manual install	General-purpose apps
Self-hosted PostgreSQL	❌ No	Hardware cost only	Manual install	Full control, high volume
Amazon RDS	✅ Yes	12 months free (t3.micro)	Requires extension	Enterprise production

Supabase is the right choice if you want the fastest path from Firecrawl output to a queryable database. If you need branching for testing schema migrations or a purely serverless billing model, Neon is the stronger alternative.

---

FAQ

How do I store Firecrawl results in a database?

Use the Firecrawl JavaScript or Python SDK to call /scrape or /crawl, then write the response directly to a PostgreSQL table using an upsert on the URL column. The complete schema and ingestion script are in Steps 2–4 above.

Can I use PostgreSQL with Firecrawl?

Yes. Firecrawl does not have a native database integration, but its REST API returns standard JSON you can persist to any PostgreSQL instance. Supabase provides the simplest managed PostgreSQL option with a JavaScript client that works well with the Firecrawl JS SDK.

What database works best with Firecrawl?

PostgreSQL (via Supabase or Neon) works well for structured storage and full-text search. If your primary use case is semantic similarity search for RAG pipelines, add pgvector to PostgreSQL or use a dedicated vector database like Pinecone or Qdrant alongside your relational store.

How do I avoid storing duplicate pages?

Add a UNIQUE constraint on the url column and use upsert with onConflict: 'url' in every write operation. This overwrites stale rows instead of inserting duplicates on every re-crawl.

Does Firecrawl have a built-in storage option?

Firecrawl stores crawl job results temporarily in its API (accessible via GET /crawl/{jobId}) for a short window. For persistent, queryable storage you must write results to your own database.

How do I set up full-text search on crawled content?

Add a tsvector generated column over your title and markdown fields (see the SQL in Step 2) and create a GIN index on it. Then use textSearch from the Supabase client or @@ with websearch_to_tsquery in raw SQL.

---

Next steps

You now have a fully operational Firecrawl → Supabase ingestion pipeline. From here:

• Add pgvector: Install the pgvector Supabase extension and generate embeddings from your markdown column to enable semantic search alongside keyword search.
• Schedule re-crawls: Use Supabase Edge Functions with a pg_cron job to run the incremental re-crawl function on a nightly schedule.
• Build a RAG pipeline: Feed your crawled_pages table into a vector database and connect it to an LLM — see the RAG pipeline guide for architecture details.
• Try Firecrawl: Start a free Firecrawl account — 500 credits/month at no cost.