Firecrawl + Next.js: Build a Content Aggregator 2026

Contains affiliate links. Disclosure.

A Next.js content aggregator backed by Firecrawl: ingestion worker scrapes sources, an API route validates and stores records, and the UI renders a feed with citations. Keep scraping in async jobs, not in the render path, to avoid timeouts and failures affecting users.

Get Firecrawl →

Architecture

1. Ingestion — Cron or queue calls Firecrawl to scrape source URLs
2. API route — Validates, normalizes, stores (DB or file)
3. Feed API — Returns paginated articles
4. UI — Server component renders feed with citations

Project Structure

app/
  api/ingest/route.ts
  api/feed/route.ts
  dashboard/page.tsx
lib/
  firecrawl.ts
  normalize.ts
  storage.ts

Firecrawl Client

// lib/firecrawl.ts
import Firecrawl from '@mendable/firecrawl-js';

export const firecrawl = new Firecrawl({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

Ingestion API Route

// app/api/ingest/route.ts
import { NextResponse } from 'next/server';
import { firecrawl } from '@/lib/firecrawl';

export async function POST(req: Request) {
  const { url } = await req.json();
  if (!url) return NextResponse.json({ error: 'URL required' }, { status: 400 });

  const data = await firecrawl.scrapeUrl(url, {
    formats: ['markdown', 'links'],
  });

  const normalized = {
    url,
    title: data.metadata?.title ?? '',
    markdown: data.markdown ?? '',
    scrapedAt: new Date().toISOString(),
  };
  // Persist to your storage (e.g. database)
  await saveArticle(normalized);

  return NextResponse.json({ ok: true, url });
}

Feed API Route

// app/api/feed/route.ts
export async function GET(req: Request) {
  const { searchParams } = new URL(req.url);
  const page = Number(searchParams.get('page')) || 1;
  const limit = 20;
  const articles = await getArticles({ page, limit });
  return NextResponse.json(articles);
}

Dashboard Page (Server Component)

// app/dashboard/page.tsx
async function Feed() {
  const res = await fetch(`${process.env.NEXT_PUBLIC_BASE_URL}/api/feed`);
  const { articles } = await res.json();

  return (
    <div>
      {articles.map((a) => (
        <article key={a.url}>
          <h2>{a.title}</h2>
          <a href={a.url} target="_blank" rel="noopener">Source</a>
        </article>
      ))}
    </div>
  );
}

Best Practices

• Use queue or cron for ingestion, not user-triggered requests
• Cache feed responses and revalidate on a schedule
• Store canonical URL, published date, source domain
• Keep failed ingestion records for retry and diagnostics

Next step

Add RAG retrieval to make the aggregator searchable with natural language.

Frequently Asked Questions

Yes. Use Firecrawl in API routes or background jobs. Expose normalized results to the frontend via API.

No. Keep scraping in async ingestion jobs to avoid timeouts and failure propagation to users.

Canonical deduplication, source scoring, visible citations, and freshness metadata in every record.

Use @mendable/firecrawl-js. Methods: scrape, crawl, map (check SDK docs for exact names).