Redis Queue for Distributed Web Scraping: Bull, Celery 2026

Contains affiliate links. Disclosure.

Use Redis as a job queue for distributed scraping: LPUSH/BRPOP, Bull/BullMQ (Node.js), or Celery (Python). Deduplicate URLs with Redis SET. Implement rate limiting with a token bucket. For managed scaling without queues, use Apify.

Redis as Job Queue: LPUSH/BRPOP

Simple producer-consumer pattern. Producers push URLs; workers block on BRPOP.

# producer.py
import redis
r = redis.Redis(host='localhost', port=6379, db=0)

urls = ['https://example.com/1', 'https://example.com/2']
for url in urls:
    r.lpush('scraper:urls', url)

# worker.py
import redis
import requests

r = redis.Redis(host='localhost', port=6379, db=0)
while True:
    _, url = r.brpop('scraper:urls', timeout=30)
    if url:
        url = url.decode()
        # scrape(url)

URL Deduplication with Redis SET

Avoid re-scraping the same URL across runs.

def push_url_if_new(r, url):
    if r.sadd('scraper:seen', url):
        r.lpush('scraper:urls', url)
        return True
    return False

Use a hash for very long URLs to save memory:

import hashlib
def push_url_if_new(r, url):
    key = hashlib.sha256(url.encode()).hexdigest()
    if r.sadd('scraper:seen', key):
        r.lpush('scraper:urls', url)
        return True
    return False

Celery (Python) with Redis

Celery provides retries, scheduling, and result backends.

# tasks.py
from celery import Celery

app = Celery('scraper', broker='redis://localhost:6379/0', backend='redis://localhost:6379/1')

@app.task(bind=True, max_retries=3)
def scrape_url(self, url):
    try:
        # ... fetch and parse
        return {'url': url, 'title': title}
    except Exception as e:
        self.retry(exc=e, countdown=60)

# Start worker
celery -A tasks worker --loglevel=info --concurrency=4

# Enqueue
scrape_url.delay('https://example.com/page')

BullMQ (Node.js)

BullMQ supports priorities, delays, and rate limiting.

const { Queue, Worker } = require('bullmq');

const queue = new Queue('scraper', {
  connection: { host: 'localhost', port: 6379 },
});

// Producer
await queue.add('scrape', { url: 'https://example.com' }, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 5000 },
});

// Worker
const worker = new Worker('scraper', async (job) => {
  const { url } = job.data;
  // ... scrape
  return result;
}, { connection: { host: 'localhost', port: 6379 }, concurrency: 4 });

Rate Limiting with Token Bucket

Limit requests per second across workers. Use a Lua script for atomicity.

-- token_bucket.lua
local key = KEYS[1]
local rate = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = 1

local bucket = redis.call('hmget', key, 'tokens', 'last')
local tokens = tonumber(bucket[1]) or capacity
local last = tonumber(bucket[2]) or now

tokens = math.min(capacity, tokens + (now - last) * rate)
if tokens >= requested then
  tokens = tokens - requested
  redis.call('hset', key, 'tokens', tokens, 'last', now)
  redis.call('expire', key, 60)
  return 1
else
  return 0
end

Call with EVAL; return 1 if allowed, 0 if rate limited.

Distributed Architecture

[Producers]  →  Redis (queue + dedup)  →  [Workers]
                     ↑                          ↓
                     └────── [PostgreSQL] ←──────┘

Multiple workers on multiple machines share the same Redis. See PostgreSQL storage for writing results.

Next step

Start with LPUSH/BRPOP and Redis SET for dedup. Move to Celery or BullMQ for retries and scheduling. For managed scaling, use Apify.

Frequently Asked Questions

Use a Redis queue. Multiple workers consume URLs. Deduplicate with Redis SET. Add more workers as needed.

Multiple worker processes/machines share a central URL queue. Each worker pulls URLs, scrapes, pushes new URLs back. No single point of failure.

Use Redis LIST for the queue (LPUSH/BRPOP). Use Redis SET for URL deduplication. Consider Celery or BullMQ for retries and scheduling.