Backup and Recovery for Scraping Infrastructure 2026

Contains affiliate links. Disclosure.

Backup scraped data, PostgreSQL databases, configs, and actor code. Use the 3-2-1 rule: three copies, two media types, one offsite. This guide covers rsync, pg_dump, S3 lifecycle, and recovery testing.

For teams that prefer managed storage and zero backup ops, Apify keeps datasets, runs, and actor code in the cloud with built-in versioning.

What to Back Up

Component	Priority	Tool	Retention
Scraped data (PostgreSQL)	Critical	pg_dump	7–30 days
Actor/scraper code	Critical	Git + rsync	Unlimited
Config files	High	rsync	30 days
Redis state (if used)	Medium	RDB/AOF	7 days
Proxy creds / API keys	High	Encrypted vault	90 days

Never store secrets in plain text. Use environment variables or a secrets manager.

3-2-1 Backup Rule

• Three copies: primary data + two backups.
• Two media types: e.g. local disk + S3 or another cloud.
• One offsite: one copy outside your server/datacenter.

Database Backup: PostgreSQL

Use pg_dump for logical backups. Custom format (-Fc) supports parallel restore and compression.

# Single database, custom format
pg_dump -Fc -U postgres -d scraping_db -f /backup/scraping_db_$(date +%Y%m%d).dump

# All databases
pg_dumpall -U postgres > /backup/all_$(date +%Y%m%d).sql

Restore:

pg_restore -U postgres -d scraping_db -v /backup/scraping_db_20260317.dump

File Backup: rsync

Back up code, configs, and logs to a remote host or S3-compatible storage.

# rsync to remote (exclude node_modules, __pycache__)
rsync -avz --delete \
  --exclude='node_modules' \
  --exclude='__pycache__' \
  --exclude='.env' \
  /opt/scrapers/ backup-server:/backups/scrapers/

# rsync to local backup partition
rsync -avz /opt/scrapers/ /mnt/backup/scrapers/

S3 Lifecycle for Scraped Exports

If you export scraped data to S3 or compatible storage, use lifecycle rules to tier or expire old exports.

# Example lifecycle rule (AWS S3 / MinIO)
Rules:
  - ID: expire-old-exports
    Status: Enabled
    Filter:
      Prefix: scraped-exports/
    Expiration:
      Days: 90
    Transitions:
      - Days: 30
        StorageClass: STANDARD_IA

Backup Schedule

Frequency	What	Where
Daily	pg_dump	Local + offsite
Daily	rsync code/config	Remote server
Weekly	Full snapshot	Optional VM snapshot
Monthly	Archive to cold storage	S3 Glacier / cold tier

Cron example:

# /etc/cron.d/scraping-backup
0 2 * * * postgres /usr/local/bin/backup-pg.sh >> /var/log/backup.log 2>&1
0 3 * * * scraper rsync -avz /opt/scrapers backup-server:/backups/ >> /var/log/backup.log 2>&1

Recovery Testing

Test restores at least quarterly. Document RTO (recovery time objective) and RPO (recovery point objective).

# Test PostgreSQL restore (use a temp DB)
createdb -U postgres test_restore
pg_restore -U postgres -d test_restore -v /backup/scraping_db_latest.dump
# Verify row counts, run smoke tests
dropdb -U postgres test_restore

Monitoring Backup Health

Track backup success/failure and age. Use Prometheus or a simple script:

# Alert if latest backup is older than 25 hours
MAX_AGE=90000  # seconds
FILE="/backup/scraping_db_latest.dump"
if [ -f "$FILE" ]; then
  AGE=$(($(date +%s) - $(stat -c %Y "$FILE")))
  [ $AGE -gt $MAX_AGE ] && echo "Backup stale: ${AGE}s old"
fi

See monitoring scrapers with Grafana for full observability.

Next step

Implement daily pg_dump and rsync this week. Add recovery testing to your quarterly runbook. For managed scraping with built-in storage, use Apify.

Frequently Asked Questions

Use pg_dump for PostgreSQL. Export to S3 or rsync to a remote server. Follow the 3-2-1 rule: three copies, two media types, one offsite.

Restore PostgreSQL from pg_dump, redeploy code from Git or rsync backup. Use VM snapshots for fast recovery if your host (e.g. Liquid Web) supports them.

Daily for databases and code. Weekly full snapshots if available. Test restores at least quarterly.

Apify stores datasets and runs in the cloud. You still should version actor code in Git and export critical datasets periodically for compliance or migration.