Cron Jobs for Web Scraping: Schedule Scrapers on Linux 2026
Schedule web scrapers with cron: use the 5-field crontab format, set environment variables correctly, and log output. Prefer systemd timers for modern Linux. For managed scheduling with no server, use Apify.
Crontab Syntax (5 Fields)
┌───────────── minute (0-59)
│ ┌───────────── hour (0-23)
│ │ ┌───────────── day of month (1-31)
│ │ │ ┌───────────── month (1-12)
│ │ │ │ ┌───────────── day of week (0-7, 0 and 7 = Sunday)
│ │ │ │ │
* * * * * command
Special characters: * (any), , (list), - (range), */n (every n units).
Practical Cron Examples for Scrapers
# Edit crontab
crontab -e
# Every day at 2:00 AM
0 2 * * * /opt/scrapers/daily-price-check.sh >> /var/log/price-scraper.log 2>&1
# Every 6 hours
0 */6 * * * /opt/scrapers/hourly-feed.sh >> /var/log/feed-scraper.log 2>&1
# Every Monday at 3:00 AM (weekly full crawl)
0 3 * * 1 /opt/scrapers/weekly-full-crawl.sh >> /var/log/full-crawl.log 2>&1
# Every 15 minutes (aggressive — use rate limiting)
*/15 * * * * /opt/scrapers/quick-check.sh >> /var/log/quick-check.log 2>&1
Environment Variables in Cron
Cron runs with a minimal environment. Set PATH and load .env inside your script.
# Crontab: set SHELL and PATH
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin
MAILTO=ops@example.com
0 2 * * * /opt/scrapers/run.sh >> /var/log/scraper.log 2>&1
# run.sh — load env before execution
#!/bin/bash
set -e
source /opt/scrapers/.env
cd /opt/scrapers
node scrape.js
Logging Cron Output
Always capture stdout and stderr. Rotate logs to avoid disk fill.
# Redirect both streams
0 2 * * * /opt/scrapers/run.sh >> /var/log/scraper.log 2>&1
# Or split
0 2 * * * /opt/scrapers/run.sh > /var/log/scraper.log 2> /var/log/scraper.err
Add logrotate:
# /etc/logrotate.d/scraper
/var/log/scraper*.log {
daily
rotate 7
compress
missingok
}
Preventing Overlapping Runs
Use flock to ensure only one instance runs at a time.
# In crontab
0 2 * * * flock -n /tmp/daily-scraper.lock /opt/scrapers/daily.sh >> /var/log/scraper.log 2>&1
-n = non-blocking; if lock is held, cron skips this run.
systemd Timers as Modern Alternative
systemd timers offer better logging, dependency management, and random delays. Use them on systemd-based Linux.
# /etc/systemd/system/scraper-daily.service
[Unit]
Description=Daily price scraper
[Service]
Type=oneshot
User=scraper
WorkingDirectory=/opt/scrapers
ExecStart=/opt/scrapers/daily.sh
# /etc/systemd/system/scraper-daily.timer
[Unit]
Description=Run scraper daily at 2 AM
[Timer]
OnCalendar=*-*-* 02:00:00
RandomizedDelaySec=300
Persistent=true
[Install]
WantedBy=timers.target
sudo systemctl enable scraper-daily.timer
sudo systemctl start scraper-daily.timer
Common Cron Mistakes
| Mistake | Fix |
|---|---|
| No PATH | Set PATH in crontab or script |
| No stderr capture | Use 2>&1 |
| Overlapping runs | Use flock or systemd |
| Wrong user | Run crontab -u scraper -e for service user |
| Dependencies not loaded | Source .env or set vars in crontab |
| No idempotency | Design scrapers to be safely re-runnable |
Apify Scheduling as Managed Alternative
Apify provides built-in scheduling. No cron, no server. Set a schedule in the Actor or via API. Runs execute in the cloud with proxy rotation and built-in storage.
// Apify API: create scheduled run
await apifyClient.schedules().getOrCreate('my-daily-scraper', {
crontab: '0 2 * * *', // 2 AM daily
action: 'RUN',
actorId: 'your-actor-id',
});
Use flock to prevent overlap. Log everything. Prefer systemd timers on modern Linux. For zero server ops, use Apify scheduling.
Use crontab -e and add a line like '0 2 * * * /opt/scrapers/run.sh >> /var/log/scraper.log 2>&1'. Or use systemd timers for better logging.
crontab is the Linux scheduler. Five fields: minute, hour, day-of-month, month, day-of-week. Asterisks mean 'every'.
Depends on data freshness needs. Daily for most use cases. Hourly for feeds. Every 15 min only if rate limits and target ToS allow.
Cron runs with minimal env. Set PATH, source .env in the script, and capture stderr with 2>&1 to debug.