Skip to main content

Build a Price Monitoring System on Liquid Web

· 3 min read
Yassine El Haddad
Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

A self-hosted price monitoring system on Liquid Web needs a scraper, storage, a scheduler, and alerting. This guide outlines the architecture and implementation options.

For managed price monitoring without running servers, Apify offers Actors for Amazon, Google Shopping, and e-commerce sites.

System Architecture​

ComponentRoleOptions
ScraperFetch prices from target sitesPlaywright, Apify Actor, Firecrawl
StoragePersist historical pricesPostgreSQL, SQLite
QueueOptional job queueRedis, Bull/BullMQ
SchedulerRun scraper on intervalcron, node-cron
AlertingNotify on price dropsEmail, Slack, webhook
cron (hourly) → Scraper → PostgreSQL
↓
Redis (optional queue)
↓
Alert if price < threshold → Email/Slack
VolumeRAMStorageLiquid Web Plan
10–50 products4 GB80 GB$8.50/mo
100–500 products8 GB240 GB$22.50/mo
1000+ products16 GB440 GB$45/mo

See Liquid Web VPS for current plans.

Step 1: Database Schema​

CREATE TABLE products (
id SERIAL PRIMARY KEY,
url TEXT UNIQUE NOT NULL,
name TEXT,
target_price DECIMAL(10,2),
created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE price_history (
id SERIAL PRIMARY KEY,
product_id INT REFERENCES products(id),
price DECIMAL(10,2) NOT NULL,
scraped_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_price_history_product ON price_history(product_id);
CREATE INDEX idx_price_history_scraped ON price_history(scraped_at);

Step 2: Scraper Options​

Option A: Playwright (self-hosted)

// Minimal price extraction
const { chromium } = require('playwright');
const price = await page.locator('.price').textContent();

Deploy via Docker or bare VPS.

Option B: Apify Actor

Use Apify Store (e.g. Amazon Scraper, Google Shopping). Call via API from your cron job. Apify handles proxies and anti-bot.

Option C: Firecrawl

Firecrawl returns markdown. Parse with regex or LLM for structured price extraction.

Step 3: Scheduler (Cron)​

# Run scraper every hour
0 * * * * cd /home/user/price-monitor && node scraper.js >> /var/log/price-monitor.log 2>&1

Or use node-cron inside a long-running Node process for finer control.

Step 4: Alerting​

When current_price <= target_price:

  • Email — Nodemailer, SendGrid, or Resend
  • Slack — Incoming webhook
  • Make.com — Webhook trigger → email/Slack/other actions

Docker Compose Stack​

services:
scraper:
build: .
env_file: .env
depends_on:
- postgres
- redis

postgres:
image: postgres:16-alpine
environment:
POSTGRES_DB: pricemonitor
POSTGRES_USER: scraper
POSTGRES_PASSWORD: ${DB_PASSWORD}
volumes:
- pgdata:/var/lib/postgresql/data

redis:
image: redis:alpine

volumes:
pgdata:

When to Use Apify Instead​

  • Anti-bot targets — Amazon, Walmart, etc. Apify manages proxies and rotation.
  • Many domains — Different selectors per site. Apify Store has pre-built Actors.
  • Low ops preference — No servers to maintain.

Apify price monitoring Actors can feed data to your PostgreSQL via webhook or scheduled API calls.

Next step

Start with 5–10 products and a simple Playwright scraper. Add alerting once the pipeline is stable.

Frequently Asked Questions

Scraper (Playwright or Apify) + PostgreSQL for history + cron for scheduling + email/Slack for alerts when price drops below threshold.

Yes. Use a VPS (e.g. Liquid Web), run scraper + DB + cron. Apify is the managed alternative.

Scraper (Playwright/Apify), database (PostgreSQL), scheduler (cron), alerting (email/Slack). For complex sites, add proxies.

Common mistakes and fixes

Scraper gets blocked by target site

Add delays, rotate User-Agents. Use residential proxies or Apify for anti-bot targets.

PostgreSQL connection refused from container

Use host.docker.internal or host network. Ensure pg_hba.conf allows connections from scraper.

Advertise on this page