Skip to main content

Build a Price Monitoring System on Liquid Web

· 3 min read
Yassine El Haddad
Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

A self-hosted price monitoring system on Liquid Web needs a scraper, storage, a scheduler, and alerting. This guide outlines the architecture and implementation options.

For managed price monitoring without running servers, Apify offers Actors for Amazon, Google Shopping, and e-commerce sites.

System Architecture

ComponentRoleOptions
ScraperFetch prices from target sitesPlaywright, Apify Actor, Firecrawl
StoragePersist historical pricesPostgreSQL, SQLite
QueueOptional job queueRedis, Bull/BullMQ
SchedulerRun scraper on intervalcron, node-cron
AlertingNotify on price dropsEmail, Slack, webhook
cron (hourly) → Scraper → PostgreSQL

Redis (optional queue)

Alert if price < threshold → Email/Slack
VolumeRAMStorageLiquid Web Plan
10–50 products4 GB80 GB$8.50/mo
100–500 products8 GB240 GB$22.50/mo
1000+ products16 GB440 GB$45/mo

See Liquid Web VPS for current plans.

Step 1: Database Schema

CREATE TABLE products (
id SERIAL PRIMARY KEY,
url TEXT UNIQUE NOT NULL,
name TEXT,
target_price DECIMAL(10,2),
created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE price_history (
id SERIAL PRIMARY KEY,
product_id INT REFERENCES products(id),
price DECIMAL(10,2) NOT NULL,
scraped_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_price_history_product ON price_history(product_id);
CREATE INDEX idx_price_history_scraped ON price_history(scraped_at);

Step 2: Scraper Options

Option A: Playwright (self-hosted)

// Minimal price extraction
const { chromium } = require('playwright');
const price = await page.locator('.price').textContent();

Deploy via Docker or bare VPS.

Option B: Apify Actor

Use Apify Store (e.g. Amazon Scraper, Google Shopping). Call via API from your cron job. Apify handles proxies and anti-bot.

Option C: Firecrawl

Firecrawl returns markdown. Parse with regex or LLM for structured price extraction.

Step 3: Scheduler (Cron)

# Run scraper every hour
0 * * * * cd /home/user/price-monitor && node scraper.js >> /var/log/price-monitor.log 2>&1

Or use node-cron inside a long-running Node process for finer control.

Step 4: Alerting

When current_price <= target_price:

  • Email — Nodemailer, SendGrid, or Resend
  • Slack — Incoming webhook
  • Make.com — Webhook trigger → email/Slack/other actions

Docker Compose Stack

services:
scraper:
build: .
env_file: .env
depends_on:
- postgres
- redis

postgres:
image: postgres:16-alpine
environment:
POSTGRES_DB: pricemonitor
POSTGRES_USER: scraper
POSTGRES_PASSWORD: ${DB_PASSWORD}
volumes:
- pgdata:/var/lib/postgresql/data

redis:
image: redis:alpine

volumes:
pgdata:

When to Use Apify Instead

  • Anti-bot targets — Amazon, Walmart, etc. Apify manages proxies and rotation.
  • Many domains — Different selectors per site. Apify Store has pre-built Actors.
  • Low ops preference — No servers to maintain.

Apify price monitoring Actors can feed data to your PostgreSQL via webhook or scheduled API calls.

Apify Affiliate Banner 728x90Apify Affiliate Banner 728x90Apify Affiliate Banner 300x50Apify Affiliate Banner 300x50
Next step

Start with 5–10 products and a simple Playwright scraper. Add alerting once the pipeline is stable.

Frequently Asked Questions

Scraper (Playwright or Apify) + PostgreSQL for history + cron for scheduling + email/Slack for alerts when price drops below threshold.

Yes. Use a VPS (e.g. Liquid Web), run scraper + DB + cron. Apify is the managed alternative.

Scraper (Playwright/Apify), database (PostgreSQL), scheduler (cron), alerting (email/Slack). For complex sites, add proxies.

Common mistakes and fixes

Scraper gets blocked by target site

Add delays, rotate User-Agents. Use residential proxies or Apify for anti-bot targets.

PostgreSQL connection refused from container

Use host.docker.internal or host network. Ensure pg_hba.conf allows connections from scraper.