How to build an Apify Actor health probe

Contains affiliate links. Disclosure.

At scale, a green checkmark in the Console is not enough. You need predictable signals: how many rows landed, whether fields match expectations, how long runs take, and immediate notice when something regresses. This guide builds a real Apify Actor health probe out of platform features, code-level logging, webhooks, and lightweight validation, so you can treat scrapers like any other production service.

Quick Answer

An Apify Actor health probe is a set of automated checks that run after each execution and confirm the scrape actually worked, not just that it didn't crash: a failed-run webhook, a row-count baseline against recent runs, and a schema check on the output. Alert when any one fails.

Patterns here were checked against current Apify docs and APIs, most recently 2026-09-09. For no-code alerting, start with Slack integration and the integrations hub.

Monitoring strategy: layers

Layer	What you observe	Typical tools
Platform	Run status, duration, compute, default dataset size	Apify Console, email/Slack integrations
Logs	Step-level context, retries, HTTP codes	Structured Actor.log, log export, streaming
Data health	Row counts, required fields, value ranges	Post-run validation job, webhook + script, dbt tests downstream
Automation	Incidents, paging, ticket creation	Webhooks → your API, Zapier, n8n, PagerDuty

Start with failed-run webhooks and dataset item count floors; add schema checks once the scraper is stable.

Built-in Console monitoring

Use Apify’s run overview first:

• Status timeline: running, succeeded, failed, timed out, aborted.
• Resource metrics: CPU/memory hints for right-sizing.
• Default dataset & KV: quick links to outputs.

Enable integrations (Slack, Zapier, etc.) at the Actor or task level for team-visible notifications before you build custom services.

Webhook setup for alerts

Webhooks are the backbone of event-driven monitoring. Register them on the Actor (or use account-level patterns if your org standardizes there).

Events to subscribe to first

• ACTOR.RUN.FAILED: code errors, unhandled exceptions, platform failures.
• ACTOR.RUN.TIMED_OUT: run exceeded max run time; often configuration or site slowness.
• ACTOR.RUN.ABORTED: manual or policy stop, useful for audit trails.
• ACTOR.RUN.SUCCEEDED: hook for validation jobs or downstream sync (Airbyte, warehouse load).

Payload essentials

Webhook bodies include the run id, Actor id, status, defaultDatasetId, and stats (including runtime), plus links you can forward to Slack or a ticket system. See Apify webhooks and payload templates.

Operational practices

• Return an HTTP 2xx response quickly from your endpoint; offload heavy work to a queue.
• Verify a shared secret or signature if exposed to the public internet.
• Use Apify static IPs if your receiver allowlists ingress.

For visual orchestration of the same events, route webhooks to n8n or Make.

What if I can't expose a public webhook endpoint?

Poll instead. If your receiver sits behind a corporate firewall or VPN with no inbound route, webhooks can't reach it. Call GET /v2/actor-runs/{runId} (or the equivalent method on the Apify API client) on a schedule and read the status field. Apify documents eight statuses: READY (initial), RUNNING, TIMING-OUT and ABORTING (transitional), and SUCCEEDED, FAILED, TIMED-OUT and ABORTED (terminal). Keep polling until the status is one of the four terminal values, then alert on anything that isn't SUCCEEDED. It's one extra API call per check instead of an inbound push, and it works from anywhere that can make an outbound HTTPS request.

Output validation and “run count” monitoring

Row-count and drop detection

After each successful run:

1. Fetch totalItemCount (or equivalent) for defaultDatasetId via the Apify API.
2. Compare to a rolling baseline (median of last N runs) or a hard minimum (e.g. “at least 50 products”).
3. If the count drops sharply, open an alert. Common causes are robots, HTML changes, or proxy issues.

Schedule the same check from a cron job outside Apify if you want independence from the scraper process.

Schema validation

Actor output schema (when you publish or maintain Actors) documents expected dataset shape for users and tooling. Keep it aligned with real items.

In code, validate a sample of items before exit:

• Required keys (url, price, title, …).
• Types (price is numeric, inStock is boolean).
• Sanity bounds (price > 0, string length limits).

On validation failure, fail the run with a clear log line so the webhook path fires and you do not silently load bad data into BI.

Golden-run regression tests

Keep a small fixture input (fixed URLs or search terms) and run it on a schedule or in CI. Compare item count and a hash of sorted keys to a baseline. This catches DOM refactors before production schedules hit the same markup.

Structured logging

Plain strings are hard to query. Prefer structured fields consumable by Datadog, OpenSearch, or BigQuery log sinks.

Python

from apify import Actor

async def main() -> None:
    async with Actor:
        log = Actor.log
        log.info(
            "batch_ok",
            extra={
                "record_count": 150,
                "http_status": 200,
                "duration_ms": 450,
                "label": "product_list",
            },
        )

JavaScript

import { Actor } from 'apify';

Actor.log.info('batch_ok', {
    record_count: 150,
    http_status: 200,
    duration_ms: 450,
    label: 'product_list',
});

Guidelines

• Use stable field names across versions.
• Log counts, not full PII.
• On exceptions, log.exception / stack traces in Console accelerate triage.

Streaming logs in real time

For long runs, poll or stream logs instead of waiting for completion:

import logging

from apify_client import ApifyClient

client = ApifyClient('YOUR_TOKEN')
run_client = client.run('RUN_ID')

logger = logging.getLogger('apify_run')
with run_client.get_streamed_log(to_logger=logger):
    ...  # logs are redirected to `logger` for as long as the run stays open

get_streamed_log() returns a context manager (or a start/stop pair), not a line-by-line iterator. Pass it a Logger and read the run's logs from there. Streaming helps with live debugging and temporary dashboards during incidents.

Scheduled health checks (outside the Actor)

Combine:

• Apify schedules for the scraper itself.
• A second lightweight Actor, or external cron, that:
  • Lists recent runs via API.
  • Asserts “at least one success in the last 24h” for critical jobs.
  • Posts to Slack if the pipeline is stale.

This catches silent schedule misconfiguration or paused tasks that pure run-level webhooks might miss.

Putting it together

1. Subscribe webhooks for failed and timed out runs → Slack or incident tool.
2. Log structured counters and HTTP outcomes inside the Actor.
3. Validate dataset rows (sample + totals) before marking success.
4. Baseline row counts and alert on drops.
5. Review weekly in the Console for duration and cost drift.

Apify webhook documentation →

Updated 2026-09-09: fixed the Python log-streaming example (get_streamed_log() is a context manager, not a line-by-line iterator) and added a polling path for readers who can't expose a public webhook endpoint.

Frequently Asked Questions

Add a webhook on ACTOR.RUN.FAILED and ACTOR.RUN.TIMED_OUT that POSTs to your endpoint, Slack Incoming Webhook, or an automation tool. The payload includes run and dataset identifiers so you can link directly to logs.

After ACTOR.RUN.SUCCEEDED, read the default dataset item count via the API and compare it to a rolling median or a configured minimum. Alert when the drop exceeds a percentage threshold.

It means verifying that dataset items include required fields and types, either with an Actor output schema for documentation and UI, or with explicit checks in code before the run finishes. Fail the run when validation does not pass so bad data does not propagate.

Console monitoring and standard webhooks do not add a separate Apify line item. Extra cost usually comes from additional API calls, validation Actors, or third-party log and alerting services you attach.

Open the run in the Console, read structured log fields, and reproduce locally with the same input. Stream logs during development. If webhooks fire, use the payload’s run id to deep-link your team to the exact failure.

Yes. Run test Actors from GitHub Actions after deploy, assert on dataset counts or schema in the workflow, and fail the pipeline if regressions appear. See the Apify GitHub integration docs for push and CI patterns.