Advanced Actor Monitoring and Logging

Contains affiliate links. Disclosure.

At scale, a green checkmark in the Console is not enough. You need predictable signals: how many rows landed, whether fields match expectations, how long runs take, and immediate notice when something regresses. This guide combines Apify platform features with code-level logging, webhooks, and lightweight validation so you can treat scrapers like any other production service.

Quick Answer

Monitor Apify Actors with output schema validation, run count monitoring, webhook alerts, and scheduled health checks. Set up alerts for failed runs or unexpected output drops.

Patterns here were checked against current Apify docs and APIs. For no-code alerting, start with Slack integration and the integrations hub.

Monitoring strategy: layers

Layer	What you observe	Typical tools
Platform	Run status, duration, compute, default dataset size	Apify Console, email/Slack integrations
Logs	Step-level context, retries, HTTP codes	Structured Actor.log, log export, streaming
Data health	Row counts, required fields, value ranges	Post-run validation job, webhook + script, dbt tests downstream
Automation	Incidents, paging, ticket creation	Webhooks → your API, Zapier, n8n, PagerDuty

Start with failed-run webhooks and dataset item count floors; add schema checks once the scraper is stable.

Built-in Console monitoring

Use Apify’s run overview first:

• Status timeline: running, succeeded, failed, timed out, aborted.
• Resource metrics: CPU/memory hints for right-sizing.
• Default dataset & KV: quick links to outputs.

Enable integrations (Slack, Zapier, etc.) at the Actor or task level for team-visible notifications before you build custom services.

Webhook setup for alerts

Webhooks are the backbone of event-driven monitoring. Register them on the Actor (or use account-level patterns if your org standardizes there).

Events to subscribe to first

• ACTOR.RUN.FAILED: code errors, unhandled exceptions, platform failures.
• ACTOR.RUN.TIMED_OUT: run exceeded max run time; often configuration or site slowness.
• ACTOR.RUN.ABORTED: manual or policy stop, useful for audit trails.
• ACTOR.RUN.SUCCEEDED: hook for validation jobs or downstream sync (Airbyte, warehouse load).

Payload essentials

Webhook bodies include run id, Actor id, status, defaultDatasetId, stats (including runtime), and links you can forward to Slack or a ticket system. See Apify webhooks and payload templates.

Operational practices

• Return HTTP 2xx quickly from your endpoint; offload heavy work to a queue.
• Verify a shared secret or signature if exposed to the public internet.
• Use Apify static IPs if your receiver allowlists ingress.

For visual orchestration of the same events, route webhooks to n8n or Make.

Output validation and “run count” monitoring

Row-count and drop detection

After each successful run:

1. Fetch totalItemCount (or equivalent) for defaultDatasetId via the Apify API.
2. Compare to a rolling baseline (median of last N runs) or a hard minimum (e.g. “at least 50 products”).
3. If the count drops sharply, open an alert. Common causes are robots, HTML changes, or proxy issues.

Schedule the same check from a cron outside Apify if you want independence from the scraper process.

Schema validation

Actor output schema (when you publish or maintain Actors) documents expected dataset shape for users and tooling. Keep it aligned with real items.

In code, validate a sample of items before exit:

• Required keys (url, price, title, …).
• Types (price is numeric, inStock is boolean).
• Sanity bounds (price > 0, string length limits).

On validation failure, fail the run with a clear log line so the webhook path fires and you do not silently load bad data into BI.

Golden-run regression tests

Keep a small fixture input (fixed URLs or search terms) and run it on a schedule or in CI. Compare item count and hash of sorted keys to a baseline. This catches DOM refactors before production schedules hit the same markup.

Structured logging

Plain strings are hard to query. Prefer structured fields consumable by Datadog, OpenSearch, or BigQuery log sinks.

Python

from apify import Actor

async def main() -> None:
    async with Actor:
        log = Actor.log
        log.info(
            "batch_ok",
            extra={
                "record_count": 150,
                "http_status": 200,
                "duration_ms": 450,
                "label": "product_list",
            },
        )

JavaScript

import { Actor } from 'apify';

Actor.log.info('batch_ok', {
    record_count: 150,
    http_status: 200,
    duration_ms: 450,
    label: 'product_list',
});

Guidelines

• Use stable field names across versions.
• Log counts, not full PII.
• On exceptions, log.exception / stack traces in Console accelerate triage.

Streaming logs in real time

For long runs, poll or stream logs instead of waiting for completion:

from apify_client import ApifyClient

client = ApifyClient('YOUR_TOKEN')
run_client = client.run('RUN_ID')

for line in run_client.get_streamed_log():
    print(line)

If your client version names the method differently, use the SDK’s log streaming API for the run resource. Streaming helps live debugging and temporary dashboards during incidents.

Scheduled health checks (outside the Actor)

Combine:

• Apify schedules for the scraper itself.
• A second lightweight Actor or external cron that:
  • Lists recent runs via API.
  • Asserts “at least one success in the last 24h” for critical jobs.
  • Posts to Slack if the pipeline is stale.

This catches silent schedule misconfiguration or paused tasks that pure run-level webhooks might miss.

Putting it together

1. Subscribe webhooks for failed and timed out runs → Slack or incident tool.
2. Log structured counters and HTTP outcomes inside the Actor.
3. Validate dataset rows (sample + totals) before marking success.
4. Baseline row counts and alert on drops.
5. Review weekly in the Console for duration and cost drift.

Apify webhook documentation →

Frequently Asked Questions

Add a webhook on ACTOR.RUN.FAILED and ACTOR.RUN.TIMED_OUT that POSTs to your endpoint, Slack Incoming Webhook, or an automation tool. The payload includes run and dataset identifiers so you can link directly to logs.

After ACTOR.RUN.SUCCEEDED, read the default dataset item count via the API and compare it to a rolling median or a configured minimum. Alert when the drop exceeds a percentage threshold.

It means verifying that dataset items include required fields and types, either with an Actor output schema for documentation and UI, or with explicit checks in code before the run finishes. Fail the run when validation does not pass so bad data does not propagate.

Console monitoring and standard webhooks do not add a separate Apify line item. Extra cost usually comes from additional API calls, validation Actors, or third-party log and alerting services you attach.

Open the run in the Console, read structured log fields, and reproduce locally with the same input. Stream logs during development. If webhooks fire, use the payload’s run id to deep-link your team to the exact failure.

Yes. Run test Actors from GitHub Actions after deploy, assert on dataset counts or schema in the workflow, and fail the pipeline if regressions appear. See the Apify GitHub integration docs for push and CI patterns.