Apify and Airbyte: ELT Pipelines for Web-Scraped Data

Contains affiliate links. Disclosure.

To connect Apify to Airbyte, add the native Apify Dataset source in Airbyte, enter your Apify API token and dataset ID, run schema discovery, then map that source to a destination (BigQuery, Snowflake, PostgreSQL, and more) and schedule the sync. No custom extract scripts required.

Airbyte is an open-source data integration platform that moves data from sources to destinations with scheduled syncs, schema evolution, and operational monitoring. Apify exposes scraped rows as datasets, and Airbyte's Apify Dataset connector reads those items so you can load web data into warehouses and lakes.

Quick Answer

Apify integrates with Airbyte through a native Apify Dataset source connector. Sync Apify dataset results to BigQuery, Snowflake, PostgreSQL, or other destinations via scheduled Airbyte pipelines.

This page follows the Apify Airbyte integration and Airbyte's Apify Dataset source (verified May 2026). For the underlying token and dataset endpoints, see the Apify API tutorial; for cleaning rows before they hit a warehouse, see processing scraped data; for storage concepts (datasets vs. key-value store), see the Apify storage guide.

Why pair Apify with Airbyte?

Benefit	Why it matters
Warehouse-native analytics	Join scraped facts with CRM, billing, and product data in SQL.
Repeatable syncs	Airbyte handles scheduling, retries, and connector upgrades.
Many destinations	Same Apify dataset can feed BigQuery, Snowflake, Postgres, S3-backed lakes, and more, subject to Airbyte's connector catalog.
Operational clarity	Sync history and row counts surface pipeline breaks faster than ad-hoc CSV exports.

Prerequisites

• Apify account and API token from Settings → API & Integrations.
• A dataset ID produced by a completed Actor run (Console → Storage → Datasets).
• Airbyte Cloud or self-hosted Airbyte with the Apify source available in your workspace version.

Setup guide

Step 1: Collect Apify credentials

1. Sign in to the Apify Console.
2. Copy an API token with permission to read datasets.
3. Run your Actor (or task) once and note the default dataset ID attached to that run.

If the schema changes when you add fields, re-run discovery in Airbyte after the Actor output stabilizes.

Step 2: Add the Apify source in Airbyte

1. In Airbyte, open Sources → New source.
2. Search for Apify and select the Apify Dataset connector.
3. Paste the API token and dataset ID.
4. Complete the test and discover schema so streams and fields appear.

The Apify Dataset connector exposes four streams (verified May 2026):

Stream	What it returns
item_collection	Dataset items with dynamic schema, compatible with any Actor output. Use this for most scraping pipelines.
item_collection_website_content_crawler	Static schema stream for Website Content Crawler datasets.
dataset	Metadata for a single dataset.
dataset_collection	Lists all datasets on the account.

Both full refresh and incremental sync modes are supported. Verify the latest stream list before production mapping.

Step 3: Choose a destination

Common choices:

Destination	Typical use
BigQuery	Google Cloud analytics, ML feature stores
Snowflake	Central warehouse for BI tools
PostgreSQL	App-adjacent reporting, smaller teams
Redshift	AWS analytics stack
S3 / Delta / Iceberg	Lakehouse patterns (where supported by Airbyte)

Configure credentials per Airbyte's destination docs, then run a connection test.

Step 4: Create the connection

1. Select the Apify source and your destination.
2. Map fields: Airbyte lists columns discovered from sample items.
3. Choose sync frequency (hourly, daily, cron, manual).
4. Pick a sync mode:
   • Full refresh: replace tables each run (simple, higher warehouse churn).
   • Incremental: append or merge by cursor (supported by the connector, better for large datasets).
5. Save and trigger the first sync; validate row counts against the Apify dataset item count.

Airbyte's general tutorial: set up a connection.

Trigger a sync after an Actor finishes

To avoid stale data:

1. Add an Apify webhook on ACTOR.RUN.SUCCEEDED.
2. From your orchestrator (small service, Lambda, or workflow tool), call Airbyte's API to start a sync for the connection. See Airbyte's public API docs for POST /v1/connections/sync and authentication.
3. Optionally verify defaultDatasetId in the webhook payload matches the dataset your connection reads.

References: Apify webhooks and Airbyte's API documentation for your deployment type.

Example architecture

Scheduled Apify Actor
        │
        ▼
   Apify dataset
        │
        ▼
 Airbyte connection (incremental)
        │
        ▼
 Snowflake / BigQuery / Postgres
        │
        ▼
   BI or dbt models

Narrative: A price-monitoring Actor runs every morning on Apify schedules. Airbyte incrementally loads new dataset rows into Snowflake. dbt builds curated tables; Looker or Tableau reads the marts.

Airbyte Cloud vs. self-hosted

	Airbyte Cloud	Self-hosted OSS
Operations	Managed upgrades	You operate Docker/Kubernetes
Cost	Usage-based pricing	Infrastructure + your time
Data residency	Vendor region policies	Your VPC
Best for	Fastest path to value	Strict compliance or existing k8s

Troubleshooting

Issue	What to check
401 from Apify	Token expired or missing dataset:read scope; regenerate in Console.
Empty sync	Dataset ID from wrong run; confirm item count in Apify Storage.
Schema errors	Actor output shape changed; reset connection or adjust column mapping.
Late data	Sync schedule vs. Actor schedule; add webhook-triggered sync after runs.

Production checklist

Lock dataset IDs per environment (dev/staging/prod), document which Actor version writes each dataset, and monitor Airbyte failure notifications alongside Apify run failures.

Frequently Asked Questions

Create an Apify Dataset source in Airbyte, enter your Apify API token and the dataset ID from Console → Storage → Datasets, then test the source and connect it to your warehouse destination with the sync mode you need.

Any destination supported by your Airbyte edition (commonly BigQuery, Snowflake, PostgreSQL, Redshift, and cloud object stores) can consume the rows emitted by the Apify Dataset source.

When the Apify Dataset connector exposes incremental sync for your streams, you can limit each run to new or changed items. Check the connector documentation for the current sync modes and cursor fields.

Configure a webhook on ACTOR.RUN.SUCCEEDED and call Airbyte's API to start a sync for the relevant connection ID, optionally validating defaultDatasetId in the payload.

In the Apify Console, open Storage → Datasets, select the dataset created by your run, and copy its ID from the dataset detail view or from the run's defaultDatasetId field.

Yes. Apify Dataset is a native Airbyte source connector, available across Airbyte editions including Cloud and self-hosted, and documented on both docs.airbyte.com and docs.apify.com. It exposes four streams and supports full refresh and incremental sync.

Apify includes free monthly credits on the free plan; Airbyte Open Source is free to self-host aside from infrastructure, and Airbyte Cloud is usage-based. Verify current pricing on both vendors' sites.