Apify Dataset Connector for Airbyte

Contains affiliate links. Disclosure.

Apify Dataset is the official name of the connector Airbyte uses to pull data out of Apify. Both vendors use that exact name: Apify's own integration doc tells you to open the Sources tab in Airbyte and select Apify Dataset. To connect the two: add the native Apify Dataset source in Airbyte, enter your Apify API token and dataset ID, run schema discovery, then map that source to a destination (BigQuery, Snowflake, PostgreSQL, S3-backed lakes, and more) and schedule the sync. No custom extract scripts required.

Airbyte is an open-source data integration platform that moves data from sources to destinations with scheduled syncs, schema evolution, and operational monitoring. Apify exposes scraped rows as datasets, and Airbyte's Apify Dataset connector reads those items so you can load web data into warehouses and lakes.

Can Apify export directly to S3, BigQuery, or Snowflake?

Yes, through this same Apify Dataset connector. Point an Airbyte connection at BigQuery, Snowflake, PostgreSQL, Redshift, or an S3-backed lake and every scraped dataset syncs on schedule, no custom extract script. A Google Sheets destination also exists, but Airbyte rate-limits it hard: fine for a small preview, not for production volume.

This page follows the Apify Airbyte integration and Airbyte's Apify Dataset source (verified September 2026). For the underlying token and dataset endpoints, see the Apify API tutorial; for cleaning rows before they hit a warehouse, see processing scraped data; for storage concepts (datasets vs. key-value store), see the Apify storage guide.

Why pair Apify with Airbyte?

Benefit	Why it matters
Warehouse-native analytics	Join scraped facts with CRM, billing, and product data in SQL.
Repeatable syncs	Airbyte handles scheduling, retries, and connector upgrades.
Many destinations	Same Apify dataset can feed BigQuery, Snowflake, Postgres, S3-backed lakes, and more, subject to Airbyte's connector catalog.
Operational clarity	Sync history and row counts surface pipeline breaks faster than ad-hoc CSV exports.

Prerequisites

• Apify account and API token from Settings → API & Integrations.
• A dataset ID produced by a completed Actor run (Console → Storage → Datasets).
• Airbyte Cloud or self-hosted Airbyte with the Apify source available in your workspace version.

Setup guide

Step 1: Collect Apify credentials

1. Sign in to the Apify Console.
2. Copy an API token with permission to read datasets.
3. Run your Actor (or task) once and note the default dataset ID attached to that run.

If the schema changes when you add fields, re-run discovery in Airbyte after the Actor output stabilizes.

Step 2: Add the Apify source in Airbyte

1. In Airbyte, open Sources → New source.
2. Search for Apify and select the Apify Dataset connector.
3. Paste the API token and dataset ID.
4. Complete the test and discover schema so streams and fields appear.

The Apify Dataset connector (v2.2.56 as of September 2026) exposes four streams:

Stream	What it returns
item_collection	Dataset items with dynamic schema, compatible with any Actor output. Use this for most scraping pipelines.
item_collection_website_content_crawler	Static schema stream for Website Content Crawler datasets.
dataset	Metadata for a single dataset.
dataset_collection	Lists all datasets on the account.

Airbyte lists Full Refresh Sync: Yes and Incremental Sync: Yes for this connector, but it states that at the connector level and gives no per-stream breakdown, and the connector's published manifest defines no cursor field on any of the four streams (both checked September 2026). Treat incremental as something to confirm in the sync-mode dropdown after schema discovery, not a given. Stream names occasionally shift between connector versions. Re-run schema discovery after an Airbyte upgrade.

Step 3: Choose a destination

Common choices:

Destination	Typical use
BigQuery	Google Cloud analytics, ML feature stores
Snowflake	Central warehouse for BI tools
PostgreSQL	App-adjacent reporting, smaller teams
Redshift	AWS analytics stack
S3 / Delta / Iceberg	Lakehouse patterns (where supported by Airbyte)

Configure credentials per Airbyte's destination docs, then run a connection test.

Step 4: Create the connection

1. Select the Apify source and your destination.
2. Map fields: Airbyte lists columns discovered from sample items.
3. Choose sync frequency (hourly, daily, cron, manual).
4. Pick a sync mode:
   • Full refresh: replace tables each run (simple, higher warehouse churn).
   • Incremental: append or merge by cursor, where Airbyte offers it for your stream (better for large datasets).
5. Save and trigger the first sync; validate row counts against the Apify dataset item count.

Airbyte's general tutorial: set up a connection.

Production checklist

Lock dataset IDs per environment (dev/staging/prod), document which Actor version writes each dataset, and monitor Airbyte failure notifications alongside Apify run failures. This is the step teams skip in a quick proof-of-concept and regret once the pipeline feeds a real dashboard.

Trigger a sync after an Actor finishes

To avoid stale data:

1. Add an Apify webhook on ACTOR.RUN.SUCCEEDED.
2. From your orchestrator (small service, Lambda, or workflow tool), call Airbyte's public API to start a sync: POST /jobs with body {"connectionId": "<your-connection-id>", "jobType": "sync"}. (Updated 2026-09-09: an earlier version of this page pointed to POST /v1/connections/sync, which Airbyte's current API reference does not document. Job creation now goes exclusively through /jobs.)
3. Optionally verify defaultDatasetId in the webhook payload matches the dataset your connection reads.

References: Apify webhooks and Airbyte's public API reference for authentication.

Example architecture

Scheduled Apify Actor
        │
        ▼
   Apify dataset
        │
        ▼
 Airbyte connection (incremental)
        │
        ▼
 Snowflake / BigQuery / Postgres
        │
        ▼
   BI or dbt models

Narrative: A price-monitoring Actor runs every morning on Apify schedules. Airbyte incrementally loads new dataset rows into Snowflake. dbt builds curated tables; Looker or Tableau reads the marts.

Airbyte Cloud vs. self-hosted

	Airbyte Cloud	Self-hosted OSS
Operations	Managed upgrades	You operate Docker/Kubernetes
Cost	Standard tier starts at $10/month, volume-based (verified September 2026)	Free software, infrastructure + your time
Data residency	Vendor region policies	Your VPC
Best for	Fastest path to value	Strict compliance or existing k8s

On the Apify side, the Free plan includes $5/month in usage credit at $0.20/CU, enough to test this pipeline before committing to a paid plan (verified September 2026).

Troubleshooting

Issue	What to check
401 from Apify	Token expired or missing dataset:read scope; regenerate in Console.
Empty sync	Dataset ID from wrong run; confirm item count in Apify Storage.
Schema errors	Actor output shape changed; reset connection or adjust column mapping.
Late data	Sync schedule vs. Actor schedule; add webhook-triggered sync after runs.

Frequently Asked Questions

Create an Apify Dataset source in Airbyte, enter your Apify API token and the dataset ID from Console → Storage → Datasets, then test the source and connect it to your warehouse destination with the sync mode you need.

Any destination supported by your Airbyte edition (commonly BigQuery, Snowflake, PostgreSQL, Redshift, and cloud object stores) can consume the rows emitted by the Apify Dataset source.

Airbyte's connector page lists both full refresh and incremental sync support for the Apify Dataset source, but it states that at the connector level rather than per stream, and the connector's published manifest defines no cursor field on the four streams (checked September 2026). After schema discovery, check which sync modes Airbyte actually offers for your stream before designing around incremental loads.

Configure a webhook on ACTOR.RUN.SUCCEEDED and call Airbyte's public API (POST /jobs, with connectionId and jobType: "sync" in the body) to start a sync for the relevant connection, optionally validating defaultDatasetId in the payload.

In the Apify Console, open Storage → Datasets, select the dataset created by your run, and copy its ID from the dataset detail view or from the run's defaultDatasetId field.

Yes. Apify Dataset is a native Airbyte source connector, available across Airbyte editions including Cloud and self-hosted, and documented on both docs.airbyte.com and docs.apify.com. It exposes four streams; Airbyte lists full refresh and incremental sync support at the connector level.

Apify's Free plan includes $5/month in usage credit at $0.20/CU, enough to test this pipeline (verified September 2026). Airbyte Open Source is free to self-host aside from infrastructure; Airbyte Cloud has no free tier and its Standard pricing starts at $10/month, billed by data volume.