Processing scraped data on Apify: API, SDKs, exports, and pipelines

Contains affiliate links. Disclosure.

After a run finishes, your results live in Apify Storage, most often a dataset of JSON objects you pushed with push_data(). The next step is to read, filter, transform, and move that data into spreadsheets, databases, or warehouses without loading millions of rows into RAM.

This guide covers REST API access, JavaScript and Python clients, CSV/JSON/Excel exports, query parameters for shaping responses, and patterns for large datasets and downstream ETL.

Quick Answer

Process Apify datasets using the REST API, JavaScript/Python SDK, or direct exports to CSV/JSON/Excel. For large datasets, use the Apify API with pagination or stream results to a data warehouse.

If you are new to the platform, create a run on Apify and copy the Dataset ID from the run’s Storage tab before trying the examples below.

Where scraped data lives on Apify

Storage	Role
Dataset	Append-only list of JSON records (push_data rows); primary output for scrapers
Key-value store	Files, screenshots, HTML blobs, arbitrary binaries
Request queue	URLs left to crawl; metadata for crawlers

Limits and throughput (typical platform constraints)

Topic	Guidance
Item size	A single dataset record has a maximum payload size (a few MB); split very large blobs into the key-value store. Confirm the current cap in platform limits.
Column limits (tabular export)	Up to 2,000 columns for CSV / Excel / HTML table views (platform limits)
Write throughput	High; design clients to retry on transient errors
Read throughput	Prefer pagination; avoid hammering one dataset with unbounded parallel reads
Retention	Named storages are kept indefinitely. Unnamed (default run) storages expire after 7 days unless retained, so name important datasets for long-lived pipelines. See Apify storage usage.

Read dataset items with the REST API

Use the Dataset items endpoint with your API token and dataset ID (from the Console URL or run output).

export APIFY_TOKEN='YOUR_API_TOKEN'
export DATASET_ID='your-dataset-id'

curl -s "https://api.apify.com/v2/datasets/${DATASET_ID}/items?format=json&clean=true&limit=1000&offset=0" \
  -H "Authorization: Bearer ${APIFY_TOKEN}"

Common query parameters (see official API reference for the full list):

Parameter	Purpose
format	json, csv, xml, xlsx, rss, html
clean	Normalize output for tabular formats
limit / offset	Pagination for large datasets
fields	Include only certain fields (reduces payload size)
omit	Strip noisy fields
unwind	Flatten array fields for tabular export

Pagination pattern: loop with increasing offset until you receive fewer than limit rows.

JavaScript: apify-client

Install: npm install apify-client

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const datasetId = 'YOUR_DATASET_ID';

let offset = 0;
const limit = 1000;

while (true) {
    const { items } = await client.dataset(datasetId).listItems({ offset, limit });
    if (!items.length) break;

    for (const row of items) {
        // Transform or forward each row
        await sendToWarehouse(row);
    }

    offset += items.length;
    if (items.length < limit) break;
}

For Actor code on the platform, you often read your default dataset via the Actor SDK instead of hard-coding IDs (see the Apify SDK for JavaScript docs).

Python: apify-client

Install: pip install apify-client

Streaming with iterate_items avoids holding the whole dataset in memory:

import os

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
dataset_id = "YOUR_DATASET_ID"

for item in client.dataset(dataset_id).iterate_items():
    process(item)

To pull a single page manually:

page = client.dataset(dataset_id).list_items(limit=1000, offset=0)
for item in page.items:
    process(item)

Export from the Apify Console

For ad-hoc analysis:

1. Open the run → Storage → Dataset.
2. Choose Export and pick JSON, CSV, Excel, JSONL, etc.
3. For recurring workflows, prefer the API or SDK so exports are reproducible.

JSONL is especially handy for streaming imports into BigQuery, Snowflake stages, or object storage.

Filter and transform before downstream systems

At read time (API): use fields, omit, and unwind to reduce noise and flatten nested arrays for BI tools.

In code: map each item to a strict schema, drop nulls, parse dates to ISO-8601, and normalize URLs.

In a second Actor: split scraping and processing. The scraper only captures raw pages; a processor Actor reads the dataset or key-value store and writes cleaned rows to a new named dataset or external API. That failure isolation prevents re-crawling when your transform logic changes.

Large datasets: streaming, batching, and parallel runs

1. Never list_items() without limits on huge datasets. Always page or iterate.
2. Split crawls across multiple runs (sharded URL lists) writing into a shared named dataset, then deduplicate (see Merge, Dedup & Transform Datasets on the Store).
3. Monitor memory in processing Actors; aggregate with iterators, not giant in-memory lists.
4. Integrate to warehouses via Airbyte, Fivetran-style tools, or your own batch job reading JSONL exports.

Example: post-process HTML files into Elasticsearch

This pattern fits when a scraper saved raw HTML to disk or a key-value store and you want structured documents in Elasticsearch.

Two-step pipelines

A production setup usually has Scraper Actor → Key-value store / dataset, then Processor Actor that uses the Apify API to list keys or read items from the first run before parsing.

import os
from typing import Any, Dict

from bs4 import BeautifulSoup
from elasticsearch import Elasticsearch

from apify import Actor


def process_html_file(content: str) -> Dict[str, Any]:
    soup = BeautifulSoup(content, "html.parser")
    return {
        "title": soup.title.string if soup.title else "No Title",
        "body_text": soup.body.get_text(" ", strip=True) if soup.body else "",
    }


PROCESSORS = {
    ".html": process_html_file,
    ".htm": process_html_file,
}


async def main() -> None:
    async with Actor:
        log = Actor.log
        actor_input = await Actor.get_input() or {}
        es_url = actor_input.get("es_url")
        es_index = actor_input.get("es_index", "scraped-data")
        data_directory = actor_input.get("data_directory", "./scraped_files")

        if not es_url:
            raise ValueError("Elasticsearch URL (es_url) is required.")

        es_client = Elasticsearch(es_url)
        log.info("Connected to Elasticsearch. Index: %s", es_index)
        log.info("Processing files from directory: %s", data_directory)

        for filename in os.listdir(data_directory):
            file_path = os.path.join(data_directory, filename)
            if not os.path.isfile(file_path):
                continue

            _, extension = os.path.splitext(filename)
            processor = PROCESSORS.get(extension.lower())
            if not processor:
                log.warning("No processor for extension %s — skipping.", extension)
                continue

            try:
                with open(file_path, encoding="utf-8") as handle:
                    content = handle.read()
                doc = processor(content)
                doc["source_file"] = filename
                es_client.index(index=es_index, document=doc)
            except Exception:
                log.exception("Failed to process %s", filename)

        log.info("Finished processing all files.")


if __name__ == "__main__":
    import asyncio

    asyncio.run(main())

Wire credentials and index names via Actor input, never commit secrets to git.

Related documentation

• Apify API tutorial: authentication, run lifecycle, webhooks
• Apify integrations: Sheets, S3, Zapier/Make/n8n
• Platform storage: authoritative limits and retention

Storage and API behavior summarized here aligns with current Apify Platform documentation; verify exact limits in the official docs before capacity planning.

Open Apify and run your first Actor →

FAQ

Frequently Asked Questions

Use the REST endpoint GET /v2/datasets/{datasetId}/items with your API token, or the official JavaScript/Python apify-client with listItems/iterate_items. Both support pagination via limit and offset.

JSON is often a single array or document; JSONL writes one JSON object per line. JSONL streams efficiently into data warehouses, log pipelines, and line-based processors.

Stream with iterate_items in Python or paginate with listItems in JavaScript. Do not fetch the entire dataset in one HTTP call. For transforms, process one row at a time and write results incrementally.

Yes. The items API supports parameters such as fields, omit, unwind, clean, limit, and offset. Use fields to keep only the columns your warehouse needs.

For serious pipelines, split scraping and processing. Scrapers focus on reliable capture; processors handle parsing, normalization, and delivery. That way a bug in your Elasticsearch mapping does not force a full recrawl.

Write all shards to a named dataset or merge afterward with a dedicated Actor such as Merge, Dedup & Transform Datasets from the Apify Store.

Use managed connectors (for example Airbyte to Snowflake/BigQuery), upload JSONL to object storage and load from there, or run a small service that reads the dataset API and batches inserts into your database.