Apify and AWS S3: export datasets to cloud storage

Contains affiliate links. Disclosure.

Quick answer

To export Apify data to AWS S3, run the S3 Uploader Actor with your bucket credentials and dataset ID, or write custom code that pulls the dataset through the Apify API and uploads it with the AWS SDK. Trigger uploads automatically on run success via Actor integrations or a webhook.

Open Apify · S3 Uploader Actor →

If you already run workloads on AWS, pushing scraped rows into S3 is the straightest path into Athena, Glue, Lambda, and data-lake tooling. This page covers the no-code S3 integration pattern (Uploader Actor + integrations), a custom API + SDK pipeline, format choices, and how to keep bucket layout predictable for downstream jobs.

Three ways to export, compared

There is no single "S3 button" in the Apify Console. Instead you pick one of three patterns depending on how much control and custom logic you need.

Approach	Setup effort	Custom logic	Best when
S3 Uploader Actor + integration	Lowest (no code)	None	You want one dataset copied to a bucket after each run
Webhook to Lambda	Medium	Full (transform, fan-out, route)	You need to rename keys, validate, or write to multiple buckets
Scheduled API pull + AWS SDK	Medium (your code)	Full	You already run a Python/Node service and want to pull datasets on your own schedule

The Uploader Actor covers most no-code cases. Reach for a webhook or custom pull only when you need transforms or non-standard routing before the object lands in S3.

Why export to S3?

• Data lake & analytics: query JSONL with Athena, load Redshift or Spark, or catalog with Glue.
• Archival: move historical crawl snapshots to S3 Standard-IA or Glacier when you do not need instant access.
• Event-driven pipelines: trigger Lambda on s3:ObjectCreated:* to normalize, validate, or forward data.
• Shared access: control who reads objects with IAM policies and bucket policies instead of exporting CSVs by hand.

Setup: S3 Uploader Actor (recommended)

The S3 Uploader Actor copies an Apify dataset into a bucket without maintaining your own upload service.

1. Create AWS credentials

1. In IAM, create a user or role used only for Apify uploads.
2. Attach a least-privilege policy that allows s3:PutObject on your prefix (and s3:ListBucket on the bucket if your workflow needs listing).
3. Create an access key and store the Access Key ID and Secret Access Key securely (Secrets Manager, not git).

2. Configure the Actor input

Typical fields (see the input schema for the full list):

Input	Purpose
datasetId	Dataset to export (from a finished run)
bucket / region	Target bucket and AWS region
accessKeyId / secretAccessKey	IAM credentials
key	S3 object path; you can include run ID or date for partitioning
format	csv, json, jsonl, xml, or xlsx

Dataset items are JSON objects; CSV/JSONL flatten or serialize fields according to the Actor’s current behavior. Always run a small test export before wiring production pipelines.

3. Automate after each scraper run (Actor integration)

1. Open your scraper Actor in the Apify Console.
2. Go to Integrations → Connect (or equivalent) and add the S3 Uploader integration.
3. Trigger on run succeeded and pass the dataset from the parent run, e.g. {{resource.defaultDatasetId}}, as datasetId in the Uploader’s input.

Details: Actor integrations. For a wider tour of connectors (Slack, webhooks, storage targets), see our Apify integrations guide.

4. Automate with webhooks (flexible)

For full control (rename keys, fan-out to multiple buckets, call Step Functions):

1. Add a webhook on ACTOR.RUN.SUCCEEDED for your scraper (webhooks docs).
2. Point it at API Gateway → Lambda (or another worker) that uses the Apify API to download the dataset and PutObject to S3.

This is useful when you need transforms, virus scanning, or multi-destination routing before the object lands in S3. If you transform rows in the Lambda, our processing scraped data guide covers cleaning and deduplication patterns. To catch failed or empty runs before they reach S3, pair the webhook with advanced Actor monitoring.

Custom pipeline: Apify API + boto3

When you already have a Python service, you can stream dataset bytes straight into S3:

import boto3
from apify_client import ApifyClient

apify = ApifyClient("YOUR_APIFY_TOKEN")
s3 = boto3.client(
    "s3",
    aws_access_key_id="YOUR_ACCESS_KEY",
    aws_secret_access_key="YOUR_SECRET_KEY",
    region_name="us-east-1",
)

dataset_id = "your-dataset-id"
dataset = apify.dataset(dataset_id)
csv_bytes = dataset.get_items_as_bytes(item_format="csv")

s3.put_object(
    Bucket="my-scraping-data",
    Key=f"datasets/{dataset_id}/export.csv",
    Body=csv_bytes,
    ContentType="text/csv",
)

Swap item_format for json or use the client’s iteration methods if you prefer JSON Lines written line-by-line for very large datasets.

Export formats

Format	Best for	Typical ContentType
CSV	Spreadsheets, simple warehouse loads	text/csv
JSON	Single blob consumption	application/json
JSONL	Athena, streaming parsers, append-only logs	application/x-ndjson
XML	Legacy integrations	application/xml
XLSX	Excel-first teams	application/vnd.openxmlformats-officedocument.spreadsheetml.sheet

Athena and JSONL

For AWS Athena external tables, JSONL (one JSON object per line) is usually the least painful format: no array wrapper, easy to partition by S3 prefix (year=2026/month=03/…).

Example bucket layout

Consistent keys make partitions and lifecycle rules trivial:

s3://my-scraping-bucket/
├── actor=google-maps/
│   └── dt=2026-03-21/run=abc123/data.jsonl
├── actor=amazon-products/
│   └── dt=2026-03-21/run=def456/data.csv
└── snapshots/
    └── weekly/2026-W12/full_export.jsonl

FAQ

Frequently Asked Questions

Use the official S3 Uploader Actor (apify/s3-uploader) with your bucket and dataset ID, or build a pipeline with the Apify API and AWS SDK (for example boto3) inside Lambda or your own service.

The standard pattern is the S3 Uploader Actor plus Actor integrations or webhooks. That covers most no-code and low-code cases; custom code uses the Apify API and S3 APIs together.

At minimum s3:PutObject on the target prefix (and often s3:ListBucket on the bucket). Follow least privilege and rotate keys if they are ever exposed.

CSV for spreadsheets and simple loads, JSON for single-file dumps, JSONL for Athena and streaming processors, XLSX or XML when downstream tools require them.

Yes. Connect the S3 Uploader via Actor integrations on run success, or use a webhook to Lambda (or another worker) that fetches the dataset and uploads to S3.

From the parent run, use the template for the default dataset in integrations (for example resource.defaultDatasetId), or copy the dataset ID from the run detail page when testing manually.

Yes. Datasets hold tabular run output, but key-value store records (files, screenshots, raw HTML) can be fetched with the Apify API and uploaded with the AWS SDK, the same put_object call works for any bytes.

Run your AWS SDK pull script on a schedule (EventBridge, cron, or an Apify schedule), fetch the latest dataset with get_items_as_bytes, and upload it. This decouples export timing from the scraper run.

Apify charges for the run that produces the data and, if you use the S3 Uploader Actor, for that Actor's compute. The S3 storage, requests, and any Lambda or data transfer are billed by AWS separately.