Apify and LangChain Integration: Feed Web Data into LLMs

Contains affiliate links. Disclosure.

Quick Answer

Apify integrates with LangChain via the ApifyWrapper class (package langchain-apify). Use Apify Actors to fetch web content, then pass results to LangChain chains, agents, or RAG pipelines as Document objects.

LangChain is the common orchestration layer for LLM apps; Apify supplies fresh, structured web data without you maintaining headless browsers and extractors for every site. This guide covers the Python langchain-apify package specifically; for the full set of SDKs, the REST API, and other integrations, see the Apify for developers hub.

Re-verified against the langchain-apify package, its GitHub repo, and the official Apify LangChain integration docs on 2026-09-10: ApifyWrapper, ApifyDatasetLoader, and ApifyActorsTool are all still current.

Why pair Apify with LangChain?

• RAG: crawl docs or blogs, embed chunks, answer questions with citations.
• Agents: expose Actors as tools (ApifyActorsTool) so an agent can “open a browser” or “run a Store scraper” on demand.
• Batch analytics: run an Actor on a schedule, load the dataset, summarize or classify with an LLM.

Install

pip install langchain langchain-openai langchain-apify

Use versions compatible with your LangChain install; if imports fail, align langchain-apify with the LangChain release notes for your stack.

Environment variables

import os

os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
os.environ["APIFY_API_TOKEN"] = "your-apify-api-token"

• Apify API token
• OpenAI API keys

If you want to see what that token authenticates against outside of ApifyWrapper, the Apify API tutorial covers starting runs, polling, and pulling datasets directly.

Example 1: ApifyWrapper + Website Content Crawler → vector index

Runs the Website Content Crawler and maps each dataset row to a LangChain Document:

from langchain_apify import ApifyWrapper
from langchain.indexes import VectorstoreIndexCreator
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.documents import Document
from langchain_core.vectorstores import InMemoryVectorStore

apify = ApifyWrapper()

loader = apify.call_actor(
    actor_id="apify/website-content-crawler",
    run_input={
        "startUrls": [{"url": "https://docs.apify.com/platform/?fpr=use-apify"}],
        "maxCrawlPages": 50,
        "crawlerType": "cheerio",
    },
    dataset_mapping_function=lambda item: Document(
        page_content=item.get("text") or "",
        metadata={"source": item.get("url", "")},
    ),
)

index = VectorstoreIndexCreator(
    vectorstore_cls=InMemoryVectorStore,
    embedding=OpenAIEmbeddings(),
).from_loaders([loader])

Query the index

llm = ChatOpenAI()
query = "How do Apify Actors work?"
result = index.query_with_sources(query, llm=llm)

print(result["answer"])
print(result["sources"])

VectorstoreIndexCreator is the legacy LangChain helper

For new projects, prefer the LCEL pattern in Example 3 below: InMemoryVectorStore.from_documents(...) + a retriever + an LCEL chain. It's the direction current LangChain docs point to and makes the retrieval step explicit.

Example 2: Load an existing Apify dataset

When scraping and LLM steps are separate (scheduled Actor, then offline indexing), use ApifyDatasetLoader:

from langchain_apify import ApifyDatasetLoader
from langchain_core.documents import Document

loader = ApifyDatasetLoader(
    dataset_id="your-dataset-id",
    dataset_mapping_function=lambda item: Document(
        page_content=item.get("text") or "",
        metadata={"source": item.get("url", "")},
    ),
)

docs = loader.load()

Pipe docs into text splitters, a vector store (Chroma, Pinecone, etc.), or an LCEL chain.

Example 3: Minimal LCEL-style RAG (embed + retrieve)

After docs = loader.load():

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

embeddings = OpenAIEmbeddings()
vectorstore = InMemoryVectorStore.from_documents(documents=docs, embedding=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

template = """Answer using only the context.

Context:
{context}

Question: {question}"""
prompt = ChatPromptTemplate.from_template(template)

llm = ChatOpenAI(model="gpt-4o-mini")

def format_docs(d):
    return "\n\n".join(doc.page_content for doc in d)

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(chain.invoke("What is an Apify Actor?"))

Adjust models, store, and prompt for production.

Agents and MCP

• ApifyActorsTool: register Store Actors as tools for LangChain agents.
• Apify MCP: discover and run Actors from compatible clients; see Use MCP servers.

LangChain is one of several ways to wire Apify into an agent stack. If you're also evaluating CrewAI, LlamaIndex, LangGraph, or a vector-database export, the AI agent and vector database integrations guide lays out all three connection paths side by side.

Actors that work well with LangChain

Actor	Use case	Link
Website Content Crawler	Docs, blogs, knowledge bases	Store
RAG Web Browser	Search + fetch for RAG	Store
Google Search Results Scraper	SERP-style research	Store

Cost snapshot

Two meters run independently: Apify platform usage for the crawl, and your LLM provider's usage for embeddings and chat.

Piece	Rate	Source
Website Content Crawler, raw HTTP (crawlerType: "cheerio", used in Example 1)	$0.20/1,000 pages subject to change · verified 2026-09-10	Apify Store page
Website Content Crawler, headless browser	$0.50-$5.00 per 1,000 pages, depending on the site	Apify Store page
OpenAI text-embedding-3-small	$0.02/1M tokens subject to change · verified 2026-09-10	OpenAI pricing

At the raw-HTTP rate, the maxCrawlPages: 50 run in Example 1 costs about $0.01 in Apify platform usage. Embedding is priced per token, not per page, and at $0.02 per 1M tokens a crawl of that size embeds for a fraction of a cent. Chat completion cost depends on which model you pick and how much context you stuff into the prompt; check OpenAI's pricing page for the model in Example 3.

The Apify free plan includes $5 of platform usage a month, which covers thousands of pages at the raw-HTTP rate before you pay anything.

:::tip Try it end to end Grab an Apify API token, run the Website Content Crawler example above against your own docs site or blog, and feed the output straight into Example 3's retriever. A 50-page test run costs about a cent, cheap enough to check the loader-to-Document mapping before committing to a bigger crawl. :::

Frequently Asked Questions

Use the langchain-apify package: ApifyWrapper runs Actors and returns LangChain-compatible loaders; ApifyDatasetLoader reads an existing dataset. Map each row to a Document, then use chains, agents, or RAG components as usual.

ApifyWrapper is the LangChain helper that authenticates to Apify, starts Actors, waits for datasets, and exposes results as document loaders, so you do not hand-roll HTTP polling against the Apify API in every project.

Start with apify/website-content-crawler for full-site text. For search-driven RAG, consider apify/rag-web-browser or the Google Search Results Scraper, depending on whether you need site crawl or SERP listings.

This guide targets Python with langchain-apify. For JavaScript/TypeScript stacks, call the Apify API or client from your LangChain.js tools and map JSON rows to your document model.

Use ApifyActorsTool so the agent can invoke Actors as tools, or run an Actor upfront with ApifyWrapper, load Documents, and give the agent a retriever over that vector store.

In the Apify Console under Account → Integrations. Set APIFY_API_TOKEN (or APIFY_API_KEY per SDK version) before running ApifyWrapper.