Apify and LangChain Integration: Feed Web Data into LLMs

Contains affiliate links. Disclosure.

Quick Answer

Apify integrates with LangChain via the ApifyWrapper class (package langchain-apify). Use Apify Actors to fetch web content, then pass results to LangChain chains, agents, or RAG pipelines as Document objects.

LangChain is the common orchestration layer for LLM apps; Apify supplies fresh, structured web data without you maintaining headless browsers and extractors for every site.

Why pair Apify with LangChain?

• RAG: crawl docs or blogs, embed chunks, answer questions with citations.
• Agents: expose Actors as tools (ApifyActorsTool) so an agent can “open a browser” or “run a Store scraper” on demand.
• Batch analytics: run an Actor on a schedule, load the dataset, summarize or classify with an LLM.

Install

pip install langchain langchain-openai langchain-apify

Use versions compatible with your LangChain install; if imports fail, align langchain-apify with the LangChain release notes for your stack.

Environment variables

import os

os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
os.environ["APIFY_API_TOKEN"] = "your-apify-api-token"

• Apify API token
• OpenAI API keys

Example 1: ApifyWrapper + Website Content Crawler → vector index

Runs the Website Content Crawler and maps each dataset row to a LangChain Document:

from langchain_apify import ApifyWrapper
from langchain.indexes import VectorstoreIndexCreator
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.documents import Document
from langchain_core.vectorstores import InMemoryVectorStore

apify = ApifyWrapper()

loader = apify.call_actor(
    actor_id="apify/website-content-crawler",
    run_input={
        "startUrls": [{"url": "https://docs.apify.com/platform/?fpr=use-apify"}],
        "maxCrawlPages": 50,
        "crawlerType": "cheerio",
    },
    dataset_mapping_function=lambda item: Document(
        page_content=item.get("text") or "",
        metadata={"source": item.get("url", "")},
    ),
)

index = VectorstoreIndexCreator(
    vectorstore_cls=InMemoryVectorStore,
    embedding=OpenAIEmbeddings(),
).from_loaders([loader])

Query the index

llm = ChatOpenAI()
query = "How do Apify Actors work?"
result = index.query_with_sources(query, llm=llm)

print(result["answer"])
print(result["sources"])

VectorstoreIndexCreator is the legacy LangChain helper

For new projects, prefer the LCEL pattern in Example 3 below: InMemoryVectorStore.from_documents(...) + a retriever + an LCEL chain. It's the direction current LangChain docs point to and makes the retrieval step explicit.

Example 2: Load an existing Apify dataset

When scraping and LLM steps are separate (scheduled Actor, then offline indexing), use ApifyDatasetLoader:

from langchain_apify import ApifyDatasetLoader
from langchain_core.documents import Document

loader = ApifyDatasetLoader(
    dataset_id="your-dataset-id",
    dataset_mapping_function=lambda item: Document(
        page_content=item.get("text") or "",
        metadata={"source": item.get("url", "")},
    ),
)

docs = loader.load()

Pipe docs into text splitters, a vector store (Chroma, Pinecone, etc.), or an LCEL chain.

Example 3: Minimal LCEL-style RAG (embed + retrieve)

After docs = loader.load():

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

embeddings = OpenAIEmbeddings()
vectorstore = InMemoryVectorStore.from_documents(documents=docs, embedding=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

template = """Answer using only the context.

Context:
{context}

Question: {question}"""
prompt = ChatPromptTemplate.from_template(template)

llm = ChatOpenAI(model="gpt-4o-mini")

def format_docs(d):
    return "\n\n".join(doc.page_content for doc in d)

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(chain.invoke("What is an Apify Actor?"))

Adjust models, store, and prompt for production.

Agents and MCP

• ApifyActorsTool: register Store Actors as tools for LangChain agents.
• Apify MCP: discover and run Actors from compatible clients; see Use MCP servers.

Actors that work well with LangChain

Actor	Use case	Link
Website Content Crawler	Docs, blogs, knowledge bases	Store
RAG Web Browser	Search + fetch for RAG	Store
Google Search Results Scraper	SERP-style research	Store

Cost snapshot

Piece	Rough order of magnitude
Apify Actor run	Depends on Actor pricing × pages/events
OpenAI embeddings + chat	Cents per thousand tokens
Vector DB	Often $0 on free tiers

The Apify free plan includes monthly credits for experiments.

Frequently Asked Questions

Use the langchain-apify package: ApifyWrapper runs Actors and returns LangChain-compatible loaders; ApifyDatasetLoader reads an existing dataset. Map each row to a Document, then use chains, agents, or RAG components as usual.

ApifyWrapper is the LangChain helper that authenticates to Apify, starts Actors, waits for datasets, and exposes results as document loaders, so you do not hand-roll HTTP polling against the Apify API in every project.

Start with apify/website-content-crawler for full-site text. For search-driven RAG, consider apify/rag-web-browser or the Google Search Results Scraper, depending on whether you need site crawl or SERP listings.

This guide targets Python with langchain-apify. For JavaScript/TypeScript stacks, call the Apify API or client from your LangChain.js tools and map JSON rows to your document model.

Use ApifyActorsTool so the agent can invoke Actors as tools, or run an Actor upfront with ApifyWrapper, load Documents, and give the agent a retriever over that vector store.

In the Apify Console under Account → Integrations. Set APIFY_API_TOKEN (or APIFY_API_KEY per SDK version) before running ApifyWrapper.