Build an AI Content Pipeline: Scrape, Summarize, and Publish with LangChain and Apify
Scrape the web with Apify, run content through LangChain for summarization or transformation, then publish to a CMS or Google Sheets. The pipeline: Apify Website Content Crawler → LangChain chain → output store or API. Get started with Apify.
Pipeline Overview
| Stage | Tool | Role |
|---|---|---|
| 1. Scrape | Apify | Website Content Crawler fetches markdown from target URLs |
| 2. Transform | LangChain | PromptTemplate → ChatOpenAI/ChatAnthropic → StrOutputParser |
| 3. Publish | Apify Dataset / API | Store for traceability; export to Sheets, CMS, or webhook |
Use cases: draft blog posts from competitor articles, newsletter curation from RSS or news scrapers, product descriptions from e-commerce crawls, documentation summaries. Run on schedule with Apify Schedules and webhooks to trigger downstream publish. Content teams save hours by automating the "research → distill → format" loop. Marketing can pull competitor positioning from their blogs and generate differentiation briefs. Product teams can ingest changelogs and release notes for competitive intel. The pipeline is reusable: swap the Actor and prompt to change the source and output format.
LangChain's Apify Integration
The langchain_apify package provides ApifyWrapper. It runs any Apify Actor and maps dataset items to LangChain Document objects. Install:
pip install langchain langchain-openai langchain-apify
Set APIFY_API_TOKEN (or APIFY_API_KEY). Use call_actor with a dataset_mapping_function that converts each dataset item to a Document with page_content and metadata. The older langchain_community Apify integration is deprecated; use langchain_apify.
Step 1: Scrape with Website Content Crawler
from langchain_apify import ApifyWrapper
from langchain_core.documents import Document
apify = ApifyWrapper()
loader = apify.call_actor(
actor_id="apify/website-content-crawler",
run_input={
"startUrls": [{"url": "https://competitor.com/blog/topic"}],
"maxCrawlPages": 10,
"crawlerType": "cheerio"
},
dataset_mapping_function=lambda item: Document(
page_content=item.get("text") or item.get("markdown") or "",
metadata={"source": item.get("url", ""), "title": item.get("title", "")}
),
)
docs = loader.load()
Crawling can take a minute or more. The loader waits for the run to finish and fetches the dataset. Browse more content crawlers in the Apify Store. For RSS or news aggregation, use an RSS/Atom Actor instead of the Website Content Crawler. The mapping function stays similar: each feed item becomes a Document with URL and title in metadata. The RAG-Web-Browser Actor is another option for search-then-scrape workflows (e.g. "scrape top 5 Google results for query X").
Step 2: LangChain Chain for Summarization
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser
prompt = ChatPromptTemplate.from_messages([
("system", "You are a content analyst. Summarize the following in 2-3 paragraphs. Preserve key facts."),
("human", "{content}")
])
chain = prompt | ChatOpenAI(model="gpt-4o-mini", temperature=0.3) | StrOutputParser()
summaries = []
for doc in docs:
summary = chain.invoke({"content": doc.page_content[:8000]}) # truncate if needed
summaries.append({"url": doc.metadata["source"], "summary": summary})
Use ChatAnthropic for Claude instead of ChatOpenAI. For structured output (e.g. JSON with title, bullets, tags), use PydanticOutputParser or with_structured_output(). Batch documents to reduce API calls: concatenate several docs with separators and ask the LLM to produce one summary per doc in a list. For long documents, chunk first with RecursiveCharacterTextSplitter, summarize each chunk, then combine. See the Firecrawl RAG pipeline for chunking strategies.
Step 3: Store to Apify Dataset or External API
from apify_client import ApifyClient
client = ApifyClient()
dataset_id = "YOUR_DATASET_ID" # or create one
for s in summaries:
client.dataset(dataset_id).push_items([s])
Or POST to your CMS, Airtable, or Make.com webhook. Apify Integrations include Google Sheets—configure in the Console after the run.
Full End-to-End Example
import os
from langchain_apify import ApifyWrapper
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser
from apify_client import ApifyClient
os.environ["OPENAI_API_KEY"] = "sk-..."
os.environ["APIFY_API_TOKEN"] = "apify_api_..."
# 1. Scrape
apify = ApifyWrapper()
loader = apify.call_actor(
actor_id="apify/website-content-crawler",
run_input={
"startUrls": [{"url": "https://example.com/blog"}],
"maxCrawlPages": 5,
"crawlerType": "cheerio"
},
dataset_mapping_function=lambda item: Document(
page_content=item.get("text") or "",
metadata={"source": item.get("url", ""), "title": item.get("title", "")}
),
)
docs = loader.load()
# 2. Summarize
chain = (
ChatPromptTemplate.from_messages([
("system", "Summarize for a newsletter. 2-3 sentences. Include main takeaway."),
("human", "{content}")
])
| ChatOpenAI(model="gpt-4o-mini", temperature=0.2)
| StrOutputParser()
)
results = []
for doc in docs:
summary = chain.invoke({"content": doc.page_content[:6000]})
results.append({
"url": doc.metadata["source"],
"title": doc.metadata.get("title", ""),
"summary": summary
})
# 3. Store (use a dataset ID from Apify Console, or create via client.datasets().create())
client = ApifyClient()
dataset_id = "YOUR_DATASET_ID" # Create in Console or via API
client.dataset(dataset_id).push_items(results)
Use Cases
| Use case | Apify Actor | LangChain output | Publish |
|---|---|---|---|
| Blog drafts from competitors | Website Content Crawler | Rewrite/summarize with tone adjustment | CMS API, Notion |
| Newsletter curation | RSS or news scraper | Summarize + pick top 5 | Substack, Mailchimp, Sheets |
| Product descriptions | E-commerce crawler | Generate from specs | Shopify, WooCommerce |
| Doc summaries | Website Content Crawler | Extract key sections | Confluence, internal wiki |
Cost Estimation
- Apify: Compute units scale with crawl depth. ~0.01–0.05 USD per page. 50-page crawl: ~$0.50–2.
- OpenAI/Anthropic: ~$0.15–0.50 per 1K input tokens, ~$0.60–2 per 1K output. 5 docs × 3K tokens in, 500 out: ~$2–5 per run.
- Total per daily run: ~$3–10 depending on scale. Optimize with smaller crawls, cheaper models (gpt-4o-mini), or batching. Track usage in the Apify and OpenAI dashboards. Set budget alerts. For high-volume pipelines, consider batch processing: accumulate URLs over the day, run once at night, reduce per-run overhead.
Scheduling and Webhooks
Create an Apify Schedule to run your pipeline daily. Attach a webhook for ACTOR.RUN.SUCCEEDED to POST the dataset ID to Make.com or your API. Make fetches the dataset and publishes to Sheets or CMS. See the webhooks guide for payload structure and retry logic. For more complex flows—e.g. scrape → summarize → route by topic → different destinations—build a dedicated Actor that orchestrates the full pipeline. The Actor runs Apify crawlers, calls the LLM API, and pushes results. Schedules and webhooks still apply. This keeps logic versioned and replayable. Compare with the Claude SEC filings workflow for a similar scrape-then-analyze pattern.
Add webhooks to fire when your pipeline finishes. Route summaries to Slack or trigger a Make.com publish scenario.
Website Content Crawler for general sites. Use RSS or news Actors from the Store for articles. E-commerce Actors for product data.
ApifyWrapper integrates with LangChain loaders and document chains. Use the direct client when you need raw control or async flows.
Yes. Replace ChatOpenAI with ChatAnthropic. Same chain structure. Set ANTHROPIC_API_KEY.
Create an Apify Actor that runs your Python script, or use Apify Schedules to trigger the crawler. Add webhooks to trigger downstream publish.
Apify: more Actors, custom logic, platform. Firecrawl: clean markdown, fast. See Firecrawl RAG for that pipeline.