YouTube Transcripts for LLM and RAG Pipelines (2026)
The underused RAG corpus
Most RAG pipelines ingest PDFs, web pages, and documentation. Few teams tap into YouTube — and that is a significant gap. YouTube hosts decades of expert spoken content across every domain: medical lectures, financial analysis, engineering walkthroughs, legal commentary, academic conference talks. This content does not exist as text anywhere else.
A single channel from a domain expert can represent thousands of hours of structured knowledge. Turned into transcripts, it becomes a high-quality, domain-specific retrieval corpus for an LLM that no public dataset provides.
YouTube Transcript Scraper — Captions & AI Fallback extracts that corpus at scale — with a transcript_llm field designed for direct ingestion into any RAG framework.
The transcript_llm field
Standard transcript text includes tokens that consume context window budget without adding meaning:
[Music] Welcome back everyone so today we're going to [Applause] um talk about
[Music] the fundamentals of uh machine learning and I think [Applause]
The transcript_llm field strips these:
Welcome back everyone so today we're going to talk about the fundamentals of
machine learning and I think
Tokens stripped include: [Music], [Applause], [Laughter], (laughs), [Inaudible], stage directions, and other non-speech annotations. Whitespace is normalised. The result passes directly to a vector store without pre-processing.
To request it, add "llm" to outputFormats:
run_input = {
"startUrls": [{"url": "https://www.youtube.com/@channelname"}],
"outputFormats": ["llm"],
"maxResults": 50,
}
LangChain integration
from apify_client import ApifyClient
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("codepoetry/youtube-transcript-ai-scraper").call(
run_input={
"startUrls": [{"url": "https://www.youtube.com/playlist?list=YOUR_PLAYLIST_ID"}],
"outputFormats": ["llm"],
"languages": ["en"],
"aiFallback": True,
"maxAiMinutes": 120,
}
)
docs = [
Document(
page_content=item["transcript_llm"],
metadata={
"source": item["metadata"]["url"],
"title": item["metadata"]["title"],
"channel": item["metadata"]["channel"],
"duration_sec": item["metadata"]["duration"],
"upload_date": item["metadata"]["upload_date"],
"is_ai_generated": item.get("is_ai_generated", False),
},
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items()
if item.get("transcript_llm")
]
vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
Install dependencies: pip install apify-client langchain-core langchain-openai langchain-chroma
The is_ai_generated metadata flag lets you filter or weight results by source type downstream.
LlamaIndex integration
from apify_client import ApifyClient
from llama_index.core import Document, VectorStoreIndex
client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("codepoetry/youtube-transcript-ai-scraper").call(
run_input={
"startUrls": [{"url": "https://www.youtube.com/@expertchannel"}],
"outputFormats": ["llm"],
"languages": ["en"],
"aiFallback": False,
"maxResults": 100,
}
)
documents = [
Document(
text=item["transcript_llm"],
metadata={
"url": item["metadata"]["url"],
"title": item["metadata"]["title"],
"channel": item["metadata"]["channel"],
},
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items()
if item.get("transcript_llm")
]
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("Explain the main argument from the most recent video")
Chunking strategy
YouTube segments can be minutes long. Chunking by segment boundary produces uneven, context-poor chunks. Two better strategies:
Sentence-boundary chunking (recommended for most pipelines)
Split transcript_llm at sentence boundaries using nltk.sent_tokenize or spacy. Aim for 200–400 token chunks with 20–40 token overlap. This produces semantically coherent units independent of caption timing.
Segment-boundary chunking (when timestamps matter)
Use transcript_json instead of transcript_llm. Each segment object includes start, end, and text. Group segments into fixed-duration windows (e.g., 60 seconds) and store timestamps in metadata for source attribution in citations.
# Segment-window chunking example
segments = item["transcript_json"]
window_sec = 60
chunks = []
current, current_start = [], segments[0]["start"]
for seg in segments:
current.append(seg["text"])
if seg["end"] - current_start >= window_sec:
chunks.append({
"text": " ".join(current),
"start": current_start,
"end": seg["end"],
"url": f"{item['metadata']['url']}&t={int(current_start)}",
})
current, current_start = [], seg["end"]
if current:
chunks.append({"text": " ".join(current), "start": current_start, "end": seg["end"]})
The &t= URL parameter lets you link citations back to the exact timestamp in the original video.
Batch ingestion from a channel or playlist
For large corpus builds, use a channel URL and maxResults to control scope:
run_input = {
"startUrls": [{"url": "https://www.youtube.com/@yourexpertchannel"}],
"outputFormats": ["json", "llm"],
"languages": ["en"],
"aiFallback": True,
"maxAiMinutes": 300,
"skipAiFallbackIfLongerThan": 90,
"maxResults": 200,
"dryRun": False,
}
Before running on an unknown channel, run with dryRun: True first:
dry_run_input = {**run_input, "dryRun": True}
dry = client.actor("codepoetry/youtube-transcript-ai-scraper").call(run_input=dry_run_input)
for item in client.dataset(dry["defaultDatasetId"]).iterate_items():
print(item["metadata"]["title"], "| AI needed:", item.get("would_need_ai"), "| Est. min:", item.get("estimated_ai_min"))
Dry Runs charge no AI events and let you inspect the full cost breakdown before committing.
Cost to build a corpus
Pricing on the free Apify plan:
| Corpus size | Videos | AI needed | Estimated cost |
|---|---|---|---|
| Small (channel sample) | 50 | No | ~$0.055 |
| Medium (full channel) | 200 | No | ~$0.205 |
| Large (mixed — 20% no captions, avg 10 min) | 500 | Mixed | ~$12.41 |
| Podcast archive (all AI, avg 45 min) | 100 | Yes — 4,500 min | ~$54.00 |
On the free $5 credit: approximately 4,900 native transcripts or 400 minutes of AI transcription. For mixed corpora, native captions are always checked first — AI cost only applies to videos that need it.
See the actor pricing page for plan discounts.
Word-level timestamps
Enable wordLevel: true to receive per-word timestamps in transcript_json:
{
"start": 18.5,
"end": 21.0,
"text": "We're no strangers to love",
"words": [
{ "start": 18.5, "end": 18.8, "text": "We're" },
{ "start": 18.8, "end": 19.1, "text": "no" },
{ "start": 19.1, "end": 19.6, "text": "strangers" }
]
}
Word-level data is available for auto-generated captions and AI transcriptions (not manual captions). Use it for audio alignment, forced-alignment fine-tuning datasets, or precise citation linking.
Try YouTube Transcript Scraper — Captions & AI Fallback →
Sign up for Apify — free credits on every new account.
Whisper performs well on clear speech in widely spoken languages. Accuracy degrades with heavy accents, domain jargon, or poor audio quality. The language_probability field (0–1) indicates the model's confidence in language detection. For quality-critical corpora, treat AI transcripts as a first draft and plan a review pass for high-value content.
99 languages including English, Spanish, French, German, Portuguese, Japanese, Chinese, Arabic, and Hindi. Pass forceWhisperLanguage (ISO 639-1 code) to skip the 30-second auto-detection window — this reduces processing time by ~20% when you know the channel language in advance.
If all 500 videos have native captions, the cost on the free plan is approximately $0.51 (500 × $0.001 + $0.005 start fee). If 20% have no captions and average 10 minutes each, add 1,000 AI minutes × $0.012 = $12.00, for a total of ~$12.41. Use dryRun mode to get an exact preview before running.
Yes. Set wordLevel: true to add a words array to each transcript_json segment. Each word has start, end, and text fields. Available for auto-generated captions and AI transcriptions — not manual captions.
No. The faster-whisper model is bundled into the actor's Docker image and runs on Apify's compute. You pay through Apify's Pay-Per-Event pricing ($0.012/min on the free plan) — no separate Whisper API account or OpenAI key is needed.