Skip to main content

YouTube Transcripts for LLM and RAG Pipelines (2026)

· 7 min read
Yassine El Haddad
Software Developer & Automation Specialist

I build production AI agents, web scrapers, and automation pipelines. Most of what I publish here comes from the actual problems they run into: proxies that get banned, anti-bot stacks that fingerprint your client, RAG that drifts when the underlying data moves. Stack: Python, TypeScript, Go, FastAPI, LangChain, Crawlee, Playwright, deployed on AWS, GCP, and Cloudflare.

The underused RAG corpus​

Most RAG pipelines ingest PDFs, web pages, and documentation. Few teams tap into YouTube — and that is a significant gap. YouTube hosts decades of expert spoken content across every domain: medical lectures, financial analysis, engineering walkthroughs, legal commentary, academic conference talks. This content does not exist as text anywhere else.

A single channel from a domain expert can represent thousands of hours of structured knowledge. Turned into transcripts, it becomes a high-quality, domain-specific retrieval corpus for an LLM that no public dataset provides.

YouTube Transcript Scraper Pro extracts that corpus at scale — with a transcript_llm field designed for direct ingestion into any RAG framework.

The transcript_llm field​

Standard transcript text includes tokens that consume context window budget without adding meaning:

[Music] Welcome back everyone so today we're going to [Applause] um talk about
[Music] the fundamentals of uh machine learning and I think [Applause]

The transcript_llm field strips these:

Welcome back everyone so today we're going to talk about the fundamentals of
machine learning and I think

Tokens stripped include: [Music], [Applause], [Laughter], (laughs), [Inaudible], stage directions, and other non-speech annotations. Whitespace is normalised. The result passes directly to a vector store without pre-processing.

To request it, add "llm" to outputFormats:

run_input = {
"startUrls": [{"url": "https://www.youtube.com/@channelname"}],
"outputFormats": ["llm"],
"maxResults": 50,
}

LangChain integration​

from apify_client import ApifyClient
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("codepoetry/youtube-transcript-ai-scraper").call(
run_input={
"startUrls": [{"url": "https://www.youtube.com/playlist?list=YOUR_PLAYLIST_ID"}],
"outputFormats": ["llm"],
"languages": ["en"],
"enableAiFallback": True,
"maxAiMinutes": 120,
}
)

docs = [
Document(
page_content=item["transcript_llm"],
metadata={
"source": item["metadata"]["url"],
"title": item["metadata"]["title"],
"channel": item["metadata"]["channel"],
"duration_sec": item["metadata"]["duration"],
"upload_date": item["metadata"]["upload_date"],
"is_ai_generated": item.get("is_ai_generated", False),
},
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items()
if item.get("transcript_llm")
]

vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

Install dependencies: pip install apify-client langchain-core langchain-openai langchain-chroma

The is_ai_generated metadata flag lets you filter or weight results by source type downstream.

LlamaIndex integration​

from apify_client import ApifyClient
from llama_index.core import Document, VectorStoreIndex

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("codepoetry/youtube-transcript-ai-scraper").call(
run_input={
"startUrls": [{"url": "https://www.youtube.com/@expertchannel"}],
"outputFormats": ["llm"],
"languages": ["en"],
"enableAiFallback": False,
"maxResults": 100,
}
)

documents = [
Document(
text=item["transcript_llm"],
metadata={
"url": item["metadata"]["url"],
"title": item["metadata"]["title"],
"channel": item["metadata"]["channel"],
},
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items()
if item.get("transcript_llm")
]

index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("Explain the main argument from the most recent video")

Chunking strategy​

YouTube segments can be minutes long. Chunking by segment boundary produces uneven, context-poor chunks. Two better strategies:

Sentence-boundary chunking (recommended for most pipelines)

Split transcript_llm at sentence boundaries using nltk.sent_tokenize or spacy. Aim for 200–400 token chunks with 20–40 token overlap. This produces semantically coherent units independent of caption timing.

Segment-boundary chunking (when timestamps matter)

Use transcript_json instead of transcript_llm. Each segment object includes start, end, and text. Group segments into fixed-duration windows (e.g., 60 seconds) and store timestamps in metadata for source attribution in citations.

# Segment-window chunking example
segments = item["transcript_json"]
window_sec = 60
chunks = []
current, current_start = [], segments[0]["start"]

for seg in segments:
current.append(seg["text"])
if seg["end"] - current_start >= window_sec:
chunks.append({
"text": " ".join(current),
"start": current_start,
"end": seg["end"],
"url": f"{item['metadata']['url']}&t={int(current_start)}",
})
current, current_start = [], seg["end"]

if current:
chunks.append({"text": " ".join(current), "start": current_start, "end": seg["end"]})

The &t= URL parameter lets you link citations back to the exact timestamp in the original video.

Batch ingestion from a channel or playlist​

No code? The same channel-to-corpus pipeline runs inside Claude: chat with a YouTube channel in Claude, with every answer cited to a video and timestamp.

For large corpus builds, use a channel URL and maxResults to control scope:

run_input = {
"startUrls": [{"url": "https://www.youtube.com/@yourexpertchannel"}],
"outputFormats": ["json", "llm"],
"languages": ["en"],
"enableAiFallback": True,
"maxAiMinutes": 300,
"skipAiFallbackIfLongerThan": 90,
"maxResults": 200,
}

Before running on an unknown channel, cap the AI transcription budget with the run-level knobs:

cost_control = {
"maxAiMinutes": 300, # AI budget per run (default 30, max 600)
"skipAiFallbackIfLongerThan": 90, # skip AI fallback for videos longer than 90 minutes
}

maxAiMinutes sets how many minutes of AI transcription a single run may spend — the default is 30 and the maximum is 600. skipAiFallbackIfLongerThan skips the AI fallback for long videos so a single stream does not drain the budget, and videos longer than 120 minutes never go through AI fallback at all.

Cost to build a corpus​

Pricing on the free Apify plan:

Corpus sizeVideosAI neededEstimated cost
Small (channel sample)50No~$0.053
Medium (full channel)200No~$0.203
Large (mixed — 20% no captions, avg 10 min)500Mixed~$12.50
Podcast archive (all AI, avg 45 min)100Yes — 4,500 min~$54.10

On the free $5 credit: approximately 5,000 native transcripts or 400 minutes of AI transcription. For mixed corpora, native captions are always checked first — AI cost only applies to videos that need it.

See the actor pricing page for plan discounts.

Word-level timestamps​

Enable wordLevel: true to receive per-word timestamps in transcript_json:

{
"start": 18.5,
"end": 21.0,
"text": "We're no strangers to love",
"words": [
{ "start": 18.5, "end": 18.8, "text": "We're" },
{ "start": 18.8, "end": 19.1, "text": "no" },
{ "start": 19.1, "end": 19.6, "text": "strangers" }
]
}

For auto-generated captions and AI transcriptions, the per-word values are derived from segment timing — useful structure for audio alignment or citation linking, but treat them as approximate rather than a guaranteed field. Manual captions carry no word timing.

Start building

Try YouTube Transcript Scraper Pro →
Sign up for Apify — free credits on every new account.

Frequently Asked Questions

The built-in speech-to-text model performs well on clear speech in widely spoken languages. Accuracy degrades with heavy accents, domain jargon, or poor audio quality. The language_probability field (0–1) indicates the model's confidence in language detection. For quality-critical corpora, treat AI transcripts as a first draft and plan a review pass for high-value content.

99 languages including English, Spanish, French, German, Portuguese, Japanese, Chinese, Arabic, and Hindi. Pass forceTranscriptionLanguage (ISO 639-1 code) to skip the 30-second auto-detection window — this reduces processing time by ~20% when you know the channel language in advance.

If all 500 videos have native captions, the cost on the free plan is approximately $0.50 (500 × $0.001 + $0.0025 start fee). If 20% have no captions and average 10 minutes each, add 1,000 AI minutes × $0.012 = $12.00, for a total of ~$12.50. Use skipAiFallbackIfLongerThan to keep long videos from inflating the AI bill.

Yes. Set wordLevel: true to add a words array to each transcript_json segment. Each word has start, end, and text fields — treat the per-word values as approximate, since they are derived from segment timing. Available for auto-generated captions and AI transcriptions — not manual captions.

No. The built-in speech-to-text model runs on Apify's compute as part of the actor. You pay through Apify's Pay-Per-Event pricing ($0.012/min on the free plan) — no separate speech-to-text API account or OpenAI key is needed.

Advertise on this page