TranscriptDocs
Console

Build a transcription pipeline with yt-dlp and Whisper

If you need transcripts for videos that don't have captions, the standard DIY stack is yt-dlp to pull audio and Whisper to transcribe it. It's a genuinely solid combination and worth building yourself if transcription is part of your core product. Here's how the pieces fit together, how to get timestamps out of it, and the operational problems that only show up once you're running it on more than a handful of videos.

Step 1: pull the audio

yt-dlp handles the download. You don't want the video — just the audio track, extracted and converted:

Shell
yt-dlp -f bestaudio -x --audio-format mp3 -o audio.mp3 "https://youtu.be/VIDEO_ID"

-f bestaudio picks the best available audio-only stream so you're not downloading video you'll throw away, -x extracts the audio, and --audio-format mp3 transcodes it to MP3 via ffmpeg (which yt-dlp shells out to, so it needs to be installed and on your PATH). This works well beyond YouTube — yt-dlp supports a long list of sites, so the same command is your ingestion path for most video URLs.

Step 2: transcribe it

You've got two real options here: call the OpenAI API, or run Whisper yourself.

Option A: the OpenAI API

Simplest to stand up — no model weights, no GPU:

Python
from openai import OpenAI

client = OpenAI()
with open("audio.mp3", "rb") as f:
    result = client.audio.transcriptions.create(
        model="whisper-1",
        file=f,
        response_format="verbose_json",  # needed for per-segment timestamps
    )

for segment in result.segments:
    print(segment.start, segment.end, segment.text)

response_format="verbose_json" is the part people miss. The default response is just a flat string — no timing at all. verbose_json returns the segment-level breakdown with start, end, and text per segment, which is what you need if the output is going into an SRT/VTT file or anything else that has to sync to the audio.

Option B: run it locally

Two libraries cover this. openai-whisper is the reference implementation:

Python
import whisper

model = whisper.load_model("base")
result = model.transcribe("audio.mp3")

for segment in result["segments"]:
    print(segment["start"], segment["end"], segment["text"])

faster-whisper is a CTranslate2 reimplementation that's noticeably lighter on memory and runs well on CPU with int8 quantization:

Python
from faster_whisper import WhisperModel

model = WhisperModel("base", device="cpu", compute_type="int8")
segments, info = model.transcribe("audio.mp3")

for segment in segments:
    print(segment.start, segment.end, segment.text)

Both return segments with start/end/text — that's your timestamp data without touching the API at all. The trade-off is you're now managing model weights, picking a model size (bigger models are more accurate and slower), and, if you want reasonable speed on longer audio, a GPU.

Step 3: chunking long audio

Both the API and local Whisper have practical limits on a single file — the API has an upload size cap, and local inference on long files gets slow and memory-hungry. Past roughly the length of a long podcast episode, split first:

Shell
ffmpeg -i audio.mp3 -f segment -segment_time 600 -c copy chunk_%03d.mp3

This splits the audio into 600-second (10-minute) chunks without re-encoding. Transcribe each chunk independently, then stitch the results back together — the part that's easy to get wrong is forgetting that each chunk's timestamps start at zero. You have to add the chunk's offset back in before the timestamps mean anything relative to the original file:

Python
SEGMENT_SECONDS = 600
all_segments = []

for i, chunk_path in enumerate(sorted(chunk_paths)):
    offset = i * SEGMENT_SECONDS
    chunk_segments = transcribe(chunk_path)  # however you call Whisper
    for seg in chunk_segments:
        seg["start"] += offset
        seg["end"] += offset
        all_segments.append(seg)

Skip this step and every chunk after the first reports timestamps as if it were its own standalone file — which reads as correct until you actually seek to one and it's ten minutes off.

What breaks in production

The pipeline above works fine for one video on your laptop. Running it as a service surfaces a different set of problems.

Cloud IPs get blocked. yt-dlp running on AWS, GCP, or most VPS providers will eventually hit yt-dlp's equivalent of an HTTP 429 or a "sign in to confirm you're not a bot" wall from YouTube — datacenter IP ranges are heavily rate-limited or blocked outright, even though the exact same command works from your home IP. The fix is routing requests through a residential proxy (yt-dlp --proxy http://user:pass@host:port ...), which adds cost and another moving part to keep configured and rotated.

Long jobs need a queue. Transcribing a two-hour video takes real wall-clock time — you can't do it inline in an HTTP request/response cycle. That means a job queue, worker processes, and a way for the caller to poll or get notified when the job finishes, on top of the transcription code itself.

Cache or pay twice. If the same video gets requested more than once — common for anything popular — re-downloading and re-transcribing it is pure waste. Caching by video ID (or a hash of the audio) is close to mandatory once you have real traffic.

Cost adds up per minute of audio. Whether you're paying per-minute API usage or paying for GPU time to run Whisper yourself, cost scales with audio length, not with how novel the content is — chunking overlaps, retries after a failed job, and re-transcribing to fix a bad run all get billed the same way.

None of these are hard problems individually. Together, they're the difference between a script and a service.

Or use a managed API

If transcription isn't the product you're building — it's a step on the way to something else — it can save you the build time to call an API that's already handling proxies, queueing, caching, and chunking. transcript.land does this for YouTube, TikTok, X, Instagram, Bilibili, Facebook, RedNote, and direct audio URLs, returning captions when they exist and falling back to AI transcription automatically when they don't:

Shell
curl -X POST https://api.transcript.land/v1/transcript \
  -H "Authorization: Bearer $TRANSCRIPT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://youtu.be/VIDEO_ID", "format": "json"}'

A caption-less video returns a job_id you poll instead of the transcript directly — the same async pattern you'd build yourself, minus the queue and worker code:

Shell
curl https://api.transcript.land/v1/jobs/JOB_ID \
  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"

The CLI wraps the polling for you:

Shell
brew install ziqorg/tap/transcript
transcript login
transcript get "https://youtu.be/VIDEO_ID" -o json

Output formats are TXT, SRT, VTT, Markdown, or JSON. The free tier covers 15 minutes of audio a month, which is enough to see whether the output shape fits your pipeline before committing to anything.

Summary

  • yt-dlp -f bestaudio -x --audio-format mp3 gets you the audio; the OpenAI API with response_format="verbose_json", or openai-whisper/ faster-whisper locally, gets you segment-level timestamps.
  • Chunk long audio with ffmpeg's segment muxer, and remember to add each chunk's offset back into its timestamps when you stitch results together.
  • The parts that turn this from a script into a service are proxies (cloud IPs get blocked), a job queue, a cache, and per-minute cost — worth building if transcription is your product, worth outsourcing if it isn't.