Six Weeks, Six Billion Videos: A Builder’s Journal

What does it actually take to build a video search product — one where users type a phrase and find the moment inside millions of videos? This is the week-by-week journal of scoping, filtering, indexing, deduplicating, and shipping one, including the mid-project surprises and what the whole thing cost. The foundation throughout: a purpose-built video dataset of 6 billion videos from 700 million channels.

Journals are honest in a way case studies aren’t. This one keeps the mistakes in.

Week 1: Scoping, and the Decision Not to Crawl

The product goal: search inside video content — “find the clip where the chef folds the egg whites” — across a corpus measured in the hundreds of thousands of hours. The first architecture discussion ended quickly in favor of building collection ourselves, and then the arithmetic started.

Video platforms gate bulk access and detect collection at scale. Transcoding hundreds of thousands of hours is its own infrastructure line. And deduplication — the same clip re-encoded, cropped, and re-uploaded across channels — is a research problem, not a script. A credible self-built collection path was two quarters of engineering before the first useful index existed.

The alternative: license a filtered shard of a commercial corpus. Thordata’s video dataset — 6 billion original videos from 700 million unique channels, built for LLM and multimodal training — is delivered as structured records with captions, channel lineage, and metadata, from roughly $0.25 per 1,000 records. The decision that week: license breadth, and reserve engineering effort for the search problem itself, which is where the product’s differentiation actually lives.

Week 2: Filtering the Corpus Down to the Product

Six billion videos is not a corpus; it’s a continent. The product needed one region of it: instructional food content, English and Japanese, under ten minutes, captioned. Filtering happened upstream of storage, which is the entire economic trick of structured records — you pay for and store the shard that matches the product, not the continent:

That last filter was added late, which is Week 4’s story.

Week 3: The Indexing Pipeline

With a shard in object storage, the pipeline was unglamorous and fast to build: caption extraction, joint text-video embedding, and index upserts.

def index_shard(shard):
    for rec in shard:
        if not rec.get("caption_text"):
            continue                        # alignment pairs only
        clip_windows = segment(rec, seconds=30)
        for i, window in enumerate(clip_windows):
            index.upsert({
                "id": f"{rec['record_id']}:{i}",
                "video": rec["video_url"],
                "start": window["start"],
                "text": window["caption"],
                "channel": rec["channel_id"],
                "embedding": embed(window),   # multimodal encoder
            })

Two structural choices from the dataset’s record format paid for themselves: record_id gave stable keys across re-indexing, and channel_id made the Week 4 fix possible at all.

Week 4: The Deduplication Surprise

The first evaluation run returned the same technique demonstrated by twelve near-identical videos. Not identical — different encodings, different watermarks, slightly different crops — but the same content. The corpus’s channel diversity (700 million channels means the same popular tutorial gets re-uploaded everywhere) is exactly what makes near-duplicates common.

The fix was two passes: content-hash matching on video fingerprints to catch re-encodes, and channel-aware ranking to demote re-uploads relative to originals. The channel lineage in each record — traceable to source — made “original versus re-upload” a solvable question rather than a guess. Retrieval quality jumped more from this pass than from any embedding model change all project.

Week 5: Evaluation, and the Channel Balance Fix

The team’s evaluation set was 200 hand-labeled queries. Accuracy was good on popular techniques and poor on niche ones — and the cause was visible in the index composition: a handful of high-volume channels dominated the retrieved results. The Week 2 channel-cap filter (retroactively applied) rebalanced retrieval toward the corpus’s long tail, which is where the product’s differentiation lives: anyone can find the famous tutorial; the product’s value is finding the obscure one.

This is the quiet argument for channel-diverse datasets in product form: diversity isn’t an academic virtue, it’s retrieval coverage.

Week 6: Shipping, and the Distribution Question

Launch raised a question the corpus couldn’t answer: where do the product’s pages stand in search? A video search product lives on organic traffic, and video-rich SERPs — with video carousels and featured snippets — shift constantly.

The team added the SERP monitoring solution to track where product pages and indexed clips appear across target keywords and regions, at roughly $0.70 per 1,000 structured responses. The same vendor that supplied the dataset supplies the search-side intelligence, which means one account, one dashboard, one invoice — and rank data that lands in the same warehouse as the video index, ready for the “does ranking correlate with indexed content type” analysis on next quarter’s roadmap.

The Ledger

The pattern to notice: the data was cheap, the compute was the budget, and the engineering went where it should — into the product’s actual hard problems. That’s the whole argument for starting from a purpose-built corpus instead of building collection first.

Questions We Get Asked

Why not just use the platform APIs directly? Platform APIs serve their product, not yours: rate limits, restricted fields, terms that constrain downstream products, and no caption or channel-lineage structure. A dataset built for training and product use ships with the structure and the rights conversation already handled.

How fresh does the index need to be? For search products, freshness is a product decision: weekly is fine for most queries, daily for trending ones. Scheduled collection jobs make the cadence a config change rather than a project.

What about the multilingual corpus — was it worth it? For launch markets, yes: Japanese instructional content was materially under-served by existing products, and the corpus’s channel diversity meant enough of it existed in the shard to matter.

Did you consider search data as a retrieval corpus too? Late in the project, yes — and it’s the next roadmap item. Feeding structured SERP results for recipe and technique queries into the same index would let the product answer “which results surface for this technique” alongside “which clip demonstrates it.” The SERP monitoring solution delivers that feed in the same structured, timestamped shape the video records use, so it slots into the existing pipeline rather than a new one.

What would you do differently? Apply the channel-cap filter on day one, and start SERP visibility monitoring before launch rather than after — the baseline data we lack from the pre-launch weeks is the one thing money can’t retroactively buy. If a video search product is on your roadmap, the pilot playbook is now well-trodden: scope the shard, index, dedup, balance, and ship — six weeks, with the corpus doing the heavy lifting that would otherwise be your first two quarters.