Steal This Design Doc: Training Data for a Video Recommender That Doesn’t Get Stale

Most video recommenders decay not because the model is wrong but because the training data froze — one corpus, one distribution, one moment in time. Below is a complete, adaptable design doc for the data layer of a video recommendation system: sourcing, stratification, deduplication, freshness cadence, and metrics — written the way an internal design doc is written, so you can adapt it rather than start from a blank page.

Design docs are stealable in a way tutorials aren’t: they show the constraints and the rejections, not just the happy path. This one is a generalized composite of the data designs behind short-video and long-form recommenders, built on the sourcing options available today.

Context

A video product’s recommendation quality is a function of three data properties, in order: coverage of the content distribution (what exists), freshness of the engagement signal (what’s happening now), and balance across creators and niches (what the model doesn’t overfit). The first and third come from the corpus; the second comes from your event stream. This doc covers the corpus.

Goals and Non-Goals

Goals:

• A training corpus that represents the deployed content distribution across languages, niches, and formats
• A stratified sampling scheme that prevents creator over-representation
• A refresh cadence matched to content velocity (weeks, not quarters)
• Provenance attached to every record, because recommender audits increasingly ask where training data came from

Non-goals (explicitly rejected):

• Building proprietary collection infrastructure before licensing breadth — two quarters of engineering before the first training run is the wrong sequencing
• Chasing total corpus size as a metric — a billion videos from a narrow creator distribution trains a niche model with a big number attached
• Real-time corpus updates — engagement signal is real-time; the corpus refreshes weekly, which is enough for content distribution

Data Requirements

The corpus needs four properties, each mapping to a concrete sourcing decision:

Sampling Design

The core decision in recommender training data is stratification. Two layers:

By channel, with a cap. Without a channel cap, the corpus is dominated by high-volume uploaders and the model over-fits their style. The cap is a hyperparameter of your content strategy, not a technical constant — start at a 2× overweight for the long tail and tune by niche.

By deployment distribution. Match the corpus mix to what your product actually serves: if 30% of your traffic is tutorial content, a corpus that’s 5% tutorials trains a model surprised by your own users.

# Illustrative: shard → stratified training batches
def build_batch(shard):
    records = [r for r in shard
               if r.get("caption_text")          # aligned pairs only
               and r["duration_seconds"] <= 900]
    stratified = (stratify(records, by="channel", cap=long_tail_weight(2.0))
                         .then(by="language", target=DEPLOYMENT_MIX)
                         .then(by="duration_bucket"))
    return [featurize(r) for r in stratified]

Deduplication Plan

Near-duplicate video — re-encodes, crops, re-uploads — inflates the popular tail and skews engagement priors. Two passes:

1. Vendor-side: ask whether the corpus pipeline performs similarity dedup and at what residual rate (this is question 8 of any dataset due-diligence list; a vendor that hasn’t thought about it is telling you about their collection process).
2. Training-boundary: content-hash matching on fingerprints, plus channel-aware ranking so originals outweigh re-uploads. The channel lineage in each record is what makes “original versus re-upload” decidable rather than guessed.

Freshness Cadence

Corpus refresh is scheduled, not heroic. The operating rhythm that works: weekly incremental shards for fast-moving niches, monthly full-distribution refresh, quarterly rebalancing of the stratification weights against the deployed mix. Providers operating both datasets and collection infrastructure can run the recurring jobs as a managed service — the same cadence discipline that makes SERP monitoring work for search teams, applied to corpus maintenance.

Metrics: How You Know the Data Layer Is Working

• Distribution match score: KL divergence between corpus niche mix and deployed content mix, trended weekly
• Duplication rate: near-duplicate share per shard after both dedup passes, target under a few percent
• Cold-start coverage: fraction of newly-deployed content niches already represented in the corpus — the metric that predicts whether new content categories get fair ranking
• Engagement transfer: offline metric on a fresh holdout (never a slice of training — see any evaluation-contamination writeup for why) versus online A/B lift

Risks and Mitigations

FAQ

Why not just train on our own platform’s data?
You should — first-party engagement data is the strongest signal. But it covers what your product already surfaces, which makes it a mirror, not a map: new niches, languages, and formats are invisible until someone else’s users discover them. Corpus breadth is how a recommender sees past its own traffic.

How does this relate to search-side intelligence?
Directly: the demand side of recommendation is query behavior. Watching which topics and queries trend — through structured SERP monitoring at about $0.70 per 1,000 responses — tells the corpus team which niches to refresh harder, weeks before your own engagement data would. Distribution intelligence for a video product is corpus data plus search data, in the same warehouse.

What’s the pilot?
Two weeks: license a filtered shard matching your deployment mix, run the existing model’s next training on it, and measure cold-start coverage and engagement transfer against the current corpus. If the delta is visible, the full design above is justified; if it isn’t, you learned cheap.

Is per-record pricing actually significant at corpus scale?
It’s the enabler: at roughly $0.25 per 1,000 records, a million-record filtered shard costs a few hundred dollars — so stratification and refresh cadence become data decisions, not budget negotiations. Storage and training compute still dominate total cost; the corpus is the cheapest lever in the whole system, which is exactly why it should be the best-instrumented one. For the demand-side feed, continuous SERP data crawling prices the same way — per structured response, on a schedule.

Take the doc, replace the deployment mix with yours, and start with the two-week pilot. The sections you’ll fight about internally are the channel cap and the refresh cadence — those are product decisions wearing engineering clothes, and the design doc’s real job is making that argument once, in writing, so you don’t relitigate it every quarter.