Filtering the Noise: Candidate Generation and Retrieval

Transcript

SPEAKER_1: Last time we landed on something precise — mobile constraints are the actual architectural forcing functions behind the whole pipeline. Now we're at the heavy lifting: candidate generation. How does a system even begin narrowing down millions of items? SPEAKER_2: That framing is exactly why this stage exists. The key idea is that no single model can be both massive in scale and extreme in speed. So candidate generation is the early narrowing step — you reduce a huge item space to a shortlist before any expensive modeling runs. SPEAKER_1: How small does that shortlist get? SPEAKER_2: Usually from potentially billions down to a few hundred, maybe a few thousand candidates. The expensive scoring model is meant to run on that much smaller shortlist, not the full corpus. That separation is what makes the whole pipeline feasible. SPEAKER_1: And there are two main paradigms for building that shortlist — collaborative filtering and content-based filtering? SPEAKER_2: Exactly. Content-based filtering recommends items similar to what a user already engages with. Think of a user who watches a lot of short cooking videos — the system builds a feature profile and finds matching items. Dot product is one common similarity metric used to score those candidates. SPEAKER_1: And collaborative filtering doesn't look at item features at all? SPEAKER_2: Right — it uses patterns across many users' behaviors. If a large group of users who behave similarly also engaged with a certain article, the system infers you might too. It can be user-based or item-based, depending on whether similarity is computed between users or between items. SPEAKER_1: So which one wins in practice? SPEAKER_2: Neither alone. Hybrid approaches combine both, because each has gaps. Content-based filtering struggles when item metadata is sparse. Collaborative filtering struggles when a user is new. You blend them to cover each other's weaknesses. SPEAKER_1: Wait — that's the cold start problem. A brand new video has no interaction history, so collaborative filtering has nothing to work with. SPEAKER_2: [short pause] Exactly. For a newly uploaded video or a new creator profile, there's no behavioral signal yet. Systems use fallbacks — metadata embeddings from titles or tags, popularity priors, or creator priors if the creator has an existing track record. Some pipelines also reserve a small exploration bucket, maybe five to ten percent of candidates, specifically for new or unproven items. SPEAKER_1: So you're deliberately injecting uncertainty into retrieval. That's a real tradeoff. SPEAKER_2: A deliberate one. That kind of exploration can help new content gather feedback instead of being overlooked when the system lacks interaction history. The exploration budget keeps the catalog alive. Now, the question for everyone listening is how you actually search across millions of embeddings fast enough that no one notices. SPEAKER_1: Right — that's where two-tower models come in? SPEAKER_2: Yes. The architecture encodes users and items separately — one tower per side — each producing a dense vector embedding. Compatibility is scored using dot product or cosine similarity between those two vectors. The user embedding is trained from behavioral signals: taps, dwell time, skips, shares, follows. Positive interactions define what the model moves toward; skips define what it moves away from. SPEAKER_1: But computing similarity against millions of item vectors at request time — that's still a major speed challenge, right? SPEAKER_2: [inhale] Exactly why exact nearest-neighbor search is usually a bad idea at this scale. Even returning the mathematically most accurate matches, the latency is prohibitive. Candidate generation needs to complete within roughly a hundred milliseconds or less. So production systems use efficient retrieval methods to narrow the search quickly, instead of exhaustively comparing against the entire catalog. SPEAKER_1: Not perfect matches — good-enough matches, fast enough to be useful. And the ranker downstream compensates for any imprecision? SPEAKER_2: That's the counterintuitive part. A retrieval system returning slightly less individually relevant candidates can actually produce a better final feed after ranking. Diversity in the candidate pool gives the ranker more to work with. A perfectly homogeneous shortlist of the hundred most similar items often produces a repetitive feed — which was Greg's original complaint back in lecture one. [chuckle] SPEAKER_1: retrieval is about speed and coverage, not perfection. Get a diverse, good-enough shortlist into the ranker's hands within the latency budget. SPEAKER_2: That's it precisely. Candidate generation is an early filter against information overload — reducing a very large item space to a smaller shortlist before ranking. With that shortlist in hand, the real precision work begins. The ranker takes over, and that's where business logic, diversity rules, and deeper modeling enter the picture.