Skip to content
Sign in
Artwork for Clown Cast
Clown Cast · Yesterday · 18 min

The Expediter: Why Your Feed Needs a Policy Layer

A ranker scores each item individually. But you experience a page. In part four of the recommender series, Joey explores the re-ranker—the policy engine that handles diversity, freshness, deduplication, and editorial overrides. It's the difference between individual station cooks and the expediter at the pass. [00:00:00] YouTube's three failed fixes. [00:01:35] The ranker grades the exam, the re-ranker arranges the seating chart. [00:06:28] Why YouTube's best result never shipped. [00:11:25] How hand rules blocked better ML models. [00:17:27] The system becomes a curator. --- Sources & further reading: • Mark Wilhelm, Ajith Ramanathan, Alexander Bonomo, Sagar Jain, Ed H. Chi, Jennifer • Jaime Carbonell, Jade Goldstein, *The Use of MMR, Diversity-Based Reranking for Reordering • Abhinandan Das, Mayur Datar, Ashutosh Garg, Shyam Rajaram, *Google News Personalization • Fangzhao Wu et al., MIND: A Large-scale Dataset for News Recommendation, ACL 2020 • Gurmeet Singh Manku, Arvind Jain, Anish Das Sarma, *Detecting Near-Duplicates for Web • Yaron Greif, Pinterest Engineering, "Using PID controllers to diversify content types on • Robert B. Cialdini et al., Basking in Reflected Glory: Three (Football) Field Studies • All fetched and quoted on 2026-10-15. • Gillenwater, Practical Diversified Recommendations on YouTube with Determinantal Point Processes • CIKM 2018, doi:10.1145/3269206.3272018 — — pointwise-sorting: https://jgillenw.com/cikm2018.pdf • problem, last-layer DPP, windows, Table 1, +0.52% watch time, heuristics critique. *ACM PDF 403 • author copy read.* • Documents and Producing Summaries*, SIGIR 1998 • MMR: https://www.cs.cmu.edu/~jgc/publication/MMR_DiversityBased_Reranking_SIGIR_1998.pdf • definition, λ behavior, five-user pilot. • Scalable Online Collaborative Filtering*, WWW 2007 • item churn, time decay, Popular baseline and: https://www2007.cpsc.ucalgary.ca/papers/paper570.pdf • ~38% win, stories as article clusters. • dataset scale, 84.5% under two days, news cold: https://aclanthology.org/2020.acl-main.331.pdf • start. • Crawling*, WWW 2007 — — simhash, 64-bit / k=3: https://research.google.com/pubs/archive/33026.pdf • over 8B pages. • home feed," Pinterest Engineering Blog, 2020-07-10 • — hard-coded insertions, delayed models, global share targets, PID boosts, 4–12% range. • [primary, abstract only] Tien T. Nguyen, Pik-Mai Hui, F. Maxwell Harper, Loren Terveen, Joseph A. • Konstan, *Exploring the filter bubble: the effect of using recommender systems on content • diversity*, WWW 2014 — — "slightly: https://api.openalex.org/works/doi:10.1145/2566486.2568012 • narrowing" finding. Full text 403. • JPSP 34(3), 366–375, 1976: https://www2.psych.ubc.ca/~schaller/528Readings/Cialdini1976.pdf • apparel after wins; "we" 26% vs 13.5%. • [primary, abstract only] Jonathan A. Jensen et al., "Forty years of BIRGing," Journal of Sport • Management 30(2), 149–161, 2016 — — Study 1: https://repository.lsu.edu/kinesiology_pubs/364 • reproduced, Study 2 not. This podcast episode was fully generated by AI — research, script, voices, and production. Built with Claude, Piper TTS, and automated pipeline tooling.

0:00-18:34
Play in

transcript

No transcript — this publisher did not publish one.

show notes

A ranker scores each item individually. But you experience a page. In part four of the recommender series, Joey explores the re-ranker—the policy engine that handles diversity, freshness, deduplication, and editorial overrides. It's the difference between individual station cooks and the expediter at the pass.

[00:00:00] YouTube's three failed fixes. [00:01:35] The ranker grades the exam, the re-ranker arranges the seating chart. [00:06:28] Why YouTube's best result never shipped. [00:11:25] How hand rules blocked better ML models. [00:17:27] The system becomes a curator.

---
Sources & further reading:
• Mark Wilhelm, Ajith Ramanathan, Alexander Bonomo, Sagar Jain, Ed H. Chi, Jennifer
• Jaime Carbonell, Jade Goldstein, *The Use of MMR, Diversity-Based Reranking for Reordering
• Abhinandan Das, Mayur Datar, Ashutosh Garg, Shyam Rajaram, *Google News Personalization
• Fangzhao Wu et al., MIND: A Large-scale Dataset for News Recommendation, ACL 2020
• Gurmeet Singh Manku, Arvind Jain, Anish Das Sarma, *Detecting Near-Duplicates for Web
• Yaron Greif, Pinterest Engineering, "Using PID controllers to diversify content types on
• Robert B. Cialdini et al., Basking in Reflected Glory: Three (Football) Field Studies
• All fetched and quoted on 2026-10-15.
• Gillenwater, Practical Diversified Recommendations on YouTube with Determinantal Point Processes
• CIKM 2018, doi:10.1145/3269206.3272018 — — pointwise-sorting: https://jgillenw.com/cikm2018.pdf
• problem, last-layer DPP, windows, Table 1, +0.52% watch time, heuristics critique. *ACM PDF 403
• author copy read.*
• Documents and Producing Summaries*, SIGIR 1998
• MMR: https://www.cs.cmu.edu/~jgc/publication/MMR_DiversityBased_Reranking_SIGIR_1998.pdf
• definition, λ behavior, five-user pilot.
• Scalable Online Collaborative Filtering*, WWW 2007
• item churn, time decay, Popular baseline and: https://www2007.cpsc.ucalgary.ca/papers/paper570.pdf
• ~38% win, stories as article clusters.
• dataset scale, 84.5% under two days, news cold: https://aclanthology.org/2020.acl-main.331.pdf
• start.
• Crawling*, WWW 2007 — — simhash, 64-bit / k=3: https://research.google.com/pubs/archive/33026.pdf
• over 8B pages.
• home feed," Pinterest Engineering Blog, 2020-07-10
• — hard-coded insertions, delayed models, global share targets, PID boosts, 4–12% range.
• [primary, abstract only] Tien T. Nguyen, Pik-Mai Hui, F. Maxwell Harper, Loren Terveen, Joseph A.
• Konstan, *Exploring the filter bubble: the effect of using recommender systems on content
• diversity*, WWW 2014 — — "slightly: https://api.openalex.org/works/doi:10.1145/2566486.2568012
• narrowing" finding. Full text 403.
• JPSP 34(3), 366–375, 1976: https://www2.psych.ubc.ca/~schaller/528Readings/Cialdini1976.pdf
• apparel after wins; "we" 26% vs 13.5%.
• [primary, abstract only] Jonathan A. Jensen et al., "Forty years of BIRGing," Journal of Sport
• Management 30(2), 149–161, 2016 — — Study 1: https://repository.lsu.edu/kinesiology_pubs/364
• reproduced, Study 2 not.

This podcast episode was fully generated by AI — research, script, voices, and production. Built with Claude, Piper TTS, and automated pipeline tooling.

more episodes

All episodes