
Using LLMs as Annotators for Silver Training Data
transcript
show notes
When an LLM labels data for a downstream classifier, it isn't acting as a judge or classic weak supervision—it's an LLM-as-annotator producing silver labels. Exploring a Romanized Hebrew span-detection pipeline built with Claude Opus, this episode separates few-shot annotation and human-AI refinement loops from Snorkel-style labeling functions. It also covers the Alternative Annotator Test for spot-checking and an Activision Call of Duty study showing that using a seed set to search 66 prompt configurations outperforms fixed few-shot examples.
Key sources:
• Calderon, Reichart & Dror, The Alternative Annotator Test for LLM-as-a-Judge, ACL 2025 Long Papers pp. 16051 — https://aclanthology.org/2025.acl-long.782/
• Kocielnik et al., Prosocial Behavior Detection in Player Game Chat, arXiv:2508.05938v1 — https://arxiv.org/abs/2508.05938
• Mohta, Ak, Xu & Shen (Amazon), Are large language models good annotators? PMLR v239 (NeurIPS 2023 workshop) — https://proceedings.mlr.press/v239/mohta23a/mohta23a.pdf
Episode 6630 — myweirdprompts.com/n/6630 (permanent short link: myweirdprompts.com/732934)
Listen on Android: MWP FM Player — myweirdprompts.com/apk





