Skip to content
Artwork for My Weird Prompts
My Weird Prompts · Today · 23 min

Using LLMs as Annotators for Silver Training Data

When an LLM labels data for a downstream classifier, it isn't acting as a judge or classic weak supervision—it's an LLM-as-annotator producing silver labels. Exploring a Romanized Hebrew span-detection pipeline built with Claude Opus, this episode separates few-shot annotation and human-AI refinement loops from Snorkel-style labeling functions. It also covers the Alternative Annotator Test for spot-checking and an Activision Call of Duty study showing that using a seed set to search 66 prompt configurations outperforms fixed few-shot examples. Key sources: • Calderon, Reichart & Dror, The Alternative Annotator Test for LLM-as-a-Judge, ACL 2025 Long Papers pp. 16051 — https://aclanthology.org/2025.acl-long.782/ • Kocielnik et al., Prosocial Behavior Detection in Player Game Chat, arXiv:2508.05938v1 — https://arxiv.org/abs/2508.05938 • Mohta, Ak, Xu & Shen (Amazon), Are large language models good annotators? PMLR v239 (NeurIPS 2023 workshop) — https://proceedings.mlr.press/v239/mohta23a/mohta23a.pdf Episode 6630 — myweirdprompts.com/n/6630 (permanent short link: myweirdprompts.com/732934) Listen on Android: MWP FM Player — myweirdprompts.com/apk

0:00 · Intro-23:38

transcript

No transcript — this publisher did not publish one.

show notes

When an LLM labels data for a downstream classifier, it isn't acting as a judge or classic weak supervision—it's an LLM-as-annotator producing silver labels. Exploring a Romanized Hebrew span-detection pipeline built with Claude Opus, this episode separates few-shot annotation and human-AI refinement loops from Snorkel-style labeling functions. It also covers the Alternative Annotator Test for spot-checking and an Activision Call of Duty study showing that using a seed set to search 66 prompt configurations outperforms fixed few-shot examples.

Key sources:
• Calderon, Reichart & Dror, The Alternative Annotator Test for LLM-as-a-Judge, ACL 2025 Long Papers pp. 16051 — https://aclanthology.org/2025.acl-long.782/
• Kocielnik et al., Prosocial Behavior Detection in Player Game Chat, arXiv:2508.05938v1 — https://arxiv.org/abs/2508.05938
• Mohta, Ak, Xu & Shen (Amazon), Are large language models good annotators? PMLR v239 (NeurIPS 2023 workshop) — https://proceedings.mlr.press/v239/mohta23a/mohta23a.pdf

Episode 6630 — myweirdprompts.com/n/6630 (permanent short link: myweirdprompts.com/732934)

Listen on Android: MWP FM Player — myweirdprompts.com/apk

chapters

7 chapters