Skip to content
Artwork for Eye on AI Weekly Research Watch
Eye on AI Weekly Research Watch · August 20 · 2 min

Split the Labor: Separating Evidence Interpretation from Decision Aggregation

When LLMs are asked to synthesize conclusions from many sources, cramming everything into one prompt conflates two different jobs: understanding each piece of evidence and combining conclusions fairly. This paper identifies a subtle failure mode, "count-scale drift," where naive vote-counting shifts as more sources are added, and proposes separating interpretation from aggregation using a structured evidence format and calibrated probability pooling. Validated on a real medical/survival dataset, this approach could improve any multi-source reasoning system, from diagnostic AI panels to fraud detection to automated research synthesis tools. Authors: Zhelun Wu Paper: https://arxiv.org/abs/2608.14509v1

0:00-2:37

transcript

No transcript — this publisher did not publish one.

show notes

When LLMs are asked to synthesize conclusions from many sources, cramming everything into one prompt conflates two different jobs: understanding each piece of evidence and combining conclusions fairly. This paper identifies a subtle failure mode, "count-scale drift," where naive vote-counting shifts as more sources are added, and proposes separating interpretation from aggregation using a structured evidence format and calibrated probability pooling. Validated on a real medical/survival dataset, this approach could improve any multi-source reasoning system, from diagnostic AI panels to fraud detection to automated research synthesis tools.

Authors: Zhelun Wu

Paper: https://arxiv.org/abs/2608.14509v1