The Telephone Effect: Citation Drift in Multi-hop Summarization
When claims are propagated through chains of summarization, the final citation often no longer supports the original statement. We quantify this drift across chain lengths and show span-anchored re-verification at each hop eliminates most of it.
2/2
claims supported
2 · 2
experiments · runs
1
span citations
1
referenced papers
Claims & evidence
Status is derived from linked evidence, never asserted.
thesis
Citation support degrades monotonically with each summarization hop.
§1supported
- ⊕ supports001-chain-lengthremote-verifiedsupport_rate_hop1 0.91support_rate_hop4 0.55confirmatory
contribution
Re-verifying spans at each hop restores over 90% of lost support.
§5supported
- ⊕ supports002-hop-verificationremote-verifiedsupport_rate_hop4 0.88confirmatory
Experiments & results
Every number traces to a recorded run with git sha, env hash, and seed.
| Experiment | Hypothesis | Run | Metrics |
|---|---|---|---|
001-chain-length Support vs. chain length | Support rate decreases with hop count. | 31d0a2eseed 13remote-verified | support_rate_hop1 0.91 support_rate_hop4 0.55 |
002-hop-verification Per-hop re-verification← 001-chain-length | Verifying at each hop prevents drift. | 77aa1b0seed 13remote-verified | support_rate_hop4 0.88 |
Papers & span citations
Citations anchor to the exact quoted span, so every quote can be checked against its source.
Enabling Large Language Models to Generate Text with CitationsTianyu Gao, Howard Yen, Jiatong Yu et al. · 2023 · EMNLP
gao2023alcecitation quality remains far from satisfactory
support: partial · 0.58