Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval
Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parame...