Tracked,
then suppressed.
Give an LLM a stream of updates to the same key and ask for its current value. It reliably recalls the first value it saw — but fails on the latest. Not because it can't find it: the right answer is computed inside the model, then erased before the output. Here's the failure, the mechanism, and a small fine-tune that fixes it.
In one glance
Key findings
| Models recall a key's first value but fail on its current value | gap +6 to +53 pp · 17/19 models above the line | → §2 |
| It's primacy, not "lost in the middle" | the first write is kept; later writes are lost | → §3 |
| It worsens with updates, not context length | current-value recall decays as N grows (<4% of context used) | → §4 |
| The correct value is computed, then suppressed | it surfaces in late layers (P≈0.5) then decays before the output | → §6 |
| The model can't tell it's about to be wrong | current-value-correctness probe sits at chance (58%) | → §7 |
| A small fine-tune closes the gap | gap → ~0; the late-layer suppression is removed | → §10 |
01 — Feel the bug
Same stream. One question flips the answer.
A key gets updated many times over, interleaved with others. Ask for its first value and the model nails it. Ask for its last value and it hands back something from the middle.
02 — It's not one model
Almost every model lands above the line.
Each dot is a model. Up = recalls the first value (FVQ). Right = recalls the last value (CVQ). The diagonal is "equally good at both." Almost everything sits above it: better at first than last. (A couple of small/base models invert — the paper's "reversal" regime.)
Show the per-model numbers
03 — Where it fails
It keeps the first write and loses the rest.
Ask for the value at every position, not just first and last. The baseline shows a sharp primacy spike at position 1, a faint recency bump at the end, and a dead zone in between — the model holds onto the first write and drops everything after. Flip +LoRA and the whole curve lifts: every position becomes retrievable.
Isn't this just "lost in the middle"?
No — and the shape of the curve is the proof.
- Driven by absolute position in a long context.
- U-shaped: the middle of the context is worst; both ends recalled well.
- Appears at long context lengths.
- Driven by update order, not context position.
- Monotone primacy: the first write is kept, the most recent is lost — the worst point is the end, not the middle.
- Appears in tiny prompts (<4% of the window) — it's interference, not length.
It's the opposite of human memory
- Strong recency: the most recent item is recalled best.
- Retroactive interference dominates — new information overwrites old.
- Strong primacy: the first value sticks, the current one is lost.
- Proactive interference dominates — old information blocks new.
04 — The more updates, the worse
Every extra update buries the answer deeper.
Fix the number of keys; grow the number of updates per key. First-value recall holds steady; current-value recall decays with load. The failure tracks how much is written after the target — not context length (these prompts use under 4% of the window).
05 — Why it matters
This is the everyday failure mode of "what changed?"
The benchmark is abstract, but the pattern is not. Any time a value is set and then updated in context, the model is at risk of answering with the stale original instead of the current one — and, as §7 shows, without any sign that it's unsure.
06 — Look inside
The right answer is there — then it gets erased.
Decode the model's hidden state at every layer: the correct value, the strongest competing value, and the total mass on any tracked value. Asking for the current value, the correct answer surfaces in the late layers — then gets suppressed before the output, and a competing value wins. Flip +LoRA and the suppression is gone.
Qwen2.5-3B-Instruct, the paper's §7 mechanism probe: arbitrary single-token values, strict single-token decoding through the unembedding. Absolute accuracy runs lower than the semantic task in §1–§4, but the surfaced-then-suppressed pattern is the mechanism the paper reports.
07 — What the model knows about itself
It can't tell when it's about to be wrong.
We train a tiny probe to read, from the hidden state, whether the model will answer correctly. For the first-value task the signal is there. For the last-value task it's at chance — until LoRA installs it in the very same layers.
08 — When it's born
The bias is learned during pretraining.
Re-evaluate SmolLM2 across its public pretraining checkpoints. Early on, first and current value are close. As training proceeds, first-value recall climbs while current-value recall lags — the asymmetry grows with training. It isn't bolted on by instruction tuning; it emerges from the pretraining objective itself.
09 — Can you prompt around it?
Formatting helps — but doesn't cure it.
Same updates, four layouts: Plain key:value, Labeled, Block-grouped, and Landmark markers. Block grouping helps a lot; the default Plain format is often the worst; no single format reliably closes the gap across models. Formatting is a band-aid — it never touches the underlying readout failure (and you can't always control the format your input arrives in).
10 — The fix
A small fine-tune closes the gap.
The skill was latent, not missing. A LoRA fine-tune — trained only on this task — doesn't teach the model anything new about finding values. It removes the late-layer suppression that was tearing down the correct answer.
Show the per-cell numbers (LoRA transfer table)
Where the fix acts
The fine-tune re-routes a handful of late-layer attention heads back onto the current value.
It isn't specific to attention
A LoRA on the MLP blocks closes the gap just as well — the capability is distributed, not owned by the attention heads.
Details
Methods, limitations & how to cite
The exact task & prompt
Each trial interleaves updates to K keys, each updated N times, shuffled so no two consecutive updates share a key. The model is asked for the first (FVQ), current/last (CVQ), or k-th (IVQ) value of one target key. "First/last" is defined by position in the presented stream.
System: You are a precise data extraction tool. Output ONLY a single word — the exact value requested. No other text, no explanation. User: Read the following key-value stream. Each key gets updated multiple times. gemstone: ruby fabric: silk gemstone: jade fabric: linen gemstone: opal gemstone: pearl What was the last value of gemstone? → expected: pearl
Methods
- Models: 20 in the main set — 14 open-weight (270M–9B: Qwen2.5, Qwen3.5, Gemma-3, TinyLlama, StableLM, Pythia) + 6 proprietary frontier (GPT-4.1/-mini, Claude-4.5-Haiku/-Sonnet, Gemini-2.5-Flash/-Pro). Mamba included separately as an SSM control.
- Datasets: Semantic-Multi (named categories → real values; §1–§4) and arbitrary single-token (the §6 mechanism probe).
- Scoring: behavioural = exact-match generation; logit-lens / probing = strict single-token argmax through the unembedding (a conservative lower bound).
- Cells: open-weight scored at K=5, N=30; proprietary at the harder K=20, N=50, matching the paper.
- Intervention: LoRA (rank 16, α=32) on the attention projections, trained on the arbitrary-token task and evaluated on the semantic task (cross-distribution transfer).
- Mechanism: logit lens, linear correctness probes, and attention routing on Qwen2.5-3B-Instruct; bootstrap 95% CIs (n=2000).
Limitations & honest caveats
- The §6 logit-lens/probing runs use the arbitrary-token probe; absolute accuracy is lower than the semantic task — the pattern (surfaced-then-suppressed) is the result, not the absolute numbers.
- Single-token argmax underestimates true generation accuracy; it's used because it's internally consistent across baseline/LoRA.
- Per-point confidence intervals are wide in the position curve (20 trials/position) and logit lens (100 trials/cell) — read the shapes, not individual points.
- A minority of small / base models show a reversal (CVQ > FVQ) at high load — the asymmetry is dominant, not universal.
- The fine-tune result is demonstrated on Qwen2.5-3B (and Gemma-3-4B) at specific cells; it is a proof that the skill is latent, not a production recipe.
Cite this work
This citation covers the original PI > RI finding, preprinted on arXiv — arxiv.org/abs/2603.00270. The mechanistic analysis and fine-tune results shown on this page (§6–§10) are extended, ongoing work; a citation for that paper will be added here once it's available.
@misc{chattaraj2025trackedsuppressed,
title = {Tracked but Suppressed: How LLMs Fail Current-Value Retrieval},
author = {Chattaraj, Sourav and Raj, Kanak},
year = {2025},
eprint = {2603.00270},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2603.00270}
}