Induction heads, first pass

Started induction heads / circuit finding, following Olsson et al., “In-context Learning and Induction Heads”. Mechanism is a clean two-head composition:

  • previous-token head writes “the token before me was X” into the residual stream
  • induction head K-composes with that write, searches for prior occurrences of the current token, copies whatever followed it last time
  • [A][B]...[A] → [B]

Used a repeated-random-token sequence (the standard diagnostic) instead of IOI prompts — much cleaner signal, a handful of heads have textbook induction attention patterns as soon as I plotted per-head scores across layers.