Probing: classifiers and truthfulness

Trained linear probes on Gemma’s residual stream to test what’s linearly decodable at each layer, starting with sentiment before touching truthfulness probing. The methodological catch that’s easy to miss coming from standard ML eval habits: high probe accuracy only shows the information is linearly present, not that the model uses it for anything downstream.

Burns et al.’s “Discovering Latent Knowledge” motivation stuck with me — they specifically avoid labeled truthfulness data because supervised probes can key onto spurious correlates of the label instead of the concept itself. There’s a sharper version of this problem too: multiple different linear directions can often achieve near-identical probe accuracy on the same data. “The truth direction” as singular and well-defined is doing more work in a lot of writeups than the non-identifiability of the probe direction really supports.

Getting close to the end of this stretch of the curriculum now, which got me thinking about actually applying to ARENA this year rather than just working through material on my own indefinitely — structured feedback on this would help a lot more than another few weeks of solo reading.