Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU SparseAutoencoders

JumpReLU sparse decompositions: framing and motivation At first glance, the paper addresses a familiar tension in representation learning: achieving faithful linear decompositions of language-model activations while keeping those decompositions spars...