A linear autoencoder compresses n sparse features into m hidden dimensions
(h = Wx) and reconstructs them (x̂ = ReLU(Wᵀh + b)).
Train it and watch the rows of W arrange into geometric structure — the model
packs more features than dimensions when they are sparse.
Feature geometry
rows of W in hidden space
Weight matrix W
rows = features, columns = hidden dims
W Wᵀ interference
diagonal = ∥Wᵢ∥², off-diagonal = overlap
Per-feature
norm ∥Wᵢ∥ and bias bᵢ
Loss
weighted MSE (log scale)