Engineering & AI Research
Notes on software systems, transformer math, and mechanistic interpretability. Currently focused on understanding how language models work from first principles — the math with intuition, not just the API.
Training an SAE from scratch — dynamics, dead features, and a flat frontier
What actually happens when you train a sparse autoencoder yourself: a staircase not a gradient, a window problem, and why a 50x change in lambda did almost nothing.
SAE interpretability on GPT-2: first results
What I found after running feature exploration and causal interventions on a pretrained sparse autoencoder. One feature surprised me.
Starting the interpretability project
Why I started reverse-engineering transformers from scratch, and what sparse autoencoders have to do with it.