Engineering & AI Research
Notes on software systems, transformer math, and mechanistic interpretability. Currently focused on understanding how language models work from first principles — the math with intuition, not just the API.
GPU Inference Optimization
Profiling down to the register, then an extension that broke my assumptions twice
Nsight Compute explains exactly why the kernel loses despite moving half the memory traffic — a register count, not a guess. Then a KV-cache extension delivers a real 2x win and one result I didn't expect at all.
Getting the kernel correct, then benchmarking it honestly
96/96 correctness checks pass, then the kernel loses to naive PyTorch on every single shape tested — and why that's a result to report, not a bug to quietly fix first.
Why attention is memory-bound, and what I built to prove it
Starting a FlashAttention-style kernel in Triton from scratch — the idea, the hardware reality behind it, and why beating PyTorch was never the goal.
Mechanistic Interpretability
Training an SAE from scratch — dynamics, dead features, and a flat frontier
What actually happens when you train a sparse autoencoder yourself: a staircase not a gradient, a window problem, and why a 50x change in lambda did almost nothing.
SAE interpretability on GPT-2: first results
What I found after running feature exploration and causal interventions on a pretrained sparse autoencoder. One feature surprised me.
Starting the interpretability project
Why I started reverse-engineering transformers from scratch, and what sparse autoencoders have to do with it.