These notes are my instrument to learn and share things I find beautiful. They're mostly for myself, but it would be great if people find them useful.
Posts
- RMSNorm backward (almost) without the column reduction
The RMSNorm backward pass is memory-bandwidth-bound, and the naive weight gradient doubles the traffic. Here is how to get rid of it.