My blogs
Explore by topic
Statistics & Inference (Theory)
Modeling
Systems & Infrastructure
Interpretability & Understanding
AI for Science (Domain)
Agents & Scientific Workflows
-
Personal Thoughts on AI Agents for Science
Thoughts from applying AI agents to drug discovery: verifiable problems, incomplete scientific labels, statistical evaluation, and experimental feedback.
-
From KV Cache to a State Matrix: Linear Attention and DeltaNet
Notes on viewing linear attention as a compressed key-value state and DeltaNet as an error-correcting update to that state.
-
From Memory to Photonics: Solving the Next Bottleneck in AI Scaling
A systems view connecting FlashAttention's IO-aware lesson to distributed communication, WDM, optical I/O, and compact inverse-designed photonic devices.
-
Training a Language Model from Scratch (Part 2: FlashAttention and Device Memory)
A future-me note on why naive attention becomes a device-memory problem, how FlashAttention uses tiling, online softmax, log-sum-exp, and recomputation, and how Triton exposes the tile-level implementation model.
-
Training a Language Model from Scratch (Part 1: Building Blocks)
A future-me refresher on the main pieces behind a small Transformer language model: byte-level BPE, embeddings, RoPE, attention, normalization, loss, optimization, and decoding.