I go through architectures used in sequence modelling: FFN, CNN, RNN, SSM, and Transformers along with many efficiency optimization attempts. I provide intuitive understanding of how they work, and analyze their strengths and weaknesses. All while paying special attention (pun intended) to their computational and memory complexities.
Train a model to generalize human preference, use it to align your LLM.
I go through the research journey that lead into Mamba. I first review SSMs, then explore the models: HiPPO, S4, DSS, and finally Mamba.
Not a normal post: As I study for ML interviews, I’ll add notes here of stuff not worthy of a full post but interesting nonetheless.
I go through many core topics of ANN optimization: Gradient descent, the chain rule, backpropagation and computational graphs.
I go through the main architectural differences between a CPU and a GPU and give some intuition on how CUDA programming works.
Starting from dot-product attention, I present and give intuitive understanding of the main variants of the attention mechanism, namely: Multi-Head, Multi-Query, Grouped-Query, and Multi-Head Latent attentions.
What if instead of doing gradient descent on model parameters, we do gradient ascent on the model input?
The very basics for LeetCode success using Python.
I give an intuitive understanding of sinusoidal positional encodings, RoPE, and ALiBi. I finally asses their contribution to the learning process wrt not using any encoding or learned encodings.
