Why multi-head? And why attention is 'order-blind'
Last post: attention is one 'soft lookup'. So why multi-head at all? And why is it order-blind, needing positional encoding to rescue it? Two things, explained with minimal code.
llm
2 posts
Attention explained: it's just a soft lookup
Don't be scared by Transformer or 'Attention Is All You Need' — start with the smallest piece: attention. Explained with the hash table you already know, plus 20 lines of numpy so you can compute it yourself.
Subscribe
Subscribe via RSS to get new posts automatically.
Need something built?
Custom development and consulting for embedded, edge AI and agents.