24 min • 1 lectures
This course explains the technical pipeline of Large Language Model (LLM) inference, moving from the initial user prompt to the final generated response. It covers the fundamental steps of tokenization, embedding, and the transformer architecture. Listeners will learn how self-attention allows tokens to reference earlier context and how autoregressive decoding functions by predicting a single next token at a time. The discussion clarifies that the speed of an LLM is determined by its ability to manage memory-intensive operations during the prefill and decoding stages. By viewing the model as a sequential drafting system rather than a simultaneous writer, the content identifies the physical bottlenecks that occur when moving data across a GPU. The curriculum also focuses on modern optimization techniques used to scale inference for production environments. Key concepts include Key-Value (KV) caching, which stores previous attention states to prevent redundant calculations, and FlashAttention, which uses tiled blocks to reduce high-bandwidth memory movement. The course details PagedAttention and vLLM, explaining how virtual memory management increases throughput for multiple simultaneous users. Additionally, it examines speculative decoding, where a smaller model drafts tokens for a larger model to verify, and quantization, which reduces the precision of weights to save memory. These systems demonstrate that efficient LLM performance relies on sophisticated engineering to ensure the processor never waits for data.
