Transcript
Even with 1-bit weights, you still hit a wall. Not a memory wall this time. A math wall. Standard Transformer attention compares every token to every other token. Double the sequence length, and the attention work does not double. It quadruples. That is what quadratic scaling means in practice, Chris. A document twice as long costs four times as much to process. A document ten times as long costs a hundred times as much. Think of it like a dinner party where every guest must shake hands with every other guest. Ten guests means 45 handshakes. A hundred guests means nearly five thousand. The room does not scale. The math does not forgive. Last lecture landed on the Quantization First rule: if a model cannot survive ternary weight compression, it cannot scale economically. That solved the arithmetic cost. Now the problem shifts. Even a perfectly compressed model still pays the quadratic attention tax at inference time. Ternary weights reduce what you carry. Quadratic attention determines how long it takes to think. These are separate bottlenecks, and the second one is where the next generation of architectures is being built. Linear attention reorders operations by associating keys and values before applying the query, avoiding the full sequence-by-sequence matrix. This reduces the cost from quadratic to linear, offering a more efficient approach. But there is a real trade-off. Compressing past tokens into a fixed-size state reduces memory and inference cost. It can also lose some exact token-to-token information. That means the model may not retrieve a specific earlier token with perfect precision. You gain speed. You accept some compression loss. Before Mamba, there was S4. Structured State Space Sequence models use a parameterized state-space system to represent long-range dependencies through a recurrent state or an equivalent convolution. S4 uses a low-rank correction to a structured state-transition matrix, enabling efficient computation through specialized algebraic operations. The results were striking. S4 reported strong performance on long-range benchmarks, including the Path-X task with sequences of length 16,000. That is not a small number. Most Transformers at the time could not handle sequences anywhere near that length without enormous cost. Mamba took the S4 foundation and added something critical: selectivity. In standard state space models, the transition parameters are fixed. Mamba makes those parameters depend on the current input. That means the model can selectively propagate or suppress information as it moves through a sequence. Think of it like a river with intelligent gates. Each gate decides what flows forward and what gets filtered out, based on what just arrived. Mamba can run in recurrent form for efficient autoregressive inference, or in a convolution-like form for parallel training. The original Mamba study reported competitive language-modeling quality at linear rather than quadratic sequence scaling. Now, here is the counterintuitive part, Chris. A model that does not attend explicitly to every previous token can still perform well on long-context tasks. Why? Because most of what happened earlier in a sequence is not individually critical. The compressed state captures the pattern. But sometimes exact retrieval matters — a specific name, a precise number, a key fact stated earlier. That is where hybrid architectures come in. Research confirms that recurrence handles long-range state tracking efficiently, while attention remains useful when exact access to specific earlier tokens is important. Hybrid designs like Jamba combine Transformer attention blocks with SSM-style blocks, using each where it is strongest. Linear scaling introduces new challenges. Chip memory capacity limits context size, and memory bandwidth can bottleneck state updates, highlighting hardware constraints. And benchmark design has not fully caught up — many standard evaluations do not stress-test long-context retrieval in ways that reveal real-world failure modes. The practical benefit of linear-time architectures is also hardware-dependent. Recurrent scans reduce memory traffic, but their limited parallelism may prevent them from matching highly optimized GPU matrix multiplications in all scenarios. [short pause] The takeaway from this entire arc is a single rule: the Linearity Mandate. Future AI systems must make compute grow linearly with data length, not quadratically. Ternary weights cut the arithmetic cost per operation. Linear-time architectures cut the scaling cost across sequence length. Specialized hardware closes the gap between what algorithms demand and what silicon can deliver. These three levers — compression, linear scaling, and co-designed hardware — are not independent. They reinforce each other. Remember this, Chris: the next frontier is not a bigger model. It is a smarter one that scales without breaking the math.