What is a Mixture of Experts (MoE)?

For years, the standard way to make an AI model smarter was to make it larger. However, in a traditional "dense" model, every single parameter must be calculated for every single word the model processes. This creates a massive "compute tax"-as the model gets bigger, it becomes exponentially more expensive and slower to run. Mixture of Experts (MoE) is the architectural breakthrough that broke this linear relationship between model size and speed.

Sparse vs. Dense Architectures

In a standard dense Transformer, the "knowledge" of the model is spread across monolithic layers. Every token (word or character) passes through every neuron. An MoE model, by contrast, is sparse. It consists of many specialized sub-networks, known as Experts. Instead of activating everything, the model uses a "Router" to send each token to only the two or three most relevant experts. This allows a model to have the knowledge base of a trillion-parameter system while only using the computing power of a much smaller model.

The Role of the Router

The heart of the MoE system is the Gating Network or Router. For every input, the router performs a quick calculation to determine which experts have the necessary "skills" to handle the data. This isn't just about topic (e.g., sending math to a math expert); research shows that experts often specialize in syntax and structure, such as handling specific types of punctuation, verbs, or abstract logic. This selective activation is a form of Conditional Computation, where the model's path is determined dynamically in real-time.

The VRAM vs. Compute Trade-off

While MoE models are fast to run (low compute), they are heavy to store. Because the model needs to have all its experts ready at a moment's notice, the entire model-including the "inactive" experts-must be loaded into the GPU's memory (VRAM). This is why a model like Mixtral 8x7B runs as fast as a 13B model but requires the memory of a 47B model.

This trade-off suggests that the future of AI scaling may be less about processor speed and more about memory bandwidth. As we build even larger sparse models, the challenge shifts from doing the math to moving the data. Will we eventually develop hardware that can store "sleeping" experts more efficiently?