Sparse mixture-of-experts: routing capacity, not just adding it

Every parameter in a standard Transformer runs on every token. If a feed-forward block has a billion parameters, every one of them multiplies against every token that passes through, whether that token needed a billion parameters' worth of computation or a tenth of that. Sparse mixture-of-experts (MoE) architectures reject that assumption directly: they make capacity large and compute conditional, so a token pays only for the small part of the network it actually uses.

Adding parameters and adding compute are usually the same decision. Mixture-of-experts is what happens when a team deliberately separates them.

The mechanism: a router replaces a single feed-forward block

Switch Transformer ↗, from Fedus, Zoph, and Shazeer, replaces the single feed-forward network inside a Transformer layer with several parallel feed-forward networks — experts — plus a small router network. For each token, the router computes a score for every expert and sends the token to the top-scoring one (Switch uses top-1 routing; earlier mixture-of-experts work commonly used top-2). Only the selected expert's weights process that token for that layer; the rest sit idle for this step.

Mixtral of Experts ↗, from the Mistral AI team, applies the same idea at a scale practitioners actually deploy. Each layer holds eight experts; the router selects two per token. The published numbers make the trade concrete: the model exposes 47 billion total parameters, but only 13 billion of them are active for any single token. The paper reports quality that outperforms or matches Llama 2 70B and GPT-3.5 across evaluated benchmarks, and specifically outperforms Llama 2 70B on mathematics, code generation, and multilingual tasks — at roughly a fifth of the active compute.

Architecture / conditional compute

One token, two experts, constant compute

FIG 01 - MOTION

For a layer with n experts and top-k routing, the output for token x is a weighted combination of the selected experts:

y = Σ (i ∈ top-k) gᵢ · Eᵢ(x)

where gᵢ is the router's gate weight for expert i and Eᵢ is that expert's feed-forward function. Total parameter count scales with n; compute per token scales with k. That single equation is the entire economic argument for MoE: capacity and compute stop being the same knob.

Routing is learned, and learned things can fail

The router is trained jointly with the rest of the model, not designed by hand, and a router left unconstrained will often converge on a small favorite subset of experts — sending them most tokens while starving the rest of gradient signal. Both the Switch Transformer and Mixtral papers address this with an auxiliary load-balancing loss added to the training objective, penalizing uneven routing so that expert utilization stays roughly even across the batch.

This detail matters operationally, not just academically. An imbalanced router produces a model whose effective capacity is smaller than its parameter count suggests — some experts become undertrained specialists that are rarely invoked, while a handful absorb a disproportionate share of the model's actual behavior. Standard aggregate accuracy metrics do not surface this directly; it shows up as inconsistent quality on inputs that happen to route to a neglected expert.

Economics / capacity versus compute

Bigger pool, same per-token bill — if the router behaves

FIG 02

Capacity and compute are different curves in production

The Mixtral numbers — 47B total, 13B active — describe two separate constraints a serving team has to plan for independently:

• Memory footprint follows the total parameter count. All experts typically need to stay resident, whether in accelerator memory or distributed across devices, because any token in the next batch might route to any of them.
• Latency and throughput follow the active parameter count. This is the number that actually determines how fast a forward pass runs.
• Training cost sits between the two: gradients flow only through the experts a token actually visited, but the full parameter set still has to be stored, checkpointed, and optimized.

A team that budgets serving hardware using only the active-parameter number will be surprised by the memory bill. A team that budgets using only the total-parameter number will underestimate how fast the model can actually respond.

What this changes in production

Frontier-scale assistants

Consumer- and enterprise-facing assistants that need broad world knowledge, wide language coverage, and low latency are the clearest fit for MoE: the model can hold far more learned capacity than a dense model at the same serving speed, provided the accelerator memory to hold every expert is available. The quality-per-active-FLOP advantage is the entire reason to accept the added routing complexity.

Multilingual and multi-domain platforms

A dense model serving many languages or domains must represent all of that knowledge inside one shared parameter set that every input pushes through. A well-balanced MoE model can let different experts specialize — implicitly, without hand-labeled domain routing — on different linguistic or topical territory, without paying for a separate model per domain or a proportional latency increase per language added.

Cost-tiered inference products

A platform offering multiple quality tiers can use MoE to offer near-frontier quality at a serving cost closer to a much smaller dense model, since customers are ultimately billed and rate-limited on active compute, not total parameters. This only holds if the provider's own infrastructure economics also track active compute rather than total footprint — otherwise the memory cost of holding every expert erodes the advantage being passed to customers.

A production checklist

Before routing production traffic through a mixture-of-experts model, the team should be able to answer:

• What load-balancing loss coefficient was used during training, and is expert utilization actually even across representative traffic — not just the training distribution?
• What is the total memory footprint of every resident expert, independent of how few are active per token?
• Does the serving infrastructure support conditional computation efficiently, or does it pay the dense-model cost anyway because of how batching is implemented?
• Has quality been evaluated per-expert or per-routing-path, or only in aggregate — could a specific input class be silently routed to an undertrained expert?
• If fine-tuning this model later, will fine-tuning preserve router balance, or can it collapse routing onto fewer experts than the base model used?
• Is the added engineering complexity of routing, expert placement, and load balancing justified by a measured quality or cost advantage over the best available dense model at the same active-compute budget?

Research referenced

• Fedus, Zoph, and Shazeer (2021), Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity ↗.
• Jiang et al. (2024), Mixtral of Experts ↗.