MoE

Mixture of Experts

Architecture routing each input to a small subset of specialist sub-models.

When you'd see it: Discussions of how the largest current language models are built efficiently.

Why it matters: A Mixture of Experts routes each input to a small subset of specialist sub-networks instead of running the whole model every time, so it can be huge in total but cheap per query. It is a key trick behind several frontier models.