AI Wednesday 05 : Exploring Mixture-of-Experts (MoE)

By Sebin Thomas · March 25, 2026

Overview

This week's AI Wednesday focused on Mixture-of-Experts (MoE) — a sparse architecture that scales model capacity without proportionally scaling compute. We explored how routing and expert specialization work, looked at DeepSeek's MoE design, and compared it with other recent approaches including Chain of Experts and how Gemma models report effective parameters.

Topics

• How MoE architectures route tokens to specialized expert networks
• DeepSeekMoE: fine-grained expert segmentation and shared expert isolation
• Chain of Experts — sequential communication between experts within a layer
• Gemma and the idea of effective vs. total parameters in sparse models
• Why MoE matters for building large models at lower inference cost

Resources

• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models

Highlights

• MoE lets models carry far more parameters than are active per token — understanding routing and expert specialization is key to reading modern LLM architecture papers.

Next Week

• Topic: TITANS — Mamba, Titans, and the MIRAS Framework
• Host: Sebin Thomas