Director, a new distributed Mixture-of-Experts (MoE) serving system, was introduced by a team of researchers on June 13, 2026. This innovative system aims to minimize end-to-end latency through online proactive expert placement, addressing challenges in expert migration and request patterns.
Understanding the Proactive Expert Placement
The core of Director's approach lies in its ability to predict expert activation patterns for incoming requests. By utilizing a lightweight cascaded predictor or a low-bit quantized replica, the system enhances the efficiency of expert placement on GPUs. This technique is crucial as it allows for adjustments based on rapidly changing request patterns, which traditional methods often fail to accommodate.
Notably, the system's online migration module executes expert migrations during compute-bound phases, ensuring minimal disruption. This capability enables the system to enact changes with near-zero downtime, a significant improvement over existing solutions that struggle with latency issues.
Performance Improvements of the Director System
In extensive experiments, Director demonstrated a remarkable reduction in end-to-end latency, achieving improvements between 11% and 55% for popular MoE models such as Mistral, DeepSeek, and Qwen. These results indicate a substantial advancement in the efficiency of distributed MoE serving.
