On-Device Real-Time Subtitle Translation Optimized for Workloads Achieves 59.2% Win Rate Against Google Translate

On July 10, 2026, Tsz-To Wong presented a study on workload-driven optimization for on-device real-time subtitle translation, focusing on English-to-Traditional-Chinese in Taiwan. This research addresses significant challenges such as low latency, privacy constraints, and the need for short input and output translations.

Understanding the Optimization Challenges

The study highlights how traditional optimizations are often ineffective for on-device translations that require quick responses and short context. As the demand for real-time translation grows, optimizing for workloads becomes critical.

Wong's research indicates that after applying GGUF quantization, the relative cost of Transformer blocks diminishes, making vocabulary projection a more significant aspect of decode-time costs. This shift necessitates a new approach to translation models.

Implementation of the LocalSubs Model

To enhance translation efficiency, the research introduces the LocalSubs model, which employs a 64k-token subtitle-domain tokenizer, replacing the previous 151k-token vocabulary. This adjustment aims to streamline the translation process for short cues, which are most common in real-time settings.

Preliminary tests on a fixed 500-example subset of the OpenSubtitles2024 test set revealed that LocalSubs achieved a remarkable 59.2% tie-excluded win rate against Google Translate, as judged under GPT-4o pairwise assessments. Performance was notably robust on shorter cues, with a gradual decline observed as cue length increased.

Performance Metrics and Future Directions

Initial performance metrics from Apple M2 Metal indicate a 1.63× speedup for the 64k-vocabulary model compared to the 151k-vocabulary baseline. However, these latency results are preliminary, suggesting further refinements are necessary.

The findings underscore the potential of targeted optimizations in enhancing real-time translation applications, especially in environments where privacy and low latency are paramount. Future research will likely explore additional optimizations and broader applications of the LocalSubs model.