HERO: Benchmark Library Enhances Federated Continual Learning Evaluation

HERO is a new benchmark library designed for federated continual learning (FCL), introduced by a team of researchers including Thinh T. H. Nguyen and Le-Tuan Nguyen. This innovative library was submitted on June 13, 2026, to address the challenges faced in evaluating FCL across diverse data streams.

Understanding Federated Continual Learning

Federated continual learning evaluates how distributed clients can learn from changing data while retaining previous knowledge. Traditional methods often complicate comparisons due to varying datasets, task splits, and client data distributions. HERO aims to streamline these evaluations by clearly separating these components.

The main benchmark within HERO, HERO-Core, utilizes parameters such as α for client data skew and ρ for task-order mismatch. This allows for more accurate assessments of various FCL methods, especially on datasets like CIFAR-100 and TinyImageNet.

Key Features of HERO

The HERO library distinguishes itself by providing benchmark streams that are reproducible and setting-aware. It includes method implementations, configurations, and reporting scripts. This comprehensive approach helps researchers better understand the performance of different FCL strategies under various conditions.

Moreover, the library includes a graph-based case study on OGB-MolPCBA, demonstrating how domain shifts impact input distribution while keeping the prediction task constant. This case study illustrates the library's ability to expose challenges beyond image-based evaluations.

Impact on Future Research

The findings from HERO's evaluations reveal that method performance can vary significantly between easy and heterogeneous settings. For instance, average accuracy may not fully represent the performance of the bottom 10% of clients. Additionally, the results indicate that task-order mismatches can favor different strategies compared to synchronized evaluations.

HERO's release of benchmark streams and tools marks a significant advancement in the field of federated continual learning, promoting reproducibility and aiding researchers in their evaluations.