MedRealMM, a new benchmark for online medical consultation in China, was introduced on July 10, 2026. This project aims to address the limitations of existing benchmarks that do not align well with real clinical practices. Developed by a team of researchers led by Runhan Shi, this benchmark utilizes genuine patient-doctor interactions collected from a nationwide Chinese internet hospital.
Understanding the MedRealMM Benchmark
The MedRealMM benchmark focuses on multimodal online medical consultations, integrating both text and images to create a more realistic evaluation of large language models (LLMs). Traditional benchmarks often rely on synthetic data, which fails to capture the complexities of real-world clinical interactions. By using a Multimodal Clinical Challenge Point (MCCP) extraction framework, the team identifies critical moments in consultations and standardizes response generation tasks.
The dataset comprises 5,620 authentic multimodal cases across 64 clinical departments. Each case is evaluated using a rubric developed by physicians, rewarding clinically desirable responses while penalizing unsafe or contradictory answers. This approach aims to improve the quality of AI-generated clinical advice.
Evaluation of Large Language Models
The research evaluates 19 different LLMs, including both text-only and multimodal systems. Initial findings reveal that integrating image information significantly enhances the clinical performance of these models. However, despite some advanced models meeting or exceeding positive clinical criteria compared to human physicians, they also produce more negative outcomes, highlighting a critical area for improvement in safety-sensitive applications.
As AI continues to play a larger role in healthcare, the importance of reliable benchmarks like MedRealMM cannot be overstated. They provide a framework for assessing the capabilities of AI in real-world medical settings, ensuring that these technologies can be safely and effectively integrated into clinical practice.
Availability and Future Directions
The MedRealMM dataset will be publicly accessible on Hugging Face, aiming to facilitate further research and development in the field of AI-driven medical consultation. As the landscape of online healthcare evolves, benchmarks like this will be essential for guiding the deployment of safe and effective AI solutions.
- 5,620 real-world multimodal cases
- 64 clinical departments covered
- Evaluation of 19 LLMs
🤖 This article was rewritten by Feed and Figures' editorial AI from a report originally published by arXiv AI. Facts and quotes are preserved from the original; the rewrite focuses on clarity and structure. For the unedited original, see the source link below.
