In the rapidly advancing field of artificial intelligence, large multimodal models (LMMs) have shown great promise in tasks such as visual question answering and complex reasoning. However, these models still struggle with certain visual tasks, such as object localization, counting, and text recognition, limiting their use in specific real-world applications. Now, a research team led by Professor Menai Kan from the Institute of Computing Technology, Chinese Academy of Sciences (ICT, CAS) has proposed a novel method that patches the visual capabilities of LMMs by collaborating with smaller, task-specific models. This approach, published on 15 September 2026, in Frontiers of Computer Science, addresses these shortcomings, allowing LMMs to achieve higher accuracy and efficiency in a wide range of visual tasks.
The research introduces a dynamic collaboration between LMMs and smaller models tailored to specific visual functions. By decomposing complex queries into smaller visual tasks, the LMM uses a model selector to determine whether to rely on its own capabilities or invoke a specialized smaller model. This decision-making process is optimized using a reinforcement learning strategy that ensures the most effective model is chosen for each task. The result is a significant enhancement in the visual performance of LMMs, particularly in areas where they typically underperform, such as in fine-grained visual perception.
The team's method was rigorously tested across several datasets, including LLaVA-Bench, MM-VET, VizWiz, and Blmm, which focus on visual question answering and specialized visual tasks. The results clearly demonstrate the efficacy of the approach, with the model collaboration strategy outperforming existing LMM-based methods by a significant margin. By integrating smaller task-specific models, the approach not only enhances the visual abilities of LMMs but also offers a scalable and flexible framework for addressing visual challenges in real-world applications.
