As large language models (LLMs) become increasingly embedded in chatbots, virtual assistants, translation services, coding tools, and other AI-powered applications, delivering responses quickly and efficiently has become a growing challenge. Because these models generate text one token at a time, inference can be slow and computationally expensive, particularly for larger models. While speculative decoding has emerged as a promising approach to accelerate inference, many existing methods either require additional model training or struggle to perform consistently across different hardware platforms.
Professor Le-Minh Nguyen, together with doctoral student Dinh-Truong Do and Dr. Nguyen-Khang Le from Japan Advanced Institute of Science and Technology (JAIST), has developed UniSpec, a plug-and-play, training-free speculative decoding framework that speeds up LLM inference without changing model outputs or requiring any additional model training. The framework automatically calibrates the optimal draft size for each hardware platform, estimates confidence scores for retrieved n-grams, and builds a more effective draft tree through confidence-guided expansion. The team also introduced Multi-SpecBench, a multilingual benchmark spanning seven languages and seven generation tasks, providing a broader framework for evaluating speculative decoding beyond English.
Together, these innovations delivered up to 2.6× faster inference than existing training-free speculative decoding methods across multiple LLM architectures, hardware platforms, and languages while producing outputs identical to standard autoregressive decoding. The researchers have also publicly released both the UniSpec implementation and the Multi-SpecBench benchmark to support further research and development.
"UniSpec is a training-free speculative decoding framework that accelerates LLM inference while remaining fully lossless and requiring no additional model training", said Prof. Nguyen.
This paper was made available online on July 01, 2026, and was published in Volume 1: Long Papers, pages 6288–6310 in the Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics . The conference took place between July 2–7, 2026. The paper was selected as an ACL 2026 Best Paper nominee and ultimately received the SAC Highlight Award.
Speculative decoding works by allowing a draft process to predict several candidate tokens before the original language model verifies them simultaneously, reducing the number of decoding steps needed to generate text. However, previous training-free methods typically rely on fixed draft sizes and overlook differences in hardware performance, limiting their efficiency across devices. UniSpec addresses this challenge by automatically adapting to each hardware platform, selecting an optimal draft size based on hardware characteristics, and prioritizing high-confidence token candidates to maximize throughput.
To evaluate the framework, the team tested UniSpec using Llama-3 and Qwen-3 language models on multiple NVIDIA GPU platforms, including the A100, A40, RTX A6000, and RTX 3090. They compared its performance with state-of-the-art training-free speculative decoding methods and found that UniSpec consistently delivered faster inference across different models, hardware platforms, and languages while maintaining identical outputs. The newly developed Multi-SpecBench benchmark further demonstrated the framework's effectiveness on multilingual tasks, extending evaluation beyond the English-centric benchmarks commonly used in previous studies.
"Because UniSpec is training-free and automatically adapts to different hardware, it can be integrated into existing LLM systems without additional model retraining, reducing deployment cost while improving inference efficiency," said Prof. Nguyen.
According to the researchers, the framework could improve a wide range of real-world AI applications, including virtual assistants, customer support systems, multilingual translation, retrieval-augmented generation (RAG), code generation, educational AI tutors, mathematical reasoning assistants, and large-scale cloud AI services. Because the framework does not require retraining or changes to the underlying model, it can be integrated into existing LLM systems as a plug-and-play solution, potentially reducing deployment costs while improving inference efficiency. The study does have limitations. The current evaluation focuses on seven languages and has not yet been extended to morphologically rich languages such as Arabic. In addition, the framework assumes access to model logits during inference, which may limit its applicability in some closed-source or black-box AI systems. Future research will explore broader language coverage, dynamic hardware environments, and deployment in additional real-world settings.
"Over the next 5–10 years, hardware-aware and training-free inference optimization techniques like UniSpec could become an essential component of practical AI infrastructure, helping make powerful language models more accessible, scalable, and environmentally sustainable," concluded Prof. Nguyen.
