HKU Benchmarks AI Reasoning on Chinese Tasks

HKU Business School today released its Large Language Model (LLM) reasoning capability assessment report, benchmarking the reasoning capabilities of 36 leading LLMs using Chinese language and characters. The study was a comprehensive testing of differences in reasoning performance across 36 different models.

The report reveals that GPT-o3 topped the basic logic ability assessment, while Gemini 2.5 Flash took the lead in the contextual reasoning ability assessment. For overall reasoning capability, Doubao 1.5 Pro (Thinking) ranked first, followed closely by Chat GPT-5. Several Chinese LLMs, including Doubao 1.5 Pro, Qwen 3 (Thinking), and DeepSeek-R1, also attained a high ranking on the list, demonstrating the superior reasoning capabilities of Chinese LLMs when it comes to Chinese inputs.

From OpenAI o1's pioneering introduction of its reasoning model to DeepSeek-R1's focus on problem-solving abilities, the LLM market continues to evolve, with reasoning capabilities increasingly being evaluated for reasoning power and accuracy. In light of this, the Artificial Intelligence Evaluation Lab (AIEL) (https://www.hkubs.hku.hk/aimodelrankings_en) at HKU Business School, led by Professor Jack Jiang, developed a comprehensive evaluation system covering both basic logic and contextual reasoning capabilities. Using test sets of varying difficulty, they benchmarked LLMs using Chinese inputs.

Test subjects included 36 mainstream LLMs from China and the United States, including 14 reasoning models, 20 general-purpose models, and two unified systems. The results showed that for basic logical inference, the gap between reasoning models and general-purpose models was relatively small. However, for contextual reasoning, the advantages of reasoning models also became more visible. Moreover, comparisons of models from the same company revealed that reasoning models generally perform better in contextual reasoning, confirming that the overall competitiveness of a model's architectures is best revealed across complex tasks.

Professor Jiang said, "The reasoning capabilities of LLMs are inextricably linked to their cultural and linguistic environments. As the reasoning capabilities of large models gain increasing attention, we hope to use this evaluation system to identify the 'strongest brains' when it comes to the Chinese context. This will then drive the continuous improvement of reasoning capabilities across various models, further optimising efficiency and costs, and enabling them to realise their value in a wider range of application scenarios."

Evaluation Scope and Methodology

In the study, 90% of the questions were original or meticulously adapted, while 10% were selected from Mainland China's high school entrance exams, college entrance exams, and well-known datasets. This approach aimed to authentically test the models' independent reasoning capabilities.

As for question complexity, 60% were simple questions and 40% were complex. A progressively more complex assessment process was employed to accurately characterise the model's reasoning capabilities.

The model's reasoning capabilities were scored based on accuracy (correctness or reasonableness), logical coherence and conciseness.

Basic Logical Inference Capability

In the Basic Logical Inference capability assessment, GPT-o3 took first place, followed closely by Doubao 1.5 Pro (Thinking). Some models, such as Llama 3.3 70B and 360 Zhinao 2-o1, exhibited significant weaknesses in basic logic.

Table 1: Ranking for Basic Logical Inference Capability

Contextual Reasoning Capability

In the Contextual Reasoning Capability ranking, Gemini 2.5 Flash took first place, excelling in common sense reasoning and subject reasoning. Doubao 1.5 Pro (Thinking) excelled in common sense reasoning, while Gemini 2.5 Pro demonstrated strengths in discipline-based reasoning and decision-making under uncertainty, with both tied for second place. Grok 3 (Thinking), as well as GPT, Ernie, DeepSeek, Hunyuan, and Qwen also performed well.

Table 2: Ranking for Contextual Reasoning Capability

Composite Ranking Results

In terms of composite capabilities, the 36 models showed significant differences. Doubao 1.5 Pro (Thinking) took the top spot, demonstrating its superior performance in both basic logic inference and contextual reasoning. GPT-5 was the close second, with GPT-o3 and Doubao 1.5 Pro placing third and fourth, respectively.

Table 3. Composite Ranking

Click here to view the complete "Large Language Model Reasoning Capability Evaluation Report."

Reviewing the above rankings, we can see that many LLMs from China performed exceptionally well and have made rapid progress, demonstrating the unique advantages and strong potential of China's LLM industry in the Chinese language.