Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning
(Submitted on 25 Jan 2024 (v1), last revised 8 May 2024 (this version, v3))
Abstract: Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose an evaluation strategy called Positional Error Variance for assessing multiple-choice questions. The strategy aims to perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs. The data and code are available at this https URL
Submission history
From: Xinya Wu [view email][v1] Thu, 25 Jan 2024 08:22:10 GMT (4923kb,D)
[v2] Fri, 26 Jan 2024 09:46:03 GMT (9098kb,D)
[v3] Wed, 8 May 2024 07:34:06 GMT (11455kb,D)
Link back to: arXiv, form interface, contact.