Published as a conference paper at ICLR 2026

CE ChemEval

A Multi-level and Fine-grained Chemical Capability Evaluation for Large Language Models

Yuqing Huang*, Rongyang Zhang*, Xuesong He, Xuyang Zhi, Hao Wang, Nuo Chen, Zongbo Liu, Xin Li, Feiyang Xu, Deguang Liu, Huadong Liang, Yi Li, Jian Cui, Yin Xu, Shijin Wang, Qi Liu, Defu Lian, Guiquan Liu, Enhong Chen
University of Science and Technology of China; iFLYTEK Co., Ltd.
*Equal Contributors. Corresponding Authors: Hao Wang, Xin Li, Guiquan Liu, Enhong Chen. Contact: wanghao3@ustc.edu.cn
Paper PDF Dataset Code Results BibTeX
ChemEval benchmark generation pipeline

Benchmark generation pipeline. ChemEval combines academic literature, official datasets, expert screening, filtering, and Q&A construction into four progressive chemical evaluation levels.

News

[2026-06-09] ChemEval project page, README, GitHub Pages, and repository metadata are updated for the ICLR 2026 release.

Introduction

ChemEval is a fine-grained benchmark for evaluating large language models and multimodal large language models in chemistry. The benchmark targets the gap between general language ability and the chemical expertise required for literature understanding, molecular reasoning, synthesis planning, and multimodal evidence interpretation.

The ICLR 2026 version organizes the evaluation into 4 progressive levels, 13 capability dimensions, and 62 text and multimodal tasks. It reports 1,960 text examples and 1,200 multimodal examples, combining curated open-source datasets with expert-authored materials.

4Progressive levels
13Capability dimensions
62Text and multimodal tasks
3,160Evaluation examples

ChemEval Benchmark

Overview

Advanced Knowledge Question Answering

Objective and subjective chemistry QA covering terminology, concepts, quantitative analysis, and cross-modal reasoning.

Literature Understanding

Information extraction, inductive generation, and molecular name recognition from chemical literature and image data.

Molecular Understanding

SMILES/IUPAC conversion, molecular property prediction, and molecular description from structures, spectra, and text.

Scientific Knowledge Deduction

Retrosynthesis, reaction condition recommendation, reaction outcome prediction, and mechanism analysis for chemical synthesis.

ChemEval task and reaction-type distribution
Distribution of ChemEval task modalities and major chemical reaction types.

Experimental Results

General LLMs

General models are strong at literature understanding and instruction following, but struggle with molecular understanding and scientific inference.

Chemical LLMs

Chemistry-specialized models show advantages on terminology and molecular property tasks, but can lose broad instruction-following ability.

Multimodal Reasoning

Multimodal chemistry tasks expose gaps in visual structure recognition and chemical knowledge application.

Representative multi-level zero-shot text task results on ChemEval
Representative multi-level 0-shot text task results on ChemEval.
Three-shot performance changes relative to zero-shot on ChemEval text tasks
3-shot performance changes relative to 0-shot text-task performance.
Impact of model scaling on ChemEval task performance
Impact of model scaling on selected ChemEval tasks.
Performance overview of multimodal ChemEval tasks
Performance overview of multimodal ChemEval tasks.

Code And Data

The repository provides answer extraction, LLM-based grading, and code-based metric scripts for textual and multimodal ChemEval tasks. The dataset is released on Hugging Face and the evaluation code is hosted on GitHub. 

python "Textual/code evaluate/1_Split_filename.py"
python "Textual/code evaluate/2_Extract.py"
python "Textual/code evaluate/3_Evaluate.py"

python "Multimodel/3_code_evaluate/1Extract.py"
python "Multimodel/3_code_evaluate/2Evaluate.py"

BibTeX

@inproceedings{
huang2026chemeval,
title={ChemEval: A Multi-level and Fine-grained Chemical Capability Evaluation for Large Language Models},
author={Yuqing Huang and Rongyang Zhang and Xuesong He and Xuyang Zhi and Hao Wang and Nuo Chen and Zongbo Liu and Xin Li and Feiyang Xu and Deguang Liu and Huadong Liang and Yi Li and Jian Cui and Yin Xu and Shijin Wang and Qi Liu and Defu Lian and Guiquan Liu and Enhong Chen},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=JrqjSkEPrX}
}