Benchmark release mmpisa-four-models-es-zh-en-kk-ru-2026-09-08
Model accuracy on mmpisa, using multiple-choice-v1. This release records the tested languages, scores, explicit language comparisons and downloadable evidence.
Scores and uncertainty intervals
These scores cover academic multiple-choice questions. They can’t tell you how well a model handles every task in a language. If the gap’s interval includes zero, we can’t call the direction of the difference. The same effort label can also mean different computing budgets across APIs.
How we got these numbers| Model | Reasoning effort | Spanish | Chinese | English | Kazakh | Russian |
|---|---|---|---|---|---|---|
| Mercury 2.5 Preview Inception | High | 96.0% | 88.0% | 96.0% | 92.0% | 88.0% |
| Mercury 2.5 Preview Inception | Low | 80.0% | 76.0% | 80.0% | 64.0% | 72.0% |
| gpt-oss-120b OpenAI | High | 92.0% | 88.0% | 100.0% | 88.0% | 96.0% |
| gpt-oss-120b OpenAI | Low | 92.0% | 88.0% | 92.0% | 80.0% | 88.0% |
| Qwen3.7 Flash Qwen | High | 92.0% | 100.0% | 96.0% | 92.0% | 96.0% |
| Qwen3.7 Flash Qwen | Low | 92.0% | 100.0% | 96.0% | 84.0% | 96.0% |
| Solar Pro 4 Upstage | High | 88.0% | 92.0% | 96.0% | 96.0% | 92.0% |
| Solar Pro 4 Upstage | Low | 80.0% | 100.0% | 96.0% | 88.0% | 92.0% |
Accuracy by repeat
Mercury 2.5 Preview · High
Spanish
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
Chinese
25 unique questions per language · 1 repeat
- Repeat 1: 88.0%
English
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
Kazakh
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
Russian
25 unique questions per language · 1 repeat
- Repeat 1: 88.0%
0 refusals · 0 unparseable answers
Selected completed responses: cost unknown. Total run spending is recorded in execution.json.
Mercury 2.5 Preview · Low
Spanish
25 unique questions per language · 1 repeat
- Repeat 1: 80.0%
Chinese
25 unique questions per language · 1 repeat
- Repeat 1: 76.0%
English
25 unique questions per language · 1 repeat
- Repeat 1: 80.0%
Kazakh
25 unique questions per language · 1 repeat
- Repeat 1: 64.0%
Russian
25 unique questions per language · 1 repeat
- Repeat 1: 72.0%
0 refusals · 5 unparseable answers
Selected completed responses: cost unknown. Total run spending is recorded in execution.json.
gpt-oss-120b · High
Spanish
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
Chinese
25 unique questions per language · 1 repeat
- Repeat 1: 88.0%
English
25 unique questions per language · 1 repeat
- Repeat 1: 100.0%
Kazakh
25 unique questions per language · 1 repeat
- Repeat 1: 88.0%
Russian
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
0 refusals · 2 unparseable answers
Selected completed responses: cost unknown. Total run spending is recorded in execution.json.
gpt-oss-120b · Low
Spanish
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
Chinese
25 unique questions per language · 1 repeat
- Repeat 1: 88.0%
English
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
Kazakh
25 unique questions per language · 1 repeat
- Repeat 1: 80.0%
Russian
25 unique questions per language · 1 repeat
- Repeat 1: 88.0%
0 refusals · 3 unparseable answers
Selected completed responses: cost unknown. Total run spending is recorded in execution.json.
Qwen3.7 Flash · High
Spanish
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
Chinese
25 unique questions per language · 1 repeat
- Repeat 1: 100.0%
English
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
Kazakh
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
Russian
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
0 refusals · 0 unparseable answers
Selected completed responses: cost unknown. Total run spending is recorded in execution.json.
Qwen3.7 Flash · Low
Spanish
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
Chinese
25 unique questions per language · 1 repeat
- Repeat 1: 100.0%
English
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
Kazakh
25 unique questions per language · 1 repeat
- Repeat 1: 84.0%
Russian
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
0 refusals · 3 unparseable answers
Selected completed responses: cost unknown. Total run spending is recorded in execution.json.
Solar Pro 4 · High
Spanish
25 unique questions per language · 1 repeat
- Repeat 1: 88.0%
Chinese
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
English
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
Kazakh
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
Russian
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
0 refusals · 0 unparseable answers
Selected completed responses: cost unknown. Total run spending is recorded in execution.json.
Solar Pro 4 · Low
Spanish
25 unique questions per language · 1 repeat
- Repeat 1: 80.0%
Chinese
25 unique questions per language · 1 repeat
- Repeat 1: 100.0%
English
25 unique questions per language · 1 repeat
- Repeat 1: 96.0%
Kazakh
25 unique questions per language · 1 repeat
- Repeat 1: 88.0%
Russian
25 unique questions per language · 1 repeat
- Repeat 1: 92.0%
0 refusals · 2 unparseable answers
Selected completed responses: cost unknown. Total run spending is recorded in execution.json.
Experiment settings
Protocol: multiple-choice-v1
Dataset version: 22b1caa65980ca1fa295acc31ee5e35200e0c8c0
Download
You can check the math without paying to run the models again. Download all the release files, then use the benchmark CLI to check their checksums and recompute the scores. No model calls or API keys needed.
- manifest.json (External website)
- resolved.json (External website)
- identity.json (External website)
- dataset.jsonl (External website)
- dataset-manifest.json (External website)
- items.jsonl (External website)
- analysis.json (External website)
- aggregate.json (External website)
- aggregate.csv (External website)
- attempts.jsonl (External website)
- execution.json (External website)
- ATTRIBUTION.md (External website)
File checksums · SHA-256
- resolved.json
- dbb43f80e4c0b974bf7b231398b6cc4b785c07f1d4c0ab15d65a8b8cce12baa8
- identity.json
- a66bfe597e17a5dc3dfcee43d6a31a58148180ad12958f91592bfc421726e78e
- dataset.jsonl
- 2a976424699c6a3f9c8f7510d2de600242b77ffe80eac3367412d6d8112d59d9
- dataset-manifest.json
- 81c75ee0967e1ce708e83958d9f53d880de4735d0036144ac3b9463372a5e0ce
- items.jsonl
- 190e9205368c431ba3cd07b3c2e6b4f909ca884962003a629cb664212510ac1c
- analysis.json
- 752f38e6242843537041ad21fcd2f1a96d7fe919112212adae2bb8edd5a10a13
- aggregate.json
- 3a39e023fe11a5f67828374828a845ec63b23386d4215d0cf9104eceabbbb1ad
- aggregate.csv
- 3f8ef040dba4f8bbdfbe38ca66addd92830a179c57562da5fbfd8c874aca5e97
- attempts.jsonl
- e074c4614728f44271070e932e82e1d87243bae62bc0ea743bef7f59a2be28d6
- execution.json
- b15ad6382be0e0eebe611ba14b195f9b464781b2f7474048d0011ec301b80e5d
- ATTRIBUTION.md
- 6993d7ef4b5780ffd7418287354c3796fc5ab02f15fb4a3e0a039fc43f9222a9
Cite this release
Copy the citation for your paper or report, or export it as BibTeX.
BibTeX
@misc{llang-gap-mmpisa-four-models-es-zh-en-kk-ru-2026-09-08,
author = {{Limit 115}},
title = {{Lang Gap: mmpisa}},
year = {2026},
howpublished = {Benchmark release},
note = {Release mmpisa-four-models-es-zh-en-kk-ru-2026-09-08. Created 2026-09-08. Protocol: multiple-choice-v1.},
url = {https://llang-gap-web.vercel.app/en/releases/mmpisa-four-models-es-zh-en-kk-ru-2026-09-08/}
}