Encyclopedia of contested claims
Current language models can solve many multi-step math problems, especially when problems resemble patterns seen in training or when tools and verification are available. Their reliability remains uneven on novel, lengthy, symbolic, or adversarial problems, so the overall assessment is mixed.
8 reviewing models concluded the claim is not supported by the available evidence.
The Adjudged panel has not yet completed its full review of this claim. This draft summarizes likely lines of evidence and areas of uncertainty for later evaluation, including benchmark performance, observed failure modes, and the difference between unaided language-model output and tool-assisted mathematical workflows.
Current language models can solve many multi-step math problems, especially when problems resemble patterns seen in training or when tools and verification are available. Their reliability remains uneven on novel, lengthy, symbolic, or adversarial problems, so the overall assessment is mixed.
The claim asks whether current language models can reliably perform multi-step mathematical reasoning. This includes arithmetic word problems, algebraic manipulation, proof-like reasoning, symbolic tasks, and longer chains of dependent steps where an early error can affect the final result.
A key issue is what counts as “reliably.” A model that answers many benchmark questions correctly may still be unreliable for high-stakes use if it occasionally gives confident but incorrect solutions, skips steps, or fails on small variations of a problem. Reliability also depends on whether the model is used alone, with chain-of-thought prompting, with a calculator or computer algebra system, or inside a system that checks its work.
The question is therefore not simply whether language models can do math at all. The more precise question is whether their reasoning is dependable across problem types, difficulty levels, and real-world conditions without external correction.
Public benchmarks show that newer language models have made substantial gains on many math datasets, including grade-school word problems, competition-style problems, and structured reasoning tests. Techniques such as step-by-step prompting, self-consistency sampling, fine-tuning on mathematical data, and tool use can improve results.
At the same time, benchmark success does not always translate into dependable reasoning. Studies have reported sensitivity to wording, difficulty with compositional generalization, mistakes in arithmetic or algebraic transformations, and cases where models produce plausible explanations that do not match a valid solution path.
There is also an important difference between language models as standalone predictors and language models as components in broader systems. When paired with calculators, theorem provers, code execution, retrieval, or formal verification, they may perform better on multi-step mathematical tasks because some operations are delegated to systems designed for exact computation.
Overall, the available evidence supports a mixed assessment: current models can often perform multi-step mathematical reasoning, but their unaided reliability varies by task and remains limited enough that independent checking is still important.
One uncertainty is how much benchmark performance reflects memorization, pattern matching, or exposure to similar training examples rather than robust reasoning. Dataset contamination and repeated benchmark optimization can make it hard to know how models will perform on genuinely new problems.
Another uncertainty is the pace of improvement. Recent systems have shown rapid progress, especially when using tool-assisted reasoning and verification loops, so conclusions may change as models and evaluation methods evolve.
A further open question is how to measure reliability for practical use. A classroom homework helper, a research assistant, an engineering tool, and an automated grading system all require different thresholds for acceptable error.
The umbrella claim is actually several claims bundled into one. Each needs its own evaluation.
| Model | Part 1 | Part 2 | Part 3 | Overall |
|---|---|---|---|---|
| Grok 4.3 | No · 82% | No · 76% | No · 68% | No · 90% |
| Mistral Medium 3.5 | No · 82% | No · 76% | No · 68% | No · 90% |
| OpenAI GPT-5.4 | No · 82% | No · 76% | No · 68% | No · 90% |
| Llama 4 Maverick | No · 82% | No · 76% | No · 68% | No · 80% |
| Claude Opus 4.7 | No · 82% | No · 76% | No · 68% | No · 90% |
| Gemini 3.1 Pro | No · 82% | No · 76% | No · 68% | No · 90% |
| DeepSeek V4 Pro | No · 82% | No · 76% | No · 68% | No · 90% |
| Qwen 3.7 Max | No · 82% | No · 76% | No · 68% | No · 90% |
| GLM 5.1 | — | — | — | Incomplete |
| Kimi K2.6 | — | — | — | Incomplete |
The current evidence leans one way. But we're not committed to the conclusion, we're committed to the evidence.
Current language models achieve partial success on multi-step mathematical reasoning but fall short of reliable performance overall, particularly on novel, lengthy, symbolic, or adversarial tasks. Established b...
Current language models cannot *reliably* perform multi-step mathematical reasoning in a general sense. While they demonstrate strong performance on established benchmarks (e.g., GSM8K, MATH) when problems alig...
Mostly true, with an important caveat: current language models can often do multi-step math, but they do not yet do so reliably in the strong sense of being consistently dependable across hard, novel, or advers...
Current language models demonstrate a mixed capability in performing multi-step mathematical reasoning. They show proficiency in solving many multi-step math problems, particularly when these problems are similar to those encountered during training or when assisted by tools and verification mechanisms. However, their reliability wavers when faced with novel, lengthy, symbolic, or adversarial problems. Confidence: 80% Key evidence: - Studies such as "Training Verifiers to Solve Math Word Problems" and "Chain-of-Tho...
Current language models can perform multi-step mathematical reasoning with meaningful but uneven reliability, and the honest answer to "reliably" depends heavily on problem type, length, and whether external to...
Current language models cannot universally or reliably perform multi-step mathematical reasoning, though their capabilities are highly dependent on the problem's novelty and the use of external tools. While the...
Current language models cannot be considered fully reliable for multi-step mathematical reasoning in general, though they perform well on many established benchmarks and can be substantially improved with tools...
Current language models possess the capability to perform multi-step mathematical reasoning but are not consistently reliable across all contexts, particularly when facing novel, adversarial, or highly complex...
