DeepSeek V4 Benchmarks: MMLU, HumanEval & SWE-bench
DeepSeek V4 Benchmarks: MMLU, HumanEval & SWE-bench
Fellow benchmark hunters, this page has changed shape since the first version.
DeepSeek V4 is no longer just a rumor or an internal benchmark leak. DeepSeek released the V4 preview on April 24, 2026, followed by an official technical report on April 26 and public model cards for DeepSeek-V4-Pro and DeepSeek-V4-Flash.
So the useful question is different now:
Which DeepSeek V4 benchmark numbers are official, which ones are independently reproduced, and which ones still need a cautious label before you use them in infrastructure planning?
Here is the clean version.
Benchmark Summary Table
Bottom line after the July update: V4 is real, the weights and official benchmark tables are public, and the original “unverified leak” framing should be retired.
The caution label still matters. But it now means “official self-reported, awaiting broad third-party reproduction,” not “rumor.”
Coding: HumanEval and SWE-bench
HumanEval changed the most in this update.
The early 90% V4 HumanEval claim no longer matches the official base-model table. DeepSeek’s V4 technical report lists:
That is still an improvement over V3.2, but it is not the headline.
I stopped here because HumanEval is too clean. One function. One prompt. Hidden tests. Useful, yes. Enough to judge a coding model in 2026? Not really.
The real coding signal is SWE-bench.
DeepSeek reports:
That is the story.
If independent leaderboards reproduce the 80.6% SWE-bench Verified result, V4-Pro moves into the same practical band as the strongest closed-source coding agents. V4-Flash is the one I would watch even more closely, because 79.0% at Flash pricing changes the cost side of the equation.
HumanEval tells you V4 can write functions.
SWE-bench tells you whether it can survive real repositories.
Those are not the same test.
Reasoning: MMLU, MMLU-Pro, and GPQA
V4 MMLU numbers are now traceable.
DeepSeek reports:
For instruction and reasoning modes, the stronger comparison is MMLU-Pro:
This changes the old conclusion.
V4 is not just a coding-scale release. The Pro-Max mode is competitive on reasoning and knowledge benchmarks too. R1 still matters historically, but V4 is now the live DeepSeek family member to evaluate for mixed coding, long-context, and reasoning workflows.
The caveat is simple: these are official DeepSeek numbers. Treat them as real reported results, not independent proof.
Long Context: The 1M-Token Claim
Both V4-Pro and V4-Flash support a 1M-token context window.
That is official in the model cards and in DeepSeek’s API pricing page, which also lists a maximum output length of 384K.
The more interesting technical claim is efficiency. DeepSeek says V4-Pro uses only 27% of the single-token inference FLOPs and 10% of the KV cache required by V3.2 in the 1M-token setting.
That matters. A million-token context is not useful if serving cost or latency makes it mostly decorative.
But here is the part I would not skip: a 1M-token window is not the same thing as perfect 1M-token task reliability.
DeepSeek’s own long-context table reports:
That is useful. It is not magic.
For real work, I would test V4 on the exact shape of your context: codebase navigation, contract review, research synthesis, logs, or multi-document QA. “1M context” is the starting condition, not the final answer.
Pricing: Why V4-Flash Matters
DeepSeek’s API docs list two V4 API models:
That pricing is the reason V4-Flash deserves attention.
V4-Pro is stronger. No mystery there.
But V4-Flash-Max landing close to V4-Pro-Max on SWE-bench Verified, LiveCodeBench, and MMLU-Pro makes it the practical model to test first for high-volume workflows.
One more date to note: DeepSeek says deepseek-chat and deepseek-reasoner are scheduled to be deprecated on July 24, 2026, with compatibility mapping to V4-Flash modes.
If you have production code using those older names, this is the part to check.
Independent vs Official Results
This section needed the biggest correction.
What is official now:
- DeepSeek V4 preview release: April 24, 2026.
- Technical report: arXiv:2606.19348, submitted April 26, 2026.
- Model cards and weights: V4-Pro and V4-Flash on Hugging Face.
- Model shape: Pro = 1.6T total / 49B activated; Flash = 284B total / 13B activated.
- Context length: 1M tokens.
- API models:
deepseek-v4-proanddeepseek-v4-flash.
What is still not the same as independent proof:
- V4-Pro-Max 80.6% on SWE-bench Verified.
- V4-Pro-Max 93.5 on LiveCodeBench.
- V4-Pro-Max 87.5 on MMLU-Pro.
- V4-Pro-Max 90.1 on GPQA Diamond.
Those numbers are official DeepSeek numbers, not rumors. But benchmark articles should still distinguish official self-reporting from broad third-party reproduction.
That distinction sounds boring until you are choosing a model for production. Then it becomes the whole game.
How to Read These Benchmarks
HumanEval measures function-level code generation. It is clean and reproducible, which is why people keep using it. Its weakness is just as obvious: it does not test multi-file reasoning, dependency tracing, or real repository context.
SWE-bench measures real software engineering. Real GitHub issues. Real repositories. Patches that must actually work. High SWE-bench scores are much harder to hand-wave.
MMLU measures broad knowledge. It is useful for comparing general capability floors, but frontier models are crowding near the top.
MMLU-Pro is more discriminating. Harder questions. Less room for lucky guessing.
GPQA Diamond measures expert-level science reasoning. It is a better stress test than plain general knowledge.
LiveCodeBench is useful because it tracks coding performance in a more current setting than older static coding benchmarks.
Long-context benchmarks tell you whether a model can actually retrieve and use information deep inside a large input. This matters more now because every serious model claims a huge context window.
The mistake is treating one benchmark as the whole model.
I have made that mistake. It feels efficient. It is not.
The Number That Actually Earns Your Attention
Here is where I land after the July update: the V4 benchmark story is not HumanEval.
The official HumanEval base score is lower than the early rumor, and the benchmark itself is too narrow to decide production coding workflows.
The number that matters is SWE-bench Verified: 80.6% for V4-Pro-Max, with V4-Flash-Max close behind at 79.0%.
If independent leaderboards reproduce that band, V4 becomes one of the first open-weight model families that can sit near the closed-source frontier for real software engineering tasks while keeping DeepSeek-style pricing.
For planning today, I would treat V4 as the live evaluation target and V3.2 as the fallback baseline.
Do not make the old mistake of treating V4 as unreleased.
At Macaron, we turn model evaluation into structured, executable workflows, so you can test AI against your actual tasks instead of only reading synthetic benchmark tables. Try Macaron with one real workflow and judge the results yourself.
FAQ
Q: What are the confirmed DeepSeek V4 benchmark scores? DeepSeek’s official V4 report lists V4-Pro-Max at 80.6% on SWE-bench Verified, 93.5 on LiveCodeBench, 87.5 on MMLU-Pro, and 90.1 on GPQA Diamond. These are official DeepSeek numbers, not broad independent reproduction yet.
Q: When was DeepSeek V4 released? DeepSeek released the V4 preview on April 24, 2026. The technical report was submitted to arXiv on April 26, 2026.
Q: What are the DeepSeek V4 model sizes? V4-Pro has 1.6T total parameters with 49B activated per token. V4-Flash has 284B total parameters with 13B activated per token. Both support 1M-token context.
Q: Is DeepSeek V4 better than V3.2? On DeepSeek’s official tables, yes across the core headline areas: long-context efficiency, MMLU, MMLU-Pro, SWE-bench Verified, and LiveCodeBench. The open question is how closely independent testing reproduces the official table.
Q: Is V4-Pro or V4-Flash the better default? V4-Pro is the stronger model. V4-Flash is the value model. The interesting benchmark fact is that V4-Flash-Max reaches 79.0% on SWE-bench Verified while being much cheaper through the API.
Q: What changed from the first version of this article? The article originally treated V4 as unreleased and benchmark numbers as leaks. That is no longer accurate after the April 24 preview release, April 26 technical report, and public Hugging Face model cards.
From next article:
Apply to become Macaron's first friends
Read & discover
Free calculators
Company
