MBPP Benchmark 评估报告

Mostly Basic Python Programming — EvalPlus 增强测试

DeepSeek V4 Pro

📊 执行摘要

MBPP Base pass@1
78.8%
298 / 378 通过
MBPP Plus pass@1
68.8%
260 / 378 通过
空白答案
13.2%
50 / 378 未生成代码
Base→Plus 衰减
10.1%
40 个任务 Base 通过但 Plus 失败
MBPP Base 通过率78.8%
78.8%
MBPP Plus 通过率 (EvalPlus 增强)68.8%
68.8%

本次测试对 DeepSeek V4 Pro 模型在 MBPP（Mostly Basic Python Programming）基准测试上的代码生成能力进行了系统评估。 测试使用 EvalPlus 框架，包含 378 个 Python 编程任务，分别评估基础测试（Base）和增强测试（Plus）两个层面的通过率。 DeepSeek V4 Pro 在 Base 测试中取得了 78.8% 的通过率，在更严格的 Plus 测试中下降了 10.1% 个百分点至 68.8%。 值得注意的是，模型在 50 个任务（13.2%）中未能生成有效代码。

🔬 测试方法论

MBPP 基准简介

MBPP (Mostly Basic Python Programming) 是由 Google Research 在 2021 年提出的代码生成基准测试， 包含 974 个由众包工作者编写的入门级 Python 编程问题。每个问题包含自然语言描述、一个解决方案和三个测试用例。

本次测试使用经过清理的 MBPP-sanitized 版本（378 个任务），每个任务包含：

EvalPlus 框架

EvalPlus 是由伊利诺伊大学厄巴纳-香槟分校（UIUC）研究团队开发的代码评估增强框架。 它通过自动生成大量额外测试用例来暴露模型生成代码中的边界条件和隐藏缺陷。 EvalPlus 的增强测试集（MBPP+）平均每个任务包含约 35 倍于原始 MBPP 的测试数量， 能够更精确地评估代码质量。

测试配置

模型DeepSeek V4 Pro
API 端点yhapiplatform.com (OpenAI 兼容代理)
温度参数0.1 (接近确定性解码)
评估模式pass@1 (单次生成)
测试框架EvalPlus (evalplus v0.2.0)
任务数量378 个 MBPP-sanitized 任务
测试时间2026-06-07 07:52
结果哈希ee43ecabebf20deef4bb776a405ac5b1

📈 测试结果

Base 测试结果分布

Plus 测试结果分布

结果细分

258
✅ Base+Plus 均通过
68.3%
40
⚠️ 仅 Base 通过
10.6%
78
❌ 均未通过
20.6%
50
🈳 空白答案
13.2%

从 Base 到 Plus 的通过率下降 10.1% 个百分点， 表明在 40 个任务中，模型生成的代码虽通过了基础测试，但在更全面的增强测试中暴露了缺陷。 这凸显了 EvalPlus 增强评估对于发现隐蔽代码问题的价值。

🏆 模型对比

以下为 DeepSeek V4 Pro 与其他主流模型在 同一 EvalPlus MBPP v0.2.0 基准上的对比。 数据来源：EvalPlus Leaderboard (2026-06-07 获取)。

MBPP Base pass@1 对比

O1 Preview (Sept 2024)
95.5%
O1 Mini (Sept 2024)
93.1%
Qwen2.5-Coder-32B-Instruct
90.5%
Gemini 1.5 Pro 002
89.7%
Claude 3.5 Sonnet (June 2024)
89.4%
GPT-4o (2024-08-06)
87.0%
GPT-4o Mini (2024-07-18)
85.0%
DeepSeek-Coder-V2-Instruct
83.5%
DeepSeek-Coder-33B-Instruct
81.7%
CodeLlama-34B-Instruct
72.8%
CodeLlama-13B-Instruct
63.4%
CodeLlama-7B-Instruct
54.6%
Llama-3.1-8B-Instruct
70.3%
Phi-3.5-Mini-Instruct
69.2%
StarCoder2-15B-Instruct
62.6%
Mistral-7B-Instruct-v0.3
57.9%
→ DeepSeek V4 Pro
78.8%

MBPP Plus pass@1 对比

O1 Preview (Sept 2024)
92.9%
O1 Mini (Sept 2024)
89.1%
Qwen2.5-Coder-32B-Instruct
83.3%
Gemini 1.5 Pro 002
82.7%
Claude 3.5 Sonnet (June 2024)
82.6%
GPT-4o (2024-08-06)
79.9%
GPT-4o Mini (2024-07-18)
75.7%
DeepSeek-Coder-V2-Instruct
76.8%
DeepSeek-Coder-33B-Instruct
74.5%
CodeLlama-34B-Instruct
65.2%
CodeLlama-13B-Instruct
55.6%
CodeLlama-7B-Instruct
47.5%
Llama-3.1-8B-Instruct
60.5%
Phi-3.5-Mini-Instruct
57.8%
StarCoder2-15B-Instruct
54.4%
Mistral-7B-Instruct-v0.3
46.3%
→ DeepSeek V4 Pro
68.8%

完整对比表

# 模型 厂商 MBPP Base MBPP Plus 衰减
1 O1 Preview (Sept 2024) OpenAI 95.5% 92.9% 2.6%
2 O1 Mini (Sept 2024) OpenAI 93.1% 89.1% 4.0%
3 Qwen2.5-Coder-32B-Instruct Alibaba (Qwen) 90.5% 83.3% 7.2%
4 Gemini 1.5 Pro 002 Google 89.7% 82.7% 7.0%
5 Claude 3.5 Sonnet (June 2024) Anthropic 89.4% 82.6% 6.8%
6 GPT-4o (2024-08-06) OpenAI 87.0% 79.9% 7.1%
7 GPT-4o Mini (2024-07-18) OpenAI 85.0% 75.7% 9.3%
8 DeepSeek-Coder-V2-Instruct DeepSeek 83.5% 76.8% 6.7%
9 DeepSeek-Coder-33B-Instruct DeepSeek 81.7% 74.5% 7.2%
10 CodeLlama-34B-Instruct Meta 72.8% 65.2% 7.6%
11 CodeLlama-13B-Instruct Meta 63.4% 55.6% 7.8%
12 CodeLlama-7B-Instruct Meta 54.6% 47.5% 7.1%
13 Llama-3.1-8B-Instruct Meta 70.3% 60.5% 9.8%
14 Phi-3.5-Mini-Instruct Microsoft 69.2% 57.8% 11.4%
15 StarCoder2-15B-Instruct BigCode 62.6% 54.4% 8.2%
16 Mistral-7B-Instruct-v0.3 Mistral AI 57.9% 46.3% 11.6%
- ◆ DeepSeek V4 Pro DeepSeek 78.8% 68.8% 10.1%

🔍 错误分析

严重问题：50 个空白答案 (13.2%)
模型在 50 个任务中完全未能生成代码，这是影响通过率的最大因素。 若排除空白答案，有效答案的 Base 通过率可达 90.9%， Plus 通过率可达 79.3%。
Base→Plus 衰减分析
40 个任务通过了 Base 测试但未能通过 Plus 测试（占全部任务的 10.6%）。 这些任务中，模型代码在基础测试中表现正确，但在边界条件、异常输入等方面存在缺陷。 常见失败模式包括：空输入处理不当、边界值错误、类型转换问题。
代码质量特征
模型倾向于生成简洁、符合 Python 风格的代码。通过的任务中代码平均长度适中， 函数包含清晰的文档字符串（Docstring）和类型注解。 失败案例主要集中在：数学/算法密集任务、字符串边界处理、复杂数据结构操作。

Plus 失败典型模式

类别典型任务说明
空输入/零值Mbpp/7, Mbpp/119, Mbpp/559未正确处理空列表、空字符串等边界输入
负数处理Mbpp/99, Mbpp/244, Mbpp/589函数在负数输入时行为异常
大数值溢出Mbpp/267, Mbpp/287在 1,000,000 级别输入时性能或正确性问题
字符串边界Mbpp/16, Mbpp/748, Mbpp/806单字符、空字符串等极端字符串输入
嵌套结构Mbpp/161, Mbpp/792深层嵌套数据结构的遍历/处理
格式解析Mbpp/427, Mbpp/593非标准格式或边界格式的输入解析

📋 完整任务列表

共 378 个任务。点击任务 ID 查看详细代码和测试结果。 ✅ Perfect: 258 ⚠️ Base Only: 40 ❌ Failed: 28 🈳 Empty: 50

任务 ID Base Plus 状态 代码长度 详情
Mbpp/2 ✓ ✓ ✅ Perfect 142 View →
Mbpp/3 ✓ ✓ ✅ Perfect 402 View →
Mbpp/4 ✓ ✓ ✅ Perfect 275 View →
Mbpp/6 ✓ ✓ ✅ Perfect 322 View →
Mbpp/7 ✓ ✗ ⚠️ Base Only 210 View →
Mbpp/8 ✓ ✓ ✅ Perfect 310 View →
Mbpp/9 ✓ ✓ ✅ Perfect 408 View →
Mbpp/11 ✗ ✗ ❌ Failed 675 View →
Mbpp/12 ✓ ✓ ✅ Perfect 511 View →
Mbpp/14 ✓ ✓ ✅ Perfect 285 View →
Mbpp/16 ✗ ✗ ❌ Empty 0 View →
Mbpp/17 ✓ ✓ ✅ Perfect 304 View →
Mbpp/18 ✓ ✓ ✅ Perfect 689 View →
Mbpp/19 ✓ ✓ ✅ Perfect 157 View →
Mbpp/20 ✓ ✓ ✅ Perfect 418 View →
Mbpp/56 ✓ ✓ ✅ Perfect 277 View →
Mbpp/57 ✓ ✓ ✅ Perfect 669 View →
Mbpp/58 ✓ ✓ ✅ Perfect 242 View →
Mbpp/59 ✓ ✓ ✅ Perfect 186 View →
Mbpp/61 ✓ ✓ ✅ Perfect 633 View →
Mbpp/62 ✓ ✓ ✅ Perfect 108 View →
Mbpp/63 ✓ ✓ ✅ Perfect 460 View →
Mbpp/64 ✓ ✓ ✅ Perfect 328 View →
Mbpp/65 ✓ ✓ ✅ Perfect 595 View →
Mbpp/66 ✓ ✓ ✅ Perfect 274 View →
Mbpp/67 ✓ ✓ ✅ Perfect 1328 View →
Mbpp/68 ✓ ✓ ✅ Perfect 369 View →
Mbpp/69 ✓ ✓ ✅ Perfect 630 View →
Mbpp/70 ✓ ✓ ✅ Perfect 335 View →
Mbpp/71 ✓ ✓ ✅ Perfect 830 View →
Mbpp/72 ✗ ✗ ❌ Empty 0 View →
Mbpp/74 ✓ ✗ ⚠️ Base Only 719 View →
Mbpp/75 ✓ ✓ ✅ Perfect 399 View →
Mbpp/77 ✓ ✓ ✅ Perfect 139 View →
Mbpp/79 ✓ ✓ ✅ Perfect 259 View →
Mbpp/80 ✓ ✓ ✅ Perfect 252 View →
Mbpp/82 ✓ ✓ ✅ Perfect 192 View →
Mbpp/84 ✗ ✗ ❌ Failed 9 View →
Mbpp/85 ✓ ✓ ✅ Perfect 291 View →
Mbpp/86 ✓ ✓ ✅ Perfect 291 View →
Mbpp/87 ✓ ✓ ✅ Perfect 265 View →
Mbpp/88 ✓ ✓ ✅ Perfect 158 View →
Mbpp/89 ✓ ✓ ✅ Perfect 166 View →
Mbpp/90 ✓ ✓ ✅ Perfect 278 View →
Mbpp/91 ✓ ✓ ✅ Perfect 414 View →
Mbpp/92 ✗ ✗ ❌ Failed 9 View →
Mbpp/93 ✓ ✓ ✅ Perfect 238 View →
Mbpp/94 ✓ ✓ ✅ Perfect 178 View →
Mbpp/95 ✓ ✓ ✅ Perfect 439 View →
Mbpp/96 ✓ ✓ ✅ Perfect 399 View →
Mbpp/97 ✗ ✗ ❌ Failed 31 View →
Mbpp/98 ✓ ✓ ✅ Perfect 391 View →
Mbpp/99 ✓ ✗ ⚠️ Base Only 392 View →
Mbpp/100 ✓ ✓ ✅ Perfect 1467 View →
Mbpp/101 ✓ ✓ ✅ Perfect 436 View →
Mbpp/102 ✓ ✗ ⚠️ Base Only 347 View →
Mbpp/103 ✗ ✗ ❌ Failed 9 View →
Mbpp/104 ✓ ✓ ✅ Perfect 339 View →
Mbpp/105 ✓ ✓ ✅ Perfect 274 View →
Mbpp/106 ✓ ✓ ✅ Perfect 328 View →
Mbpp/108 ✓ ✓ ✅ Perfect 792 View →
Mbpp/109 ✗ ✗ ❌ Empty 0 View →
Mbpp/111 ✓ ✓ ✅ Perfect 455 View →
Mbpp/113 ✓ ✗ ⚠️ Base Only 305 View →
Mbpp/116 ✓ ✓ ✅ Perfect 171 View →
Mbpp/118 ✓ ✓ ✅ Perfect 322 View →
Mbpp/119 ✓ ✗ ⚠️ Base Only 879 View →
Mbpp/120 ✗ ✗ ❌ Failed 9 View →
Mbpp/123 ✓ ✓ ✅ Perfect 782 View →
Mbpp/124 ✗ ✗ ❌ Empty 0 View →
Mbpp/125 ✗ ✗ ❌ Empty 0 View →
Mbpp/126 ✓ ✗ ⚠️ Base Only 298 View →
Mbpp/127 ✓ ✓ ✅ Perfect 284 View →
Mbpp/128 ✓ ✓ ✅ Perfect 442 View →
Mbpp/129 ✗ ✗ ❌ Failed 443 View →
Mbpp/130 ✓ ✓ ✅ Perfect 556 View →
Mbpp/131 ✓ ✓ ✅ Perfect 533 View →
Mbpp/132 ✓ ✓ ✅ Perfect 309 View →
Mbpp/133 ✓ ✓ ✅ Perfect 306 View →
Mbpp/135 ✓ ✓ ✅ Perfect 182 View →
Mbpp/137 ✓ ✗ ⚠️ Base Only 497 View →
Mbpp/138 ✗ ✗ ❌ Empty 0 View →
Mbpp/139 ✓ ✓ ✅ Perfect 284 View →
Mbpp/140 ✓ ✓ ✅ Perfect 193 View →
Mbpp/141 ✗ ✗ ❌ Empty 0 View →
Mbpp/142 ✓ ✓ ✅ Perfect 571 View →
Mbpp/145 ✓ ✓ ✅ Perfect 270 View →
Mbpp/160 ✗ ✗ ❌ Empty 0 View →
Mbpp/161 ✓ ✗ ⚠️ Base Only 691 View →
Mbpp/162 ✓ ✓ ✅ Perfect 233 View →
Mbpp/165 ✗ ✗ ❌ Failed 225 View →
Mbpp/166 ✓ ✓ ✅ Perfect 428 View →
Mbpp/167 ✓ ✓ ✅ Perfect 746 View →
Mbpp/168 ✓ ✓ ✅ Perfect 292 View →
Mbpp/170 ✓ ✓ ✅ Perfect 380 View →
Mbpp/171 ✓ ✓ ✅ Perfect 408 View →
Mbpp/172 ✓ ✓ ✅ Perfect 152 View →
Mbpp/222 ✓ ✓ ✅ Perfect 393 View →
Mbpp/223 ✓ ✗ ⚠️ Base Only 1339 View →
Mbpp/224 ✓ ✓ ✅ Perfect 592 View →
Mbpp/226 ✓ ✓ ✅ Perfect 199 View →
Mbpp/227 ✓ ✓ ✅ Perfect 109 View →
Mbpp/230 ✓ ✓ ✅ Perfect 348 View →
Mbpp/232 ✓ ✓ ✅ Perfect 568 View →
Mbpp/233 ✓ ✓ ✅ Perfect 248 View →
Mbpp/234 ✓ ✓ ✅ Perfect 104 View →
Mbpp/235 ✗ ✗ ❌ Empty 0 View →
Mbpp/237 ✗ ✗ ❌ Empty 0 View →
Mbpp/238 ✓ ✓ ✅ Perfect 242 View →
Mbpp/239 ✗ ✗ ❌ Empty 0 View →
Mbpp/240 ✓ ✓ ✅ Perfect 158 View →
Mbpp/242 ✓ ✓ ✅ Perfect 108 View →
Mbpp/244 ✗ ✗ ❌ Empty 0 View →
Mbpp/245 ✗ ✗ ❌ Empty 0 View →
Mbpp/247 ✗ ✗ ❌ Empty 0 View →
Mbpp/250 ✓ ✓ ✅ Perfect 380 View →
Mbpp/251 ✓ ✓ ✅ Perfect 444 View →
Mbpp/252 ✓ ✓ ✅ Perfect 278 View →
Mbpp/253 ✓ ✓ ✅ Perfect 390 View →
Mbpp/255 ✓ ✗ ⚠️ Base Only 411 View →
Mbpp/256 ✓ ✓ ✅ Perfect 583 View →
Mbpp/257 ✓ ✓ ✅ Perfect 285 View →
Mbpp/259 ✗ ✗ ❌ Failed 390 View →
Mbpp/260 ✗ ✗ ❌ Empty 0 View →
Mbpp/261 ✓ ✗ ⚠️ Base Only 160 View →
Mbpp/262 ✓ ✓ ✅ Perfect 351 View →
Mbpp/264 ✓ ✓ ✅ Perfect 333 View →
Mbpp/265 ✓ ✓ ✅ Perfect 634 View →
Mbpp/266 ✓ ✓ ✅ Perfect 472 View →
Mbpp/267 ✓ ✗ ⚠️ Base Only 420 View →
Mbpp/268 ✓ ✓ ✅ Perfect 238 View →
Mbpp/269 ✓ ✓ ✅ Perfect 226 View →
Mbpp/270 ✓ ✓ ✅ Perfect 281 View →
Mbpp/271 ✓ ✓ ✅ Perfect 500 View →
Mbpp/272 ✓ ✓ ✅ Perfect 295 View →
Mbpp/273 ✓ ✓ ✅ Perfect 522 View →
Mbpp/274 ✓ ✓ ✅ Perfect 509 View →
Mbpp/276 ✗ ✗ ❌ Failed 11 View →
Mbpp/277 ✓ ✓ ✅ Perfect 325 View →
Mbpp/278 ✓ ✗ ⚠️ Base Only 371 View →
Mbpp/279 ✗ ✗ ❌ Empty 0 View →
Mbpp/280 ✓ ✓ ✅ Perfect 442 View →
Mbpp/281 ✓ ✓ ✅ Perfect 317 View →
Mbpp/282 ✓ ✓ ✅ Perfect 204 View →
Mbpp/283 ✓ ✓ ✅ Perfect 407 View →
Mbpp/284 ✓ ✓ ✅ Perfect 246 View →
Mbpp/285 ✗ ✓ ⚠️ Plus Only 546 View →
Mbpp/286 ✗ ✗ ❌ Empty 0 View →
Mbpp/287 ✓ ✗ ⚠️ Base Only 346 View →
Mbpp/290 ✓ ✓ ✅ Perfect 525 View →
Mbpp/292 ✓ ✓ ✅ Perfect 258 View →
Mbpp/293 ✗ ✗ ❌ Empty 0 View →
Mbpp/294 ✓ ✗ ⚠️ Base Only 693 View →
Mbpp/296 ✗ ✗ ❌ Failed 1052 View →
Mbpp/297 ✓ ✓ ✅ Perfect 469 View →
Mbpp/299 ✓ ✓ ✅ Perfect 449 View →
Mbpp/300 ✓ ✗ ⚠️ Base Only 287 View →
Mbpp/301 ✗ ✗ ❌ Empty 0 View →
Mbpp/305 ✓ ✗ ⚠️ Base Only 687 View →
Mbpp/306 ✗ ✗ ❌ Empty 0 View →
Mbpp/308 ✗ ✗ ❌ Empty 0 View →
Mbpp/309 ✓ ✓ ✅ Perfect 273 View →
Mbpp/310 ✓ ✓ ✅ Perfect 177 View →
Mbpp/311 ✗ ✗ ❌ Empty 0 View →
Mbpp/312 ✓ ✓ ✅ Perfect 320 View →
Mbpp/388 ✓ ✓ ✅ Perfect 438 View →
Mbpp/389 ✓ ✓ ✅ Perfect 304 View →
Mbpp/390 ✓ ✓ ✅ Perfect 335 View →
Mbpp/391 ✓ ✓ ✅ Perfect 591 View →
Mbpp/392 ✗ ✗ ❌ Empty 0 View →
Mbpp/394 ✓ ✓ ✅ Perfect 154 View →
Mbpp/395 ✓ ✓ ✅ Perfect 323 View →
Mbpp/397 ✓ ✓ ✅ Perfect 135 View →
Mbpp/398 ✗ ✗ ❌ Failed 568 View →
Mbpp/404 ✓ ✓ ✅ Perfect 302 View →
Mbpp/405 ✓ ✓ ✅ Perfect 305 View →
Mbpp/406 ✓ ✗ ⚠️ Base Only 313 View →
Mbpp/409 ✓ ✓ ✅ Perfect 360 View →
Mbpp/410 ✓ ✗ ⚠️ Base Only 563 View →
Mbpp/412 ✓ ✓ ✅ Perfect 288 View →
Mbpp/413 ✓ ✓ ✅ Perfect 395 View →
Mbpp/414 ✓ ✓ ✅ Perfect 205 View →
Mbpp/415 ✗ ✗ ❌ Empty 0 View →
Mbpp/418 ✓ ✓ ✅ Perfect 270 View →
Mbpp/419 ✗ ✗ ❌ Failed 260 View →
Mbpp/420 ✓ ✓ ✅ Perfect 242 View →
Mbpp/421 ✓ ✓ ✅ Perfect 461 View →
Mbpp/422 ✓ ✓ ✅ Perfect 231 View →
Mbpp/424 ✓ ✓ ✅ Perfect 287 View →
Mbpp/425 ✓ ✓ ✅ Perfect 448 View →
Mbpp/426 ✓ ✓ ✅ Perfect 170 View →
Mbpp/427 ✓ ✗ ⚠️ Base Only 504 View →
Mbpp/428 ✓ ✓ ✅ Perfect 638 View →
Mbpp/429 ✓ ✓ ✅ Perfect 286 View →
Mbpp/430 ✗ ✗ ❌ Empty 0 View →
Mbpp/432 ✓ ✓ ✅ Perfect 553 View →
Mbpp/433 ✓ ✓ ✅ Perfect 155 View →
Mbpp/435 ✓ ✓ ✅ Perfect 154 View →
Mbpp/436 ✓ ✓ ✅ Perfect 267 View →
Mbpp/437 ✓ ✓ ✅ Perfect 258 View →
Mbpp/439 ✓ ✓ ✅ Perfect 697 View →
Mbpp/440 ✗ ✗ ❌ Empty 0 View →
Mbpp/441 ✓ ✓ ✅ Perfect 255 View →
Mbpp/445 ✓ ✓ ✅ Perfect 659 View →
Mbpp/446 ✓ ✗ ⚠️ Base Only 488 View →
Mbpp/447 ✓ ✓ ✅ Perfect 282 View →
Mbpp/448 ✗ ✗ ❌ Failed 434 View →
Mbpp/450 ✓ ✓ ✅ Perfect 209 View →
Mbpp/451 ✓ ✗ ⚠️ Base Only 395 View →
Mbpp/453 ✓ ✓ ✅ Perfect 526 View →
Mbpp/454 ✓ ✓ ✅ Perfect 174 View →
Mbpp/455 ✓ ✓ ✅ Perfect 506 View →
Mbpp/456 ✓ ✓ ✅ Perfect 294 View →
Mbpp/457 ✓ ✓ ✅ Perfect 340 View →
Mbpp/458 ✓ ✓ ✅ Perfect 128 View →
Mbpp/459 ✓ ✗ ⚠️ Base Only 164 View →
Mbpp/460 ✓ ✓ ✅ Perfect 191 View →
Mbpp/462 ✗ ✗ ❌ Empty 0 View →
Mbpp/463 ✗ ✗ ❌ Failed 9 View →
Mbpp/465 ✓ ✓ ✅ Perfect 251 View →
Mbpp/468 ✗ ✗ ❌ Empty 0 View →
Mbpp/470 ✓ ✓ ✅ Perfect 317 View →
Mbpp/471 ✓ ✓ ✅ Perfect 354 View →
Mbpp/472 ✓ ✓ ✅ Perfect 800 View →
Mbpp/473 ✗ ✗ ❌ Empty 0 View →
Mbpp/474 ✓ ✓ ✅ Perfect 179 View →
Mbpp/475 ✓ ✓ ✅ Perfect 322 View →
Mbpp/476 ✓ ✓ ✅ Perfect 333 View →
Mbpp/477 ✓ ✓ ✅ Perfect 226 View →
Mbpp/478 ✓ ✓ ✅ Perfect 357 View →
Mbpp/479 ✓ ✓ ✅ Perfect 342 View →
Mbpp/554 ✓ ✓ ✅ Perfect 280 View →
Mbpp/555 ✓ ✓ ✅ Perfect 249 View →
Mbpp/556 ✓ ✗ ⚠️ Base Only 362 View →
Mbpp/557 ✓ ✓ ✅ Perfect 450 View →
Mbpp/558 ✓ ✓ ✅ Perfect 419 View →
Mbpp/559 ✓ ✗ ⚠️ Base Only 607 View →
Mbpp/560 ✓ ✓ ✅ Perfect 413 View →
Mbpp/562 ✓ ✓ ✅ Perfect 233 View →
Mbpp/563 ✓ ✓ ✅ Perfect 336 View →
Mbpp/564 ✓ ✗ ⚠️ Base Only 512 View →
Mbpp/565 ✓ ✓ ✅ Perfect 228 View →
Mbpp/566 ✓ ✓ ✅ Perfect 333 View →
Mbpp/567 ✓ ✓ ✅ Perfect 415 View →
Mbpp/568 ✓ ✓ ✅ Perfect 255 View →
Mbpp/569 ✓ ✓ ✅ Perfect 345 View →
Mbpp/572 ✓ ✓ ✅ Perfect 249 View →
Mbpp/573 ✓ ✓ ✅ Perfect 274 View →
Mbpp/576 ✗ ✗ ❌ Empty 0 View →
Mbpp/577 ✓ ✗ ⚠️ Base Only 311 View →
Mbpp/578 ✓ ✓ ✅ Perfect 459 View →
Mbpp/579 ✓ ✓ ✅ Perfect 579 View →
Mbpp/580 ✗ ✗ ❌ Empty 0 View →
Mbpp/581 ✗ ✗ ❌ Empty 0 View →
Mbpp/583 ✓ ✓ ✅ Perfect 361 View →
Mbpp/585 ✓ ✓ ✅ Perfect 686 View →
Mbpp/586 ✓ ✓ ✅ Perfect 395 View →
Mbpp/587 ✓ ✓ ✅ Perfect 221 View →
Mbpp/588 ✓ ✓ ✅ Perfect 236 View →
Mbpp/589 ✓ ✗ ⚠️ Base Only 480 View →
Mbpp/590 ✗ ✗ ❌ Empty 0 View →
Mbpp/591 ✓ ✓ ✅ Perfect 472 View →
Mbpp/592 ✓ ✓ ✅ Perfect 424 View →
Mbpp/593 ✓ ✗ ⚠️ Base Only 362 View →
Mbpp/594 ✓ ✓ ✅ Perfect 882 View →
Mbpp/596 ✓ ✓ ✅ Perfect 252 View →
Mbpp/597 ✗ ✗ ❌ Failed 1247 View →
Mbpp/598 ✓ ✓ ✅ Perfect 333 View →
Mbpp/599 ✓ ✓ ✅ Perfect 178 View →
Mbpp/600 ✓ ✓ ✅ Perfect 126 View →
Mbpp/602 ✗ ✗ ❌ Empty 0 View →
Mbpp/603 ✗ ✗ ❌ Empty 0 View →
Mbpp/604 ✓ ✓ ✅ Perfect 156 View →
Mbpp/605 ✓ ✓ ✅ Perfect 438 View →
Mbpp/606 ✓ ✓ ✅ Perfect 245 View →
Mbpp/607 ✓ ✓ ✅ Perfect 371 View →
Mbpp/608 ✗ ✗ ❌ Failed 355 View →
Mbpp/610 ✓ ✓ ✅ Perfect 379 View →
Mbpp/611 ✓ ✓ ✅ Perfect 405 View →
Mbpp/612 ✓ ✓ ✅ Perfect 258 View →
Mbpp/614 ✓ ✓ ✅ Perfect 391 View →
Mbpp/615 ✗ ✗ ❌ Failed 9 View →
Mbpp/616 ✓ ✓ ✅ Perfect 443 View →
Mbpp/618 ✓ ✓ ✅ Perfect 343 View →
Mbpp/619 ✓ ✓ ✅ Perfect 358 View →
Mbpp/620 ✗ ✗ ❌ Empty 0 View →
Mbpp/622 ✗ ✗ ❌ Empty 0 View →
Mbpp/623 ✓ ✓ ✅ Perfect 330 View →
Mbpp/624 ✓ ✓ ✅ Perfect 225 View →
Mbpp/626 ✗ ✗ ❌ Empty 0 View →
Mbpp/628 ✓ ✓ ✅ Perfect 265 View →
Mbpp/629 ✓ ✓ ✅ Perfect 262 View →
Mbpp/630 ✓ ✗ ⚠️ Base Only 394 View →
Mbpp/631 ✓ ✓ ✅ Perfect 237 View →
Mbpp/632 ✓ ✓ ✅ Perfect 270 View →
Mbpp/633 ✓ ✓ ✅ Perfect 269 View →
Mbpp/635 ✗ ✓ ⚠️ Plus Only 337 View →
Mbpp/637 ✓ ✓ ✅ Perfect 223 View →
Mbpp/638 ✗ ✗ ❌ Empty 0 View →
Mbpp/639 ✗ ✗ ❌ Failed 9 View →
Mbpp/641 ✓ ✓ ✅ Perfect 292 View →
Mbpp/643 ✓ ✓ ✅ Perfect 305 View →
Mbpp/644 ✓ ✓ ✅ Perfect 169 View →
Mbpp/720 ✓ ✓ ✅ Perfect 152 View →
Mbpp/721 ✗ ✗ ❌ Empty 0 View →
Mbpp/722 ✓ ✓ ✅ Perfect 590 View →
Mbpp/723 ✓ ✓ ✅ Perfect 183 View →
Mbpp/724 ✓ ✓ ✅ Perfect 368 View →
Mbpp/725 ✓ ✓ ✅ Perfect 313 View →
Mbpp/726 ✓ ✓ ✅ Perfect 314 View →
Mbpp/728 ✓ ✓ ✅ Perfect 179 View →
Mbpp/730 ✓ ✓ ✅ Perfect 506 View →
Mbpp/731 ✗ ✗ ❌ Empty 0 View →
Mbpp/732 ✓ ✓ ✅ Perfect 170 View →
Mbpp/733 ✓ ✓ ✅ Perfect 585 View →
Mbpp/734 ✓ ✓ ✅ Perfect 340 View →
Mbpp/735 ✗ ✗ ❌ Empty 0 View →
Mbpp/736 ✓ ✓ ✅ Perfect 376 View →
Mbpp/737 ✓ ✓ ✅ Perfect 351 View →
Mbpp/739 ✓ ✗ ⚠️ Base Only 560 View →
Mbpp/740 ✓ ✓ ✅ Perfect 524 View →
Mbpp/741 ✓ ✗ ⚠️ Base Only 279 View →
Mbpp/742 ✓ ✓ ✅ Perfect 329 View →
Mbpp/743 ✓ ✓ ✅ Perfect 463 View →
Mbpp/744 ✓ ✓ ✅ Perfect 256 View →
Mbpp/745 ✗ ✗ ❌ Failed 9 View →
Mbpp/748 ✓ ✗ ⚠️ Base Only 303 View →
Mbpp/749 ✓ ✓ ✅ Perfect 345 View →
Mbpp/750 ✓ ✓ ✅ Perfect 165 View →
Mbpp/751 ✓ ✓ ✅ Perfect 743 View →
Mbpp/752 ✗ ✗ ❌ Failed 9 View →
Mbpp/753 ✓ ✓ ✅ Perfect 688 View →
Mbpp/754 ✗ ✗ ❌ Failed 336 View →
Mbpp/755 ✗ ✗ ❌ Failed 9 View →
Mbpp/757 ✗ ✗ ❌ Empty 0 View →
Mbpp/758 ✓ ✓ ✅ Perfect 579 View →
Mbpp/759 ✗ ✗ ❌ Empty 0 View →
Mbpp/760 ✓ ✓ ✅ Perfect 467 View →
Mbpp/762 ✓ ✓ ✅ Perfect 433 View →
Mbpp/763 ✓ ✓ ✅ Perfect 759 View →
Mbpp/764 ✓ ✓ ✅ Perfect 336 View →
Mbpp/765 ✗ ✗ ❌ Empty 0 View →
Mbpp/766 ✓ ✓ ✅ Perfect 489 View →
Mbpp/767 ✗ ✗ ❌ Empty 0 View →
Mbpp/769 ✗ ✗ ❌ Empty 0 View →
Mbpp/770 ✓ ✓ ✅ Perfect 370 View →
Mbpp/771 ✓ ✗ ⚠️ Base Only 766 View →
Mbpp/772 ✓ ✓ ✅ Perfect 600 View →
Mbpp/773 ✗ ✗ ❌ Failed 9 View →
Mbpp/775 ✓ ✓ ✅ Perfect 333 View →
Mbpp/777 ✓ ✓ ✅ Perfect 235 View →
Mbpp/778 ✓ ✓ ✅ Perfect 462 View →
Mbpp/780 ✗ ✗ ❌ Failed 16 View →
Mbpp/781 ✗ ✗ ❌ Empty 0 View →
Mbpp/782 ✗ ✗ ❌ Empty 0 View →
Mbpp/784 ✓ ✓ ✅ Perfect 582 View →
Mbpp/785 ✓ ✗ ⚠️ Base Only 915 View →
Mbpp/786 ✓ ✓ ✅ Perfect 351 View →
Mbpp/787 ✗ ✗ ❌ Failed 9 View →
Mbpp/788 ✓ ✓ ✅ Perfect 400 View →
Mbpp/790 ✓ ✗ ⚠️ Base Only 375 View →
Mbpp/791 ✓ ✓ ✅ Perfect 182 View →
Mbpp/792 ✓ ✗ ⚠️ Base Only 115 View →
Mbpp/793 ✓ ✓ ✅ Perfect 592 View →
Mbpp/794 ✗ ✗ ❌ Failed 9 View →
Mbpp/796 ✓ ✓ ✅ Perfect 240 View →
Mbpp/797 ✓ ✓ ✅ Perfect 783 View →
Mbpp/798 ✓ ✓ ✅ Perfect 241 View →
Mbpp/799 ✗ ✗ ❌ Failed 143 View →
Mbpp/800 ✓ ✗ ⚠️ Base Only 266 View →
Mbpp/801 ✗ ✗ ❌ Empty 0 View →
Mbpp/803 ✓ ✓ ✅ Perfect 350 View →
Mbpp/804 ✓ ✓ ✅ Perfect 331 View →
Mbpp/805 ✓ ✓ ✅ Perfect 149 View →
Mbpp/806 ✓ ✗ ⚠️ Base Only 416 View →
Mbpp/807 ✓ ✓ ✅ Perfect 227 View →
Mbpp/808 ✓ ✓ ✅ Perfect 302 View →
Mbpp/809 ✓ ✓ ✅ Perfect 543 View →

💡 总结与建议

总体评价
DeepSeek V4 Pro 在 MBPP 基准上的得分为 Base 78.8% / Plus 68.8%。 该成绩使模型处于中等水平，与 DeepSeek-Coder-6.7B (74.9%/65.6%) 处于同一区间。 主要拖累因素是 13.2% 的空白回答率。
与其他 DeepSeek 模型差距显著
与 DeepSeek 家族其他模型相比，DeepSeek V4 Pro (78.8%/68.8%) 明显落后于：
  • DeepSeek-Coder-V2: 83.5%/76.8%
  • DeepSeek-Coder-33B: 81.7%/74.5%
这可能与代理 API 的配置、模型版本差异或温度参数设置有关。
潜力
在成功生成代码的 328 个任务中，有效通过率为 Base 90.9% / Plus 79.3%。 这表明模型在代码生成能力本身是可观的，主要瓶颈在于 回答稳定性而非代码质量。
改进建议
  1. 降低空白率：调整 API 调用策略（如系统提示、max_tokens 限制）以减少空白回答
  2. 温度优化：尝试 temperature=0 的纯贪婪解码以获得更稳定的输出
  3. 增强边界处理：针对空输入、负数等边界条件增强代码鲁棒性
  4. 与官方结果交叉验证：在 DeepSeek 官方 API 上复现测试，排除代理引入的差异
  5. 补充 HumanEval 测试：进行 HumanEval 基准测试以获得更全面的编程能力画像

DeepSeek V4 Pro MBPP Evaluation Report | Generated: 2026-06-07

Framework: EvalPlus v0.2.0 | 378 MBPP-sanitized tasks | pass@1, temp=0.1

Comparison data: EvalPlus Leaderboard