Skip to content

Fix GSM8K generation evaluation - #121

Merged
haofrank merged 2 commits into
AMD-AGI:mainfrom
amd-nithya:fix/gsm8k-generation-eval
Oct 13, 2026
Merged

haofrank merged 2 commits into
AMD-AGI:mainfrom
amd-nithya:fix/gsm8k-generation-eval

Conversation

@amd-nithya

@amd-nithya amd-nithya commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • isolate GSM8K generation from likelihood tasks and stop at the next Question: marker
  • retain sample logs for GSM8K and HumanEval without enabling --write_out by default
  • keep HumanEval chat-template and unsafe-code flags scoped to its dedicated invocation
  • leave MMLU and HellaSwag completion/log-likelihood behavior unchanged

Validation

  • .venv/bin/python -m pytest -q tests/test_eval_concurrency_scripts.py (51 passed)
  • MiniMax-M2.5 GSM8K canary: 0.26 -> 0.85 (20 samples)
  • DeepSeek-V4 GSM8K canary: 0.02 -> 0.75 (20 samples)
  • all-four GLM regression: MMLU 0.8781, HellaSwag 0.80, GSM8K 0.90, HumanEval 0.95

Per-sample canary artifacts were inspected: the remaining DeepSeek misses were reasoning errors rather than parser failures.

Companion Crystal integration: https://github.com/AMD-AI-CHIEF-OF-STAFF/Crystal/pull/37

@amd-nithya
amd-nithya force-pushed the fix/gsm8k-generation-eval branch from d307390 to 1fcaddd Compare October 12, 2026 21:34
Comment thread Magpie/scripts/benchmark/magpie_bench_remote_compat.sh Outdated
Comment thread Magpie/scripts/benchmark/magpie_bench_remote_compat.sh
Comment thread Magpie/scripts/benchmark/magpie_bench_remote_compat.sh Outdated
@amd-nithya

Copy link
Copy Markdown
Collaborator Author

Addressed all three review comments in 8dd9218:

  • Preserve an explicit MAGPIE_EVAL_CONFIRM_UNSAFE_CODE=true opt-in for other unsafe tasks such as MBPP after HumanEval is isolated.
  • Use HumanEval sample logging's effective default when deciding whether to isolate it in mixed suites.
  • Isolate GSM8K before stock-include routing, while preserving the correct include path for its dedicated invocation.

Added a regression test for each case. Validation: tests/test_eval_concurrency_scripts.py passes (54 tests), Bash syntax validation passes, and the broader related eval set reached 96 passes; the remaining local runtime checks require unavailable optional dependencies or sandbox permissions.

@haofrank haofrank left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@haofrank
haofrank self-requested a review October 13, 2026 00:22
@haofrank
haofrank merged commit 8eb2936 into AMD-AGI:main Oct 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants