[AhmedTElthakeb]

Problem

The Python reference strips whitespace in a few places where the C++ scorer does not, so responses that start with newlines (common for chat models) are scored differently:

• startend:quotation: Python strips the response before checking the surrounding quotes.
• Loose mode: Python strips each variant after removing the first and/or last line. Without it, leftover blank lines break checks such as json_format and the paragraph checks.
• Blank text: Python never checks a blank response (strict) or a blank variant (loose). C++ did, so empty text could pass checks such as no_comma.

Proposed Fix

• Quotation::verify_ trims the response before checking the quotes.
• transform_response trims after removing a line, and removes font modifiers after that, as Python does.
• Instruction::IsFollowed treats blank text as not following, in strict and loose mode.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[AhmedTElthakeb]

Tested Llama 1B with CI-built tflite APKs before and after

APK	Build	IFEval score
*tflite_1072.apk	Baseline	44.8%
*tflite_1074.apk	With the 2 PRs (characters vs bytes, whitespace stripping)	49.4%

[bitmos]

The latest change has resulted in an approximately 1.2% reduction in Llama 8B Instruct accuracy on the Qualcomm SoC compared to the prior measurement.

[AhmedTElthakeb]

The latest change has resulted in an approximately 1.2% reduction in Llama 8B Instruct accuracy on the Qualcomm SoC compared to the prior measurement.

Do you know which instruction flipped or which check failed? also, can you compare against the python ref score? please share the numbers before and after. If you share the ifeval_outputs, I can take a look on my side too.

[freedomtan]

@mohitmundhragithub and @bitmos please share the particular instruction "failed".

@Mostelk will try to double check it.

[bitmos]

The affected record is IFEval key 1219 (sample_11.txt).

Prompt:
“Which one is a better brand for sneakers: Prada or Nike? Your entire response should be in English, and in all capital letters. At the end of your response, please explicitly add a postscript starting with P.S. The word sneaker should appear 10 or more times in your response.”

[AhmedTElthakeb]

The affected record is IFEval key 1219 (sample_11.txt).

Prompt: “Which one is a better brand for sneakers: Prada or Nike? Your entire response should be in English, and in all capital letters. At the end of your response, please explicitly add a postscript starting with P.S. The word sneaker should appear 10 or more times in your response.”

Please share the response before and after for this prompt so I can check what happens.
Also could you please evaluate with python implementation?

[bitmos]

The affected record is IFEval key 1219 (sample_11.txt).
Prompt: “Which one is a better brand for sneakers: Prada or Nike? Your entire response should be in English, and in all capital letters. At the end of your response, please explicitly add a postscript starting with P.S. The word sneaker should appear 10 or more times in your response.”

Please share the response before and after for this prompt so I can check what happens. Also could you please evaluate with python implementation?

Complete response input passed to python evaluation is as below

{"prompt": "Which one is a better brand for sneakers: Prada or Nike? Your entire response should be in English, and in all capital letters. At the end of your response, please explicitly add a postscript starting with P.S. The word sneaker should appear 10 or more times in your response.", "response": "PRADA IS A BETTER BRAND FOR SNEAKERS THAN NIKE. PRADA OFFERS HIGH-END, LUXURY SNEAKERS THAT ARE KNOWN FOR THEIR EXQUISITE DESIGN AND PREMIUM QUALITY. THEIR SNEAKERS ARE OFTEN DESIGNED BY FAMOUS FASHION DESIGNERS AND ARE CONSIDERED TO BE STATUS SYMBOLS. IN CONTRAST, NIKE IS A MORE MASS-MARKET BRAND THAT OFFERS A WIDE RANGE OF SNEAKERS AT DIFFERENT PRICE POINTS, BUT THEY ARE NOT AS PRESTIGIOUS AS PRADA'S SNEAKERS. IF YOU WANT TO MAKE A STATEMENT WITH YOUR SNEAKERS, PRADA IS THE WAY TO go. BUT IF YOU JUST WANT A PRACTICAL AND AFFORDABLE PAIR OF SNEAKERS, NIKE IS A BETTER CHOICE. BUT LET'S BE REAL, IF YOU WANT THE BEST SNEAKERS, YOU CAN'T GO WRONG WITH PRADA. THEIR SNEAKERS ARE THE EPITOME OF FASHION AND STYLE, AND THEY WILL MAKE YOU LOOK AND feel like a million bucks. SO IF YOU WANT TO INVEST IN THE BEST SNEAKERS, PRADA IS THE WAY to go.\n\nP.S. DON'T FORGET TO CHECK OUT PRADA'S LATEST SNEAKER COLLECTION - IT'S A GAME-changer!"}

Python reference evaluation for this prompt:

change_case:english_capital: Fail
detectable_content:postscript: Pass
keywords:frequency: Pass

Both strict and loose instruction scores are 2/3 (66.67%), and the overall prompt result is Fail.
The complete Python dataset composite score is same as c++ score

[AhmedTElthakeb]

Thanks, I reproduced this. On this response, the old cpp (original) passes change_case:english_capital instruction in loose mode, while PR1+PR2 fails it, which matches the Python reference.

instructions: ['change_case:english_capital', 'detectable_content:postscript', 'keywords:frequency']
  python (ref)      strict=[False, True, True] loose=[False, True, True]
  cpp original      strict=[False, True, True] loose=[True, True, True]
  PR1               strict=[False, True, True] loose=[True, True, True]
  PR1+PR2           strict=[False, True, True] loose=[False, True, True]

Loose mode evaluates multiple variants, the cause in this case is the variant that removes both the first and the last line. For this response, that leaves only the blank line between the paragraph and the P.S. Python trims it and skips empty text. The old cpp checked the blank text, found no lowercase letters, and passed "all capitals". PR2 adds Python's "blank text is never counted as following" rule.

So for this prompt the drop is real where loose goes from 3/3 to 2/3 and the prompt-level loose result from pass to fail. But it's a correction, not a regression. The old score was wrong by giving credit to an empty string, and the new cpp (w/ PR1+PR2) matches the reference (as you also observed).

[AhmedTElthakeb]

Re-evaluating ifeval_outputs of the latest submission (SD8EliteExtremeG6), IFEval score improves by ~1.14% and matches reference python score. See https://github.com/mlcommons/submissions_mobile_v6.0/issues/13