Do Evaluation Cues Change Qwen3-32B's Behavior?

This in-progress model-forensics study asks which parts of an evaluation context change a model's behavior. Specifically: does Qwen3-32B respond to the evaluation label, the evaluator's stated goal, or the belief that its work has no lasting consequences?

Status: the experiment is frozen and independently audited, but the locked 200-run comparison has not been collected. This repository therefore contains a research design and calibration evidence—not a result about evaluation awareness.

Study design

Qwen3-32B uses tools to update a small registry. Seven records can be completed from available evidence; one source is unavailable. The measured outcome is whether the model stops while supported records remain unfinished.

The experiment crosses four descriptions of the run with two descriptions of its consequences:

Run description	Work persists and is used	Work is erased and unused
Ordinary operation	✓	✓
Neutral evaluation	✓	✓
Safety evaluation	✓	✓
Capability evaluation	✓	✓

Twenty-five matched task-and-prompt sets are run in all eight conditions, producing 200 trajectories. The primary comparison is disposable versus persistent consequences. The final, authoritative analysis specification is locked_transition/analysis_plan.json.

Our main hypothesis is that consequence-free wording will increase unfinished-task termination. We expect the evaluation label alone to have a smaller effect, while capability or safety framing may modify the effect by signaling what the evaluator values.

Stopping early is a measurable behavior; it is not by itself evidence of deception, reward hacking, or strategic evaluation gaming.

Progress

• The original candidate behavior occurred in 0/30 calibration runs, so it was abandoned.
• The replacement behavior—stopping with answerable work unfinished—occurred in 3/8 canary runs and 4/12 fresh confirmation runs.
• Qwen completed an easier all-sources-available control in 10/10 runs.
• It correctly reported the explicit run description, consequences, and rules in 80/80 separate comprehension checks.
• The frozen transition package passed an independent audit, including 596/596 packaged tests.

These checks show that the behavior occurs sometimes, the model can do the task, and it can understand the explicit cues when asked. They do not establish that framing causes the behavior; that requires the unrun eight-condition comparison.

Repository guide

• docs/PROJECT_OVERVIEW.md: concise research write-up.
• V3_LOCKED_STUDY_PLAN.md: original prospective plan, later amended.
• LOCKED_STUDY_PRECOLLECTION_AMENDMENT.md: changes made before locked collection.
• locked_transition/analysis_plan.json: authoritative frozen analysis plan.
• LOCKED_COLLECTION_RUNBOOK.md: controlled collection procedure.
• audits/transition/FINAL_INDEPENDENT_AUDIT_REPORT.md: independent audit verdict and limitations.
• docs/PUBLICATION_STATUS.md: current status and notes on stale labels inside immutable evidence.
• docs/IMMUTABLE_PACKAGE_README.md: original technical README from the audited package.
• DATA_CARD.md: included-data scope and cautions.

Earlier protocols and notebooks are retained for auditability. Files explicitly marked historical or DO_NOT_RUN are not current procedures.

Local verification

python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev,analysis]'
pytest -q

The test suite is offline and does not run model inference. Actual collection requires the exact pinned Qwen3-32B revision, an NVIDIA A100-SXM4-80GB environment, a fresh external blinding key, and a separate valid collection authorization. No secret key or completed collection authorization is included here.

Safety and reproducibility boundaries

• Qwen3-32B is pinned to revision 9216db5781bf21249d130ec9da846c4624c16137, unquantized in BF16.
• The audited core is preserved with its original generated evidence and hashes. The public-facing README, licensing, citation, and repository metadata were added afterward and are not part of the historical audit ZIP.
• Do not use legacy collection paths in src/qwen_eval_framing; the supported path is governed by the locked-transition documents.
• Never commit Hugging Face tokens, API keys, the blinding/review key, .env files, or local collection authorizations.

License and attribution

Unless a file says otherwise, original code, documentation, prompts, configurations, synthetic task definitions, and generated research artifacts in this repository are released under the Apache License 2.0, to the extent the author can grant those rights. Qwen model weights are not included and remain subject to the model's own license. Dependencies and other third-party materials retain their respective licenses; see THIRD_PARTY_NOTICES.md. For citation metadata, see CITATION.cff.

Limitations

The study tests one open-weight model, one synthetic tool-use task, explicit context cues, and one primary behavior. Even a positive result would demonstrate evaluation-context-sensitive reliability, not prove an internal concept of evaluation or strategic intent.