Implement branch-aware hybrid transcrypt secrets encryption and document upgrade steps

[gspencergoog]

Summary

This PR introduces branch-aware hybrid encryption and legacy-fallback decryption to the local secrets management filter (transcrypt), enabling a smooth transition from legacy MD5 key derivation to the modern PBKDF2 standard without breaking cross-branch compatibility. It also documents the necessary manual steps for developers to upgrade their local filters.

Changes

• Hybrid & Branch-Aware transcrypt Filter (eval/bin/transcrypt):
  • Dynamic Encryption Selection: Modified the clean filter (git_clean) to inspect the current branch's transcrypt script. If the active branch supports PBKDF2, it encrypts secrets using the more secure -pbkdf2 -iter 100000 settings; otherwise, it falls back to legacy -md MD5.
  • Safe Fallback Decryption: Updated the smudge (git_smudge) and textconv (git_textconv) filters to attempt decryption using PBKDF2 first. If decryption fails, they fall back to legacy MD5, and finally to raw ciphertext. This ensures that checkouts containing a mix of old (MD5) and new (PBKDF2) encrypted files do not fail.
  • Stdout Pollution Prevention: Fixed potential stdout pollution in the smudge and textconv filters by redirecting OpenSSL error streams to /dev/null and using temporary files to safely check decryption success before piping to stdout.
• Documentation (eval/README.md):
  • Added a dedicated Upgrading transcrypt section outlining the upgrade process (bin/transcrypt --upgrade) and the checkout command required to force Git to re-decrypt files using the upgraded filters.
• Dataset Encryption:
  • Re-encrypted eval/datasets/v0_9_prompts.yaml using the new PBKDF2-based filters.

Impact & Risks

• No Breaking Changes: The legacy MD5 fallback in the smudge and textconv filters ensures that existing local repositories, branches without the upgrade, and older commits can still be checked out and decrypted.
• Action Required for Developers: Developers must run bin/transcrypt --upgrade and re-smudge their files to benefit from the upgraded security settings and avoid OpenSSL deprecation warnings.

Testing

• Filter Correctness: Verified that both legacy MD5-encrypted datasets and newly PBKDF2-encrypted datasets smudge and decrypt correctly under the updated script.
• Upgrade Process: Verified that running bin/transcrypt --upgrade followed by git checkout HEAD -- $(git ls-crypt) successfully upgrades the local filters and re-decrypts the datasets.

[jiahaog]

enabling a smooth transition from legacy MD5 key derivation to the modern PBKDF2 standard without breaking cross-branch compatibility

Would dropping this reduce complexity? I think if it's just the both of us working on the evals we should be able to deal with the transition since it's just a one off?

[gspencergoog]

enabling a smooth transition from legacy MD5 key derivation to the modern PBKDF2 standard without breaking cross-branch compatibility

Would dropping this reduce complexity? I think if it's just the both of us working on the evals we should be able to deal with the transition since it's just a one off?

Yes, it would reduce it a little. I was just wanting to make it easy if we wanted to work on "old" branches that were in progress when this lands. Having to merge into an older branch from main once this lands will be a little annoying since you have to (in the middle of a merge) do the dance to re-decrypt.