Question: generator-backed test vectors for known implementation attacks

[programsurf]

I’m opening this issue to ask whether Wycheproof would be interested in a generator-backed workflow for adding more test vectors for known implementation attacks.

Many implementation attacks are described in papers, advisories, PoCs, or bug reports, but not all of them become reusable Wycheproof vectors. A small workflow could help turn some of these known attack conditions into Wycheproof/vectorlint-compatible test cases.

PR #255 is one concrete example: a known ML-DSA implementation bug can be represented as valid/invalid verify vectors where a conforming verifier and a buggy verifier disagree.

Would this kind of workflow be useful for Wycheproof, or is it preferable to submit only the final generated JSON files?

[cpu]

Hi, thanks for opening the issue to ask.

or is it preferable to submit only the final generated JSON files?

We'd prefer to have just the generated JSON files in-repo and have folks maintain their own generator code separately.

I'm working on tooling for making contributing + integrating new JSON vectors easier (#240) so that there will be less friction with that route going forward. I should have some prototypes to share on that issue shortly (and was using #255 as one test-case for evaluation).

[programsurf]

Thanks, that makes sense.

Keeping the generator code separate and submitting only the generated JSON files to Wycheproof sounds like the right model. We can use an external generator to produce schema-valid/vectorlint-compatible JSON vectors for specific known implementation bugs.

It is also great to hear that #255 is useful as a test case for the new contribution tooling. I will follow #240 and adapt future vector submissions to that workflow as it develops.

For future generator-backed submissions, would it be useful to include a short note in each PR describing the source bug class, the conforming-vs-buggy behavior being tested, and the validation command used to check the generated JSON?

[cpu]

For future generator-backed submissions, would it be useful to include a short note in each PR describing the source bug class, the conforming-vs-buggy behavior being tested, and the validation command used to check the generated JSON?

Describing the bug class & the behaviour being tested sounds helpful. It's also useful if you can link to the code used to generate the vectors. I wouldn't worry much about describing the command used to check the generated JSON unless it's something above/beyond what we do in CI automatically.

Looking forward to more contributions :-)