Tooling to merge and update test vectors

[davidben]

Jumping off of some of the side discussion in #239, and also what I've observed myself, some way to update and add test vectors would be handy.

What I've been doing is including my scripts in PRs like #238. But those scripts have to import existing tests and modify them in-place, then keep all the IDs and test counts updated. This is a bit tedious. Maybe we can build a tool that improves this.

Thinking aloud, now that we have the source field, perhaps the tool could work something like:

go run ./tools/vectormerge testvectors_v1/foo_test.json -import my_new_tests.json \
    -source github/davidben/my-fancy-new-tests -version 0.1

The tool would:

1. Check the schema of my_new_tests.json and error if anything is wrong, but maybe be permissive about things like tcId, numberOfTests, source, undefined notes, etc., since we'll fix them anyway.
2. Look for all existing tests whose source is github/davidben/my-fancy-new-tests. If the version is greater than 0.1, return an error. Otherwise, delete them.
3. Take all tests from my_new_tests.json. Insert them at...
   • The first position of the first deleted test, if any tests were removed
   • The end if no existing tests were removed
4. Set the source of any inserted tests with the specified source and version
5. Renumber tcId and fix numberOfTests
6. Merge the notes fields, warn or error if there are any conflicts
7. Format the resulting JSON canonically

Then a generator's job is to only output standalone JSON files, in whatever formatting is convenient, and the actual merge process is taken care of by a common script.

Thoughts?

[cpu]

100% agree this would be a useful next step and preferable to trying to resurrect and maintain centralized generation code. I tried to hint at this in the development priorities but haven't had time to work on it myself.

The workflow you're sketching out matches what I've had in my mind, though I've been a bit skeptical that Go's JSON capabilities would let us do this without bumping into the canonicalized field order/formatting nits from #162 Maybe we need to push on consensus on whether a fixed ordering of fields would be acceptabe as a pre-req to merge tooling, or target Python as a first prototype.

[davidben]

Yeah, the field order is a pain. I don't feel strongly. Using Python seems fine, as does just not caring and declaring that field order is now alphabetical.

[cpu]

I don't feel strongly either, but my intuition is that these vectors are mainly handled by unmarshaling into per-language representations where the original order doesn't matter at all. With that in mind I'm slightly in favor of just alphabetizing them and using Go so maintenance is easier. I could ping a few downstreams in #162 to see if they have any objections to that.

[cpu]

FWIW the field ordering issue ends up being inconsequential in practice provided we use GOEXPERIMENT=jsonv2 and the json/v2 package, building on the OrderedObject example. I have a big branch where I've built out tooling to replace the Python formatting check in CI and a CLI that can add new vectors, update existing vectors, and replace existing vectors outright.

I want to spend a bit more time experimenting with it as a user, writing some docs, and breaking it into reviewable chunks but I think I've got the core workflow more or less sorted out. As a bonus I was able to expose most of the guts in a Go package so if folks are writing their generators in Go they can side-step a JSON intermediate format + CLI tool invocation and just programmaticallly update/generate the vectors directly. I'll update the generators I wrote in #243 to demo that use-case before calling it done.