[MarkWolters]

Write inline int8 vectors at on-disk format version 7, not version 6

Targets int8-support (#709).

Problem

#709 adds FeatureId.INLINE_BYTE_VECTORS to allFeatures(), which is the feature set of format version 6. Version 6 shipped in 4.0.1 and 4.0.2, and the Javadoc on allFeatures() says its set must not change. Graphs that use the feature have a layout 4.0.x can't read:

• Header: V6 lists each feature by ordinal, and this one is ordinal 5. 4.0.x defines only ordinals 0–4.
• Node records: each record gains dimension inline bytes, so it's longer and every field after the inline block moves.

I wrote a byte-vector graph with withVersion(6) on #709 and loaded it with jvector-base built from the 4.0.2 tag. It fails while reading the header:

RuntimeException: Error initializing OnDiskGraph at offset 0
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 5 out of bounds for length 5
    at io.github.jbellis.jvector.graph.disk.GraphIndexFormatV6.loadHeaderFeatures(GraphIndexFormatV6.java:135)

Hosts choose the format version they write so they can roll back. For example, DataStax Cassandra passes its SAI-selected version (6 today) to every JVector write. If V6 allows this feature, a host could start writing byte-vector graphs as V6 without ever opting into a new format, and rolling back to 4.0.x would then fail on those indexes. A new version makes that an explicit choice and lets 4.0.x reject the files cleanly as an unknown version.

Change

This follows the version 7 structure on ash-dev (#796), which adds FUSED_ASH the same way. Where the two branches make the same change, the text is identical, so those lines merge cleanly.
This follows the version 7 structure on ash-dev (#796), which adds FUSED_ASH the same way. Where the two branches make the same change, the text is identical, so those lines merge cleanly.

• AbstractGraphIndexFormat.allFeatures(): back to the five features V6 shipped with.
• GraphIndexFormatV6: a protected (int version, Set<FeatureId>) constructor, so a later version can reuse the V6 layout. Identical to ash-dev.
• New GraphIndexFormatV7: extends V6 and adds INLINE_BYTE_VECTORS. Same structure as ash-dev's.
• GraphIndexFormatFactory: registers version 7. Identical to ash-dev.
• OnDiskGraphIndex.CURRENT_VERSION: 6 → 7. Identical to ash-dev. Writers that don't set a version now write V7.
• Example YAMLs (18 files): onDiskIndexVersion 6 → 7, because MultiConfig requires it to equal CURRENT_VERSION. Identical to ash-dev except in the files noted under merge notes.
• Feature headers at the graph's version (removing hard coded version #797): ash-dev already has this fix. Without it, raising CURRENT_VERSION would make a withVersion(6) graph embed version 7 in its PQ and NVQ headers, so the file would no longer match what 4.0.x writes. The commit is applied as is, except for its FusedASH hunk (that class doesn't exist here). InlineByteVectors gets the same versioned writeHeader(out, version) as InlineVectors. Its header is empty, so its bytes are unchanged.
• Docs: UPGRADING.md gets an "On-disk format version 7" section, worded like ash-dev's. 709.feature.md now says int8 indexes need version 7.

On-disk effect

• V7: new. It is the V6 layout plus INLINE_BYTE_VECTORS. 4.0.x rejects it as an unsupported version.
• V6 and below: unchanged and byte-identical to 4.0.x, including PQ and NVQ headers when written with withVersion(6). A V6 writer now rejects INLINE_BYTE_VECTORS: Feature INLINE_BYTE_VECTORS is not supported by version 6.
• Default writes: now V7. Callers who need 4.0.x to read their files must pass withVersion(6), as UPGRADING.md describes.

Testing

• TestGraphIndexFormatFactory: V6 excludes INLINE_BYTE_VECTORS; V7 supports every feature.
• TestOnDiskGraphIndex.testInlineByteVectorsRequiresVersion7: a V6 writer rejects the feature. The existing testInlineByteVectorsRoundTrip now writes V7 by default.
• TestFeatureHeaderVersion (from removing hard coded version #797, without its FUSED_ASH case): for every version from 2 to CURRENT_VERSION, PQ, NVQ and separated NVQ headers are written at that version.
• TestDatasetPrefetching: its YAML uses CURRENT_VERSION instead of a hard-coded 6.
• Full jvector-tests (421) and jvector-examples (127) suites: 0 failures.

Merge notes with ash-dev

Whichever of this PR and #796 reaches main second will need to resolve these:

• FeatureId ordinals: both branches append their feature as ordinal 5. Ordinals are the on-disk feature IDs in V6+ headers, so the second branch must move its feature to ordinal 6. Neither is released, but V7 files written for testing before the merge won't load afterwards.
• GraphIndexFormatV7: both branches add this file. Keep one class whose supportedFeatures() adds both features.
• TestGraphIndexFormatFactory, UPGRADING.md, 709.feature.md: list both features for V7 and exclude both from V6.
• TestFeatureHeaderVersion: both branches add it. Take ash-dev's version, which includes the FUSED_ASH case.
• run-config.yml: ash-dev's version line carries an ASH-specific comment.
• dpr-gemma-10M.yml, dpr-gemma-1M.yml: ash-dev deletes these in favour of lowercase -10m/-1m copies. Take the delete.
• TestDatasetPrefetching: ash-dev also bumps yamlSchemaVersion to 2. Take its version.

[github-actions]

Before you submit for review:

• Does your PR follow guidelines from CONTRIBUTIONS.md?
• Did you summarize what this PR does clearly and concisely?
• Did you include performance data for changes which may be performance impacting?
• Did you include useful docs for any user-facing changes or features?
• Did you include useful javadocs for developer oriented changes, explaining new concepts or key changes?
• Did you rebase your branch onto the latest main for regression testing and PR submission?
• Did you trigger regression testing via Run Bench Main and review results?
• Did you adhere to the code formatting guidelines (TBD)
• Did you group your changes for easy review, providing meaningful descriptions for each commit?
• Did you ensure that all files contain the correct copyright header?
• Did you add documentation for this feature to the release notes directory?

If you did not complete any of these, then please explain below.

[r-devulap]

LGTM, thanks for fixing this @MarkWolters :)

[r-devulap]

Whichever of this PR and #796 reaches main second will need to resolve these:

Lets beat @tlwillke to it :)