Repository navigation
Conversation
…d a PQ entropy analysis program.
…hout loading all vectors first). Updated DistancesASH and DistancesPQ to take advantage of this. The original SiftLoader remains intact.
…nto the vector header.
…is a multiple of 64 bits.
…is a multiple of 64 bits.
…SH to acceptt landmark count.
…pooled version for performance purposes though.
…tch and mixed landmarks.
… scoring. Also made a slight change to limit copying query info from query pool (only at landmark run boundaries) during block scoring.
… scoring. Also made a slight change to limit copying query info from query pool (only at landmark run boundaries) during block scoring.
…rainer. Netlib dependencies added.
…vecs loader to SiftLoader.
…ding payload alignment exception for recall benchmarking purposes.
…ll fixes. ITQ and multi-landmarks work.
…t. Does not work yet.
…ed vectors. Recall and perf look acceptable.
…ed layout with header interleaving. Introduced ashMaskedAdd_512.
…unners The 1 s timeouts only guard against a read loop that never ends, but they also cover the test class's first use of FvecFileVectorValues; on the JDK 24 AVX-512 runner that pushed one 32-read test just over the limit. 10 s still catches a spin.
…low CI runners" This reverts commit 56d598f.
… fast scoring Every stored offset now includes -scale * <A mu_c, code>, as 2- and 4-bit codes already did, so asymmetric scoring at any width and landmark count needs only Aq. This removes the per-landmark centered queries A(q - mu_c), their preparation and the masked-add entry points that existed only for them. Code-to-code scoring adds the term back from the stored code. The FP64 projection copy used for landmark projections now holds the FP32 values that models persist, so a trained model and its reloaded copy derive identical landmark terms. 3- and 5- to 9-bit codes are now offset-binary fields: dimension j is the unsigned b-bit field at bits [j*b, (j+1)*b), with value field - (2^b - 1)/2. The values match the previous sign-plus-magnitude-bits layout, which needed a separate sign plane padded to 64-bit words. Because each field is linear in its bits, any 4 bits of a code have a 16-entry query table, so these widths use the existing nibble-LUT kernels for block and single-vector scoring, and fused graphs accept every width from 1 to 9. In fused graphs these widths store field blocks: each unit of eight dimensions holds every lane's b code bytes contiguously. The query Aq is rounded once to INT8, and a new native entry, ash_score_neighborhood_i8_c1, gathers each lane's fields, rotates them into bytes (vpmultishiftqb) and accumulates exact integer dot products (vpdpbusd); it is compiled for the AVX3_DL tier, with Highway's equivalents elsewhere. On a 32-lane neighborhood it is 1.19x faster than the LUT kernel at 3 bits and 2.05x at 8 bits on an Ice Lake class CPU, and far faster where AVX-512 is unavailable. Rounding the query moved exhaustive recall@10 on 1M datasets by at most 0.0003. A Java path with the same arithmetic covers other providers and multiple landmarks. 1-, 2- and 4-bit layouts and kernels are unchanged. Old and new code matched exhaustive recall@10 within 0.00015 at widths 1, 2, 3, 4 and 8 and C = 1, 16 and 64 on three 100k datasets and DPR-Gemma 1M, with the same scoring and encoding throughput at 1 and 2 bits.
…t VNNI kernel Automatic query projection now tries INT8 first, then FP16 with its FP32 range fallback, then FP32. INT8 was the fastest projection on every CPU measured: 1.03-1.10x FP16 on an AMX CPU and 1.2-1.5x FP32 on Ice Lake class CPUs. The query is now quantized to two INT8 digits with one scale: a coarse digit and its rounding remainder times 254. Rounding the query, not the matrix rows, caused nearly all INT8 projection error; with the second digit, exhaustive recall@10 on ada002-100k matched FP32 projection within 0.0002 at 2 and 4 bits, where single-digit INT8 lost 0.0018 and 0.0048. Both digits run through the same exact-integer kernel, and the oneDNN path computes the same result. On every AVX-512 VNNI CPU the direct tile kernel now serves single queries and batches: 512-bit tiles on CPUs with AMX, where they beat oneDNN's AMX path, and 256-bit tiles otherwise, where wide integer dot products slow the following graph search. oneDNN remains for AVX2-VNNI CPUs without AVX-512.
…rs it Offset-binary codes (3 and 5-9 bits) score single vectors with SIMD only when the backend supports canonical LUT scoring. The Panama backend supports the 2- and 4-bit LUT kernels but not that one, so forcing SIMD there threw by design and failed the test on Windows, where the native library is not loaded. The test now requests SIMD per bit width only when the backend can honor it.
…der, and API defaults - ASHVectors publishes 1-bit packed sign words as one immutable holder after they are filled, so concurrent first queries never score a partial array or one packed for another block size. - NativeBlasThreads reference counts scopes on a process-wide BLAS setter (OpenBLAS before 0.3.27): the first scope sets the count and the last restores it, so overlapping scopes on other threads no longer undo each other. - GraphSearcher.searchOneLayer uses the score provider it is given for adaptive termination. - VectorUtilSupport.ashMaskedAdd_512 has a portable default, so outside implementations compile. - KMeansPlusPlusClusterer reports its verbose progress through the logger. - MemorySegmentReader and MMapReader read single bytes without allocating.
Every Maven build printed each shell command of build_native_lib.sh plus the full Meson and Ninja output. By default the script now prints one line when each library starts building and one with its build time when it finishes. Meson and Ninja output goes to target/meson-build-<arch>.setup.log and .compile.log, and a failing step prints the end of its log and the log's path. Errors and the script's own warnings still print. Setting JVECTOR_NATIVE_BUILD_VERBOSE=1 restores all output, including the command trace.
…ipeline line A compression of None means the stage scores full-precision vectors, so the pipeline line now says FP32 instead of None (e.g. 'inMemory: build FP32 -> search FP32 -> rerank None'). Rerank keeps None, since that step is not done.
Every Maven run printed a start and finish line per native library even when Ninja had nothing to do, so repeated builds (IDE reimports, multi-step runs) filled the console. The start line now prints only for an unconfigured build directory, the slow first build, and the finish line only when Ninja rebuilt something; an up-to-date build prints nothing.
Every Maven run reran meson setup --reconfigure. For the oneDNN add-on that regenerated its CMake subproject and relinked the library each time, adding 2-3 s and a 'built' line to every build. Each build directory now records its setup arguments and is reconfigured only when they change; meson compile still regenerates on its own when a meson.build file changes. An up-to-date build now takes about 0.3 s and prints nothing.
The INT8 projection tests fail on one CI runner by one or two ULPs against the oracle but pass on every machine we can reproduce on. The tile and oracle assertions now name the CPU vendor, model, the flags that select the INT8 path (VNNI, AMX, AVX-512 FP16/BF16) and the oneDNN override, so a failure shows which path ran. The message is built only when an assertion fails.
Vector API lane reductions do not fix their summation order, so an interpreted and a compiled scoring of the same 1-bit block can differ by one ULP; under JDK 25 the test's exact comparison failed on that alone (e.g. 0.61249167 vs 0.6124917). It now compares within 1e-5. Against the code before the fix it still fails every run, with errors of 0.02 to 0.09.
r-devulap
reviewed
Nov 12, 2026
| return pointer ? static_cast<const Context*>(pointer)->retained_bytes.load(std::memory_order_acquire) : 0; | ||
| } | ||
|
|
||
| JV_ASH_INT8_TARGET int jvector_ash_int8_project_batch(void* pointer, const float* input, int count, |
Contributor
There was a problem hiding this comment.
With the addition of hand written VNNI kernels, this ideally belongs to the Highway module which has all the infrastructure to deal with SIMD code (compile time and runtime dispatch). Only problem is I am not sure how to cleanly separate the AMX portion which relies on onednn.
…tart FvecFileVectorValues loads the vectorization provider, possibly a native library, when its class initializes. That happened inside whichever timed test ran first and took 111 ms of a 1 s limit on an idle server, more on a busy CI runner. Initializing the class in @BeforeClass leaves each timeout covering only the read loop it guards; the limit is unchanged.
r-devulap
reviewed
Nov 12, 2026
…al step The projected value is (coarse + fine / 254) * scales. Instead of turning off floating-point contraction for the whole library, combine the two digit dot products exactly in 64-bit integers first, coarse * 254 + fine, so the remaining steps are one conversion and multiplications only. With no product added to anything there is nothing to fuse into an FMA, every build rounds the same way, and the combine loop stays free to vectorize. The test oracle uses the same expression, so the INT8 tests still compare exactly.
r-devulap
reviewed
Nov 12, 2026
| } | ||
|
|
||
| extern "C" { | ||
| void* jvector_ash_int8_create(const float* matrix, int rows, int columns, jvector_ash_fp16_result* result) { |
Contributor
There was a problem hiding this comment.
I was trying to think of ways we could untangle the SIMD portion in this file and move it Highway module. Could INT8 be modeled like the FP32 kernels? FP32 keeps the model in Java and calls a stateless Highway kernel (ash_project_batch_f32), so ISA dispatch comes from Highway. INT8 has C++ own the model and do its own ISA detection and VNNI intrinsics. If Java owned the int8 codes, scales and sums and a Highway kernel did the VNNI dot (SumOfMulQuadAccumulate maps to vpdpbusd), INT8 would benefit from all the Highway module infrastructure which we would avoid duplicating here. Java could still dispatch to oneDNN in the cases Highway doesn't cover, so the choice lives in one place. Is there a reason it can't?
A JIT deoptimization inside the measured window can make the JVM's own VarHandle linkage allocate a few hundred bytes. Require one of up to three windows to be allocation-free; a real allocation appears in every window.
A one-second timeout measured the CI machine as well as the read loop and failed under load after the test body had finished. The test channel now fails after a fixed number of read calls, which catches a spinning reader deterministically, and the EOF test also checks its call count.
A layer search expands a scored node only while it ranks among the best rerankK accepted nodes scored so far, or, with adaptive termination, within the cutoff of the rerankK-th; that bar only rises. A node that misses it when scored can never be expanded by that search, so it now goes to an unsorted deferred list instead of the candidate heap, and returns to the candidates at the next layer search, so later layers and resume() see the same candidates as before. Most scored neighbors are rejected with one comparison, as in DiskANN's bounded search list. The same top-rerankK heap drives adaptive termination, replacing its membership hash set: the visited set already keeps scored nodes unique, so the extra offer on each pop and the membership lookups are gone. Threshold searches keep every candidate. Results, scores, and visited and expanded counts are unchanged; a test compares bounded and unbounded searchers across stopping rules, filters, hierarchy and resume().
Graph views gather an expanded node's unvisited neighbors and score them in one batch when the score function supports it. ASH symmetric construction scoring gathers the neighbors' codes into one contiguous buffer and scores them in one bounded native call that reuses the pair kernel, so batch and single scores are identical.
ASH score functions now declare that they require adaptive stopping, and GraphSearcher stops adaptively whenever the score function requires it, whatever SearchScoreProvider wraps it. An ASH search therefore cannot fall back to the standard rule because an application wrapped its scores in a provider that did not ask for adaptive stopping; GraphSearcher.useAdaptiveTermination(false) remains the explicit override. A test checks plain, adaptive-requesting and application-written providers against each other and the override against a standard search.
ASH clamped its DOT_PRODUCT similarity at zero, so every estimate at or below -1 tied. That was a workaround for hierarchy descent starting from a zero floor, which now starts from negative infinity. Use (1 + dot) / 2 unclamped in Java and native, the scale exact DOT_PRODUCT scoring uses. Add TestUnnormalizedVectors: dot-product search over vectors whose lengths vary 100x must rank by the full dot product for exact, PQ and ASH scoring, by brute force and through a graph, judged against the same directions at unit length and against norm-blind ranking.
… scorer Adaptive stopping read graph scores as distances between unit-length vectors, so on vectors of other lengths it fell back to the standard rule or misjudged the margin, and it applied only to ASH. It now reads scores as true metric distances: d^2 = 1 / score - 1 for Euclidean, 1 - score for cosine, and for dot products the distance after the standard reduction of inner-product search to nearest-neighbor search, measured from an upper bound on the query's dot products, its length times the largest stored vector length. The margin, about 0.005 on the distance, means the same thing for every similarity function and every vector length, and on unit-length vectors the rule decides as before. A search that stops adaptively expands at least the nodes the standard rule would, and at most the nodes a standard search with twice the beam would. Without the cap, a relative margin admits very many candidates when distances are concentrated. The bar below which scored nodes are skipped never falls during a search, so skipped nodes stay skipped. The largest vector length comes from the vectors. ASH and PQ models record it for every vector they encode or whose codes they hold, and save it with the model; the inline and separated vector features and compaction store it in the index header, at format version 7. A length that is unknown, as for a model loaded from an older format, keeps dot-product searches on the standard rule. Searches through DefaultSearchScoreProvider.exact and forSimilarity, and all ASH searches, stop adaptively; providers built with the plain constructors keep the standard rule, and GraphSearcher.useAdaptiveTermination(false) turns adaptive stopping off.
Measured against standard stopping swept over overquery, at 1M and 10M, with ASH and PQ scores, with and without hierarchy, adaptive stopping gave less recall at the same throughput or nodes visited: raising the overquery reranks more candidates, which adaptive stopping does not. GraphSearcher therefore disables it by default, and useAdaptiveTermination is marked experimental. Searches that enable it behave as before. Graph construction keeps its behavior: the builder's searchers enable adaptive stopping, which applies where the build provider asks for it, as ASH's does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds ASH (Asymmetric Scalar Hashing) quantization, introduced in ASH: Asymmetric Scalar Hashing With Learned Dimensionality Reduction for High-Fidelity Vector Quantization (Tepper and Willke, CIKM 2026). ASH reduces the dimension of database vectors with a learned orthonormal projection, then quantizes each projected dimension to a few bits, while keeping queries in their original form. At the same code size it is more accurate than additive and scalar quantizers, trains and encodes quickly, and scores with SIMD lookups. This PR brings ASH to a graph index for the first time.
AsymmetricHashingtrains an ITQ or random projection;encodeAllproducesASHVectors, aCompressedVectorswith codes in compact shared pages. Graph indexes use one landmark.ASHVectorsscore construction and search.BuildScoreProvider.ashBuildScoreProviderbuilds by comparing stored codes (symmetric scoring).FusedASH(FeatureId.FUSED_ASH) stores neighbors' codes with each adjacency list and the model in the index header, at 1, 2 or 4 bits per dimension, with NVQ or full-precision reranking.QueryPreparer/PreparedQueryseparate projection from search, andBatchingExecutorqueues concurrent searches with bounded admission so that waiting queries share one batched projection.Release notes:
796.feature.md,796.enhancement.md. Documentation starts at ASH configuration and defaults and Integrating ASH queries into a database.Compatibility
CURRENT_VERSIONand the writer defaults are now 7.withVersion(6)andwrite(out, 6)keeps writing files 4.0.x reads, and a version 6 writer rejects fused ASH. DataStax Cassandra already passes its SAI-selected version to every JVector write, so it will not write version 7 until it adds an SAI version that selects it. See UPGRADING.md.OnDiskGraphIndexCompactordoes not yet compact fused ASH indexes.Dependencies
dev.ludovic.netlib:blasanddev.ludovic.netlib:lapack3.0.3 (MIT), which bringnet.sourceforge.f2j:arpack_combined_all0.1 (BSD). They are used only for ASH training, and the publishedjvectorartifact declares them. netlib calls the system BLAS and LAPACK, such as MKL or OpenBLAS, and falls back to Java implementations when none is installed, at a cost in training time.libjvector_ash_fp16-<arch>.so(19 MB on x86-64), shipped in the native JAR with its license and third-party notices. Building it needs CMake 3.13 or later;-Dnative.onednn=falsebuilds without it.Native interface
libjvector(query projection, symmetric construction scoring, neighborhood and single-vector scoring) are compatible additions under the existing handshake. Java looks each one up separately, so an older native library keeps the Java implementations; an explicitly native ASH benchmark fails instead of measuring a fallback.jvector_ash_fp16_abi, currently 1) and status codes, and no C++ exception crosses it. It loads only if present and compatible; otherwise FP16 and INT8 projection are unavailable and queries use the Highway FP32 path. See Native ABI compatibility.Validation
mvn -B clean install -DskipSIMD=false -DskipScalar=trueon JDK 22 with the native provider: 825 tests, 0 failures or errors, 6 skipped.Benchmarking
Fused ASH versus fused PQ at the same stored bytes per vector, including ASH's 5-byte header, on three large datasets. ASH has lower construction time in 6 of 6 configurations, higher recall in 12 of 12 cases and higher QPS in 22 of 24.
Construction
Total construction seconds (training, encoding, NVQ, graph build and write); lower is better.
Recall and throughput
Values: PQ → ASH. Rerank 1x and 2x rescore 10 and 20 candidates with NVQ.
PQ is marginally faster (about 4%) in the two MPNet 10M 32x cases on one core. Single-core QPS varies between graphs built from the same settings: an earlier ASH graph for that case measured 10,414 / 7,838 QPS, above PQ.
Method
One Intel Xeon 6985P-C socket (48 physical cores, two NUMA nodes), JDK 22, native provider. On-disk format 7, fused graph, degree 32, efConstruction 100, overflow 1.2, hierarchy and refinement on, NVQ reranking, search pruning off, top 10. ASH: ITQ, one landmark, D/2 projected dimensions, symmetric construction scoring, queued execution with batches of up to 8 and FP16 query projection. PQ: 8-bit subspaces sized to the same bytes, direct execution. Construction used one thread on each of 48 cores with memory interleaved across both nodes; queries ran in separate JVMs on 1 or 8 cores of one node. Construction is the mean of two builds; QPS is the mean of three samples of at least 3 s against the first build. PQ results come from an earlier run with the same machine, JDK, placement, inputs and graph parameters. Full details: ASH performance. The study can be rerun with Reproducing ASH versus PQ. During review we plan to rerun these datasets with the one-JVM recipes, run the full set of 1M datasets, and repeat both on JDK 25.