Repository navigation
Docs for 26.11.1.post1 and post2: reading large results, columnar, parallel and vertex loads, to_dataframe exactness, kill probe - #363
Merged
Conversation
…loads Document what the post1 glue changes do, checked against the code of integration/post1-candidate-3 (not against the PR text): - databases-and-queries: a "Reading large results" section (row APIs build one dict per row; to_columns, to_arrow, to_dataframe and to_polars read typed column frames with one decoder on both wheels; the rules for column types, nulls, big integers, vectors, datetimes, empty results, batch_size and memory), and the crossings of a native save(). - bulk-loads: defaults of insert_columns, insert_dataframe and insert_arrow, the single-pass row check, columnar loads (native kinds, JVM string columns), "Parallel and async inserts" (what a failed parallel load leaves behind), vertex loads through insert_dataframe and insert_arrow (batch commit, retry bounds, refusal inside a caller's transaction), and a correction: a bad scalar in create_vertices sends the whole call to the older path, which stores an undeclared int as Long instead of Integer. - vectors: how vector columns are read in bulk. - installation: the CDS archive is a file in the bundled JRE. No speed claim is added: the post1 A/B is not finished. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…date wheels Read against the code of the post1 candidate (the wheels' source commit 1d91ee8 plus the tests-only commits after it): - insert_many and create_vertices: the one-pass check raises nothing to the caller. A bad row makes json_bulk_dumps raise ValueError or TypeError inside the package, and insert_many and create_vertices catch it and take the row or older path. Reworded the sentence that said the first bad row raises the same type and message. - Columnar fallbacks on the native package: the numpy datetime64 array and the pandas NaT or NA case belong to insert_columns lists. A DataFrame goes through pyarrow.Table.from_pandas, where they are nulls and stay columnar. - Retry of a NeedRetryException: only the synchronous columnar insert retries (DocumentBatcher.writeChunks, BulkOps.writeChunks). parallel=True and the native row path (BulkOps.insertRows, no transaction of its own) do not. - Vector column rule: ColumnBatcher and ColumnFrames decide by the value type (float[] or double[]), not by the property declaration. Reworded in databases-and-queries and vectors. - Class-data archive: classes_coh.jsa is 15,065,088 bytes, 3,869,721 inside the wheel; the native wheel has no JRE. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…leave part of a call (kill probe, 240 cycles) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…me, an integer column with a null) The docs page 'What the column calls return' now says: to_dataframe() keeps an integer beyond 2**53 exact (an object column), also when the integer and the floats are in different frames of a large result; an integer column with a null is still float64 with NaN in to_columns() and to_dataframe() (rounded), and int64 with a null in to_arrow(). Two tests pin both sentences on the post2 candidate code. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, the limits that stay, and the scope of the kill probe - databases-and-queries: to_dataframe() gives an object column of the exact Python values, also across the frames of a large result; to_dataframe(convert_types=False) on the JVM wheel and the no-bridge fallback still round; an integer column with a null is float64 with NaN in to_columns() and to_dataframe() (rounded), exact in to_arrow() and to_polars(). Checked on the post1 wheel (float64) against the post2 python files on the same wheel (object). - bulk-loads: the kill probe ran on 26.11.1 and was not repeated on later packages. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-commit black 26.10.1) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The first bullet said a killed load loses nothing before the last commit, the next one says a killed create_vertices load on the JVM package can miss earlier rows: the first bullet now names that exception. The probe was also run on the post1 candidate build (2 of 240 cycles stored another count, both prefixes; released 26.11.1: 4 of 240, none a prefix; pooled by script from repros/ingest-crash-20261110 results and results-followup), so the sentence that it was not repeated on later packages was wrong. It has not been run on post2. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898
force-pushed
the
docs/post1-glue
branch
from
November 10, 2026 13:58
f9f2aac to
af0e481
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the docs now say
The pages describe the behavior of the
26.11.1.post1and26.11.1.post2builds, which the current docs do not cover.how-to/databases-and-queries.adocto_columns(),to_arrow(),to_dataframe()andto_polars()build no dict per row, and both packages decode the same column frames, so one query gives the same column types and values onarcadedb(native) andarcadedb-jvm. The section covers the optional libraries,batch_sizeandcolumns, the memory held while frames merge, null handling for each call, the form of a vector cell for each call, dates and decimals, and the result with no rows.to_dataframe()keeps an integer beyond 2^53^ exact when it sits next to a floating value (theobjectcolumn), also when the integer and the floats are in different frames of a large result (to_dataframe: an integer above 2**53 next to a float is rounded by pandas, the other readers keep it exact #340). The two reads that still round it are named:to_dataframe(convert_types=False)onarcadedb-jvm, and the fallback throughto_list()when the Java bridge classes are missing. The limit that stays is stated too: an integer column with a null isfloat64withNaNinto_columns()andto_dataframe(), so it rounds there, whileto_arrow()andto_polars()keep it exact.to_list()bullet points to the new section for large results. The paragraph on record wrappers states how many calls into the librarysave()andVertex.new_edgemake on the native build.how-to/bulk-loads.adocinsert_columnsandinsert_dataframe/insert_arrowrows, and the note that a vertex type loaded through the DataFrame and Arrow calls now commits every 10,000 rows on both packages.insert_dataframeandinsert_arrow". They state which columns stay columnar on each package, which ones take the row path, what a failed parallel load leaves behind (the whole open writer batch is abandoned, nothing is rolled back, count the stored rows after the error), and the retry rule for aNeedRetryException.create_verticesfallback paragraph now separates the scalars that send the whole call to the older path from the values inside a list or dict that send only one chunk, and says why (the stored type of an undeclared property must not depend on which chunk held the bad value). A paragraph on the binaryfloat32matrix path is added.create_verticesload onarcadedb-jvmcan leave part of a call. The wording is the one fixed against the result files in the last docs commit: 240 cycles on the released26.11.1package, 236 with a whole number of calls and 4 with another count (the stored keys not a prefix of the input in those 4), and 2 of 240 on the26.11.1.post1candidate build, both a prefix. The text says that the probe has not been run on26.11.1.post2.how-to/vectors.adochas a paragraph on reading many vectors that links to the new section.installation.adocstates the size of the class-data-sharing archive in the JVM wheel and that the native package has none.Tests
tests/test_column_frames.pygets two tests that pin the new #340 statements, so the docs cannot drift from the code: the integer stays exact when the floats are in a later frame (andto_columns()returns a list of exact values), and an integer beyond 2^53^ in a column with a null is rounded into_columns()andto_dataframe()and exact into_arrow().Checks
mainafter the26.11.1.post2release line was merged (Merge the 26.11.1.post2 release line into main #351), with no conflicts.black==26.10.1 --check,isort==9.0.2 --profile black --check-only,pyflakes, and bandit as intest-jvm.yml(--severity-level low --confidence-level lowoversrc tests) are clean on the touched Python file.npx --yes -p @antora/cli@3.2.1 -p @antora/site-generator@3.2.1 antora docs/antora-playbook.yml, the command of "Build docs site",failure_level: warn) exits 0 with no warnings, and the new anchorsreading-large-resultsandvertex-loadsrender.Refs #340
🤖 Generated with Claude Code