Skip to content

Docs for 26.11.1.post1 and post2: reading large results, columnar, parallel and vertex loads, to_dataframe exactness, kill probe - #363

Merged
tae898 merged 7 commits into
mainfrom
docs/post1-glue
Nov 10, 2026
Merged

tae898 merged 7 commits into
mainfrom
docs/post1-glue

Conversation

@tae898

@tae898 tae898 commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

What the docs now say

The pages describe the behavior of the 26.11.1.post1 and 26.11.1.post2 builds, which the current docs do not cover.

how-to/databases-and-queries.adoc

  • New section "Reading large results": to_columns(), to_arrow(), to_dataframe() and to_polars() build no dict per row, and both packages decode the same column frames, so one query gives the same column types and values on arcadedb (native) and arcadedb-jvm. The section covers the optional libraries, batch_size and columns, the memory held while frames merge, null handling for each call, the form of a vector cell for each call, dates and decimals, and the result with no rows.
  • to_dataframe() keeps an integer beyond 2^53^ exact when it sits next to a floating value (the object column), also when the integer and the floats are in different frames of a large result (to_dataframe: an integer above 2**53 next to a float is rounded by pandas, the other readers keep it exact #340). The two reads that still round it are named: to_dataframe(convert_types=False) on arcadedb-jvm, and the fallback through to_list() when the Java bridge classes are missing. The limit that stays is stated too: an integer column with a null is float64 with NaN in to_columns() and to_dataframe(), so it rounds there, while to_arrow() and to_polars() keep it exact.
  • The to_list() bullet points to the new section for large results. The paragraph on record wrappers states how many calls into the library save() and Vertex.new_edge make on the native build.

how-to/bulk-loads.adoc

  • Defaults table: insert_columns and insert_dataframe / insert_arrow rows, and the note that a vertex type loaded through the DataFrame and Arrow calls now commits every 10,000 rows on both packages.
  • New sections "Columnar loads", "Parallel and async inserts" and "Vertex types in insert_dataframe and insert_arrow". They state which columns stay columnar on each package, which ones take the row path, what a failed parallel load leaves behind (the whole open writer batch is abandoned, nothing is rolled back, count the stored rows after the error), and the retry rule for a NeedRetryException.
  • The create_vertices fallback paragraph now separates the scalars that send the whole call to the older path from the values inside a list or dict that send only one chunk, and says why (the stored type of an undeclared property must not depend on which chunk held the bad value). A paragraph on the binary float32 matrix path is added.
  • Interrupted-load section, "What is in the database": a killed create_vertices load on arcadedb-jvm can leave part of a call. The wording is the one fixed against the result files in the last docs commit: 240 cycles on the released 26.11.1 package, 236 with a whole number of calls and 4 with another count (the stored keys not a prefix of the input in those 4), and 2 of 240 on the 26.11.1.post1 candidate build, both a prefix. The text says that the probe has not been run on 26.11.1.post2.

how-to/vectors.adoc has a paragraph on reading many vectors that links to the new section. installation.adoc states the size of the class-data-sharing archive in the JVM wheel and that the native package has none.

Tests

tests/test_column_frames.py gets two tests that pin the new #340 statements, so the docs cannot drift from the code: the integer stays exact when the floats are in a later frame (and to_columns() returns a list of exact values), and an integer beyond 2^53^ in a column with a null is rounded in to_columns() and to_dataframe() and exact in to_arrow().

Checks

  • Based on main after the 26.11.1.post2 release line was merged (Merge the 26.11.1.post2 release line into main #351), with no conflicts.
  • black==26.10.1 --check, isort==9.0.2 --profile black --check-only, pyflakes, and bandit as in test-jvm.yml (--severity-level low --confidence-level low over src tests) are clean on the touched Python file.
  • The docs build (npx --yes -p @antora/cli@3.2.1 -p @antora/site-generator@3.2.1 antora docs/antora-playbook.yml, the command of "Build docs site", failure_level: warn) exits 0 with no warnings, and the new anchors reading-large-results and vertex-loads render.
  • The tests run in CI, not in this branch's local checks.

Refs #340

🤖 Generated with Claude Code

tae898 and others added 7 commits November 10, 2026 21:41
…loads

Document what the post1 glue changes do, checked against the code of
integration/post1-candidate-3 (not against the PR text):

- databases-and-queries: a "Reading large results" section (row APIs build
  one dict per row; to_columns, to_arrow, to_dataframe and to_polars read
  typed column frames with one decoder on both wheels; the rules for column
  types, nulls, big integers, vectors, datetimes, empty results, batch_size
  and memory), and the crossings of a native save().
- bulk-loads: defaults of insert_columns, insert_dataframe and insert_arrow,
  the single-pass row check, columnar loads (native kinds, JVM string
  columns), "Parallel and async inserts" (what a failed parallel load
  leaves behind), vertex loads through insert_dataframe and insert_arrow
  (batch commit, retry bounds, refusal inside a caller's transaction), and
  a correction: a bad scalar in create_vertices sends the whole call to the
  older path, which stores an undeclared int as Long instead of Integer.
- vectors: how vector columns are read in bulk.
- installation: the CDS archive is a file in the bundled JRE.

No speed claim is added: the post1 A/B is not finished.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…date wheels

Read against the code of the post1 candidate (the wheels' source commit 1d91ee8 plus the tests-only commits after it):
- insert_many and create_vertices: the one-pass check raises nothing to the caller. A bad row makes json_bulk_dumps raise ValueError or TypeError inside the package, and insert_many and create_vertices catch it and take the row or older path. Reworded the sentence that said the first bad row raises the same type and message.
- Columnar fallbacks on the native package: the numpy datetime64 array and the pandas NaT or NA case belong to insert_columns lists. A DataFrame goes through pyarrow.Table.from_pandas, where they are nulls and stay columnar.
- Retry of a NeedRetryException: only the synchronous columnar insert retries (DocumentBatcher.writeChunks, BulkOps.writeChunks). parallel=True and the native row path (BulkOps.insertRows, no transaction of its own) do not.
- Vector column rule: ColumnBatcher and ColumnFrames decide by the value type (float[] or double[]), not by the property declaration. Reworded in databases-and-queries and vectors.
- Class-data archive: classes_coh.jsa is 15,065,088 bytes, 3,869,721 inside the wheel; the native wheel has no JRE.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…leave part of a call (kill probe, 240 cycles)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…me, an integer column with a null)

The docs page 'What the column calls return' now says: to_dataframe() keeps an integer beyond 2**53 exact (an object column), also when the integer and the floats are in different frames of a large result; an integer column with a null is still float64 with NaN in to_columns() and to_dataframe() (rounded), and int64 with a null in to_arrow(). Two tests pin both sentences on the post2 candidate code.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, the limits that stay, and the scope of the kill probe

- databases-and-queries: to_dataframe() gives an object column of the exact Python values, also across the frames of a large result; to_dataframe(convert_types=False) on the JVM wheel and the no-bridge fallback still round; an integer column with a null is float64 with NaN in to_columns() and to_dataframe() (rounded), exact in to_arrow() and to_polars(). Checked on the post1 wheel (float64) against the post2 python files on the same wheel (object).
- bulk-loads: the kill probe ran on 26.11.1 and was not repeated on later packages.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-commit black 26.10.1)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The first bullet said a killed load loses nothing before the last commit, the next one says a killed create_vertices load on the JVM package can miss earlier rows: the first bullet now names that exception. The probe was also run on the post1 candidate build (2 of 240 cycles stored another count, both prefixes; released 26.11.1: 4 of 240, none a prefix; pooled by script from repros/ingest-crash-20261110 results and results-followup), so the sentence that it was not repeated on later packages was wrong. It has not been run on post2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 2f3fb5d into main Nov 10, 2026
42 checks passed
@tae898
tae898 deleted the docs/post1-glue branch November 10, 2026 15:10
@github-actions github-actions Bot added this to the 26.12.1 milestone Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant