From ee79f39c3aa8fa0eccb3fda027734874c82adfe1 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Mon, 9 Nov 2026 23:46:08 +0900 Subject: [PATCH 1/7] Docs for post1: reading large results, columnar, parallel and vertex loads Document what the post1 glue changes do, checked against the code of integration/post1-candidate-3 (not against the PR text): - databases-and-queries: a "Reading large results" section (row APIs build one dict per row; to_columns, to_arrow, to_dataframe and to_polars read typed column frames with one decoder on both wheels; the rules for column types, nulls, big integers, vectors, datetimes, empty results, batch_size and memory), and the crossings of a native save(). - bulk-loads: defaults of insert_columns, insert_dataframe and insert_arrow, the single-pass row check, columnar loads (native kinds, JVM string columns), "Parallel and async inserts" (what a failed parallel load leaves behind), vertex loads through insert_dataframe and insert_arrow (batch commit, retry bounds, refusal inside a caller's transaction), and a correction: a bad scalar in create_vertices sends the whole call to the older path, which stores an undeclared int as Long instead of Integer. - vectors: how vector columns are read in bulk. - installation: the CDS archive is a file in the bundled JRE. No speed claim is added: the post1 A/B is not finished. Co-Authored-By: Claude Sonnet 5.5 --- .../modules/ROOT/pages/how-to/bulk-loads.adoc | 58 ++++++++++++++++++- .../pages/how-to/databases-and-queries.adoc | 53 ++++++++++++++++- docs/modules/ROOT/pages/how-to/vectors.adoc | 2 + docs/modules/ROOT/pages/installation.adoc | 2 +- 4 files changed, 110 insertions(+), 5 deletions(-) diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index 8ac3b1c..22110a8 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -1,5 +1,5 @@ = Bulk loads and commit size -:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, how insert_many treats Python's cyclic garbage collector on arcadedb-jvm, which property shapes a graph batch loads in bulk on the JVM build, how to pass time-series columns to append_samples (the address path is native only), and a known limitation: a graph batch does not reuse the space of deleted edges. +:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, how insert_many treats Python's cyclic garbage collector on arcadedb-jvm, which property shapes a graph batch loads in bulk on the JVM build, how to pass time-series columns to append_samples (the address path is native only), a known limitation: a graph batch does not reuse the space of deleted edges, and how columnar loads, parallel loads and vertex loads through insert_dataframe and insert_arrow behave, including what a failed parallel load leaves behind. `insert_many`, `graph_batch` and `GraphBatch.create_vertices` load data in transactions. The commit size is the number of rows in one transaction. It decides how much work the engine holds in memory before it commits, and how many rows stay in the database when a load fails. Data that comes as a file, a DataFrame or an Arrow table has its own pages: xref:how-to/import-export.adoc[Importing and exporting data] and xref:how-to/dataframes-and-columns.adoc[DataFrames, Arrow and columns]. @@ -20,9 +20,17 @@ | `batch.create_vertices(type_name, rows)` | 1000 rows for each call into the engine, one transaction for each call (the `commit_every` of the batch) | 100000 rows for each call into the engine + +| `db.insert_columns(type_name, columns)` +| `commit_every=10000` +| `commit_every=10000` + +| `db.insert_dataframe(type_name, df)` and `db.insert_arrow(type_name, table)` +| `batch_size=10000`, one transaction for each batch +| `batch_size=10000`, one transaction for each batch |=== -The JVM defaults are the same as before. They stay until someone measures them on the JVM build. +The JVM defaults of `insert_many` and `graph_batch` are the same as before. They stay until someone measures them on the JVM build. A vertex type loaded with `insert_dataframe` or `insert_arrow` now commits every 10,000 rows on both packages: see <>. An explicit value always wins. `commit_every=0` gives one transaction for `insert_many` and one commit for each flush for `graph_batch`, on both packages. In a transaction that you open yourself, `insert_many` commits nothing. `graph_batch` refuses to run in one: see xref:how-to/transactions.adoc#graph-batch[GraphBatch inside a transaction]. The durability that a load gets is on xref:how-to/durability.adoc[Durability settings], and the async executor on xref:how-to/async-executor.adoc[The async executor]. @@ -57,6 +65,46 @@ Because the two packages have different defaults, the same program can keep diff On both packages, `insert_many` keeps rows with `datetime`, `date`, `Decimal`, `bytes`, numpy scalar, or pandas `NA` and `NaT` values on the bulk path. On the `arcadedb` (native) package, a row with a value the bulk text cannot carry (NaN or infinity, an integer beyond 64 bits, a date-time or `Decimal` inside a list or dict, or a `datetime.time`) goes in with its own statement, in the order given. +`insert_many` and `create_vertices` check their rows while they write the JSON text, in one pass over the rows, not in a loop before it. The first bad row of a batch raises the same exception type and message as before. The message names the bad value, not a row number. + +[[columnar-loads]] +== Columnar loads + +`insert_columns` takes one column for each property. `insert_arrow` and `insert_dataframe` read an Arrow table or a pandas or Polars DataFrame as columns, in batches of `batch_size` rows. + +On the `arcadedb` (native) package, a batch whose columns the library's buffers can carry crosses in one call, with no Python dict per row and no JSON text. These columns stay columnar: integers, floats, bools, text, date-times (`datetime`, `pandas.Timestamp`, and Arrow timestamps of any unit), dates, `Decimal`, and `bytes` or `bytearray`. The stored values are the ones the row path stores. + +A column the buffers cannot carry makes the batch take the row path, which stores the same values through `insert_many`. These are a column that mixes types, a nested value, a numpy `datetime64` array, an integer beyond 64 bits, a string with a lone surrogate, a non-finite `Decimal`, and a pandas `NaT` or `NA` column. `ARCADEDB_NATIVE_BULK=0` in the environment turns the columnar path off. + +On the `arcadedb-jvm` package, a column of only `str` and `None` crosses as one UTF-8 buffer with its row offsets, and a mask when a row is null. A column that mixes types, is empty, or holds more than 2 GiB of text crosses value by value. A column with a lone surrogate also crosses value by value, and is refused with the error it always raised, before anything is written. + +[[parallel-inserts]] +== Parallel and async inserts + +`insert_many(parallel=True)` and `insert_columns(parallel=True)` hand the records to the async executor's writers and wait for the writers before they return. `insert_arrow` and `insert_dataframe` do the same for a document type, one batch at a time. A vertex type refuses `parallel=True` there with a `ValueError`. + +A parallel call opens no transaction, so `commit_every` does not apply. Each writer commits every `arcadedb.asyncTxBatchSize` records (10,240 by default, and `async_executor().set_commit_every()` changes it). On the native package, `insert_columns(parallel=True)` sends the columns to the library in one call, and the library hands each record to the writers. A column set the buffers cannot carry goes as one dict per row through `insert_many(parallel=True)`. + +What a failure leaves behind: + +* A record that a writer cannot store, for example a duplicate key, abandons the whole batch that writer had open, up to 10,240 records. The valid records of that batch are not stored either. +* The call waits for all writers, then raises `ArcadeDBError`. The message gives the number of failed records and the first failure. +* Nothing is rolled back. Batches of other writers, and earlier batches of the same call, may have been stored. After such an error, count the stored rows with a query. Do not assume all or none. +* On the native package, an interrupt (`KeyboardInterrupt`) during a parallel call waits for the writers before it propagates, so no writer still runs when the exception reaches you. +* In a transaction you opened, `insert_arrow` and `insert_dataframe` with `parallel=True` do not join it. The writers commit on their own, and a rollback of your transaction does not remove those rows. + +For one record at a time, `db.async_executor().create_record(document, callback=..., error_callback=...)` queues the record. The writers call `error_callback` with the exception when they reject it (a duplicate key under a `UNIQUE` index, or a batch abandoned at a failed commit). Without an `error_callback`, the failure reaches only the executor-wide `on_error` handler, if you set one. Call `wait_completion()` before you read the records back. + +[[vertex-loads]] +== Vertex types in `insert_dataframe` and `insert_arrow` + +On both packages, `insert_dataframe` and `insert_arrow` on a vertex type load through the same columnar insert as a document type. They no longer go through `GraphBatch.create_vertices`. Use `graph_batch()` when edges follow or when you need the RIDs of the vertices. + +* The load runs with the write-ahead log setting of the database. `graph_batch()` has its own `use_wal` argument. +* Each batch of `batch_size` rows (10,000 by default) is one transaction. A failed batch is rolled back, earlier batches stay committed, and the error message says how many rows that is. +* A batch whose save or commit raises a `NeedRetryException` (an MVCC conflict with another writer, a lock timeout) is rolled back and written again from its first row. It is tried up to 10 more times, after a wait that starts at 1 second and doubles up to 10 seconds. After the last try the error is raised unchanged. The columnar insert does this for every type it loads, not only for vertices, when no transaction of yours is open. +* A vertex load inside a transaction you opened is refused with `ArcadeDBError` before anything is written, because the load commits its own batches. A document load joins your transaction. + == Change the commit size [source,python] @@ -85,7 +133,11 @@ The `arcadedb` (native) package does not change the collector. In our measuremen On the `arcadedb-jvm` package, `GraphBatch.create_vertices` and `GraphBatch.new_edges` load these property values through the bulk path (the same tagged JSON text as `insert_many`): a list, a tuple, a dict, a `datetime`, a `date`, a `Decimal` and `bytes` (also `bytearray`). A value is stored as the one-by-one path stores it. A list or tuple of only `int` and `float` keeps the older path, which crosses it as one `double[]`. -A value that the text cannot carry sends work to the older path on its own: a `NaN`, an integer beyond 64 bits, a `datetime` inside a list or a dict, or a dict key that is not a `str`. `create_vertices` falls back for the chunk of at most 100,000 rows that holds the value, and `new_edges` for the whole call. A value of another type (a `set`, a numpy or pandas value, a subclass of the types above) also takes the older path. Nothing is lost on a fallback, and the data is the same. Only the speed changes. +A value that the text cannot carry sends work to the older path on its own. In `create_vertices`, a scalar that the text would change (a `NaN`, an integer beyond 64 bits, a lone surrogate, a subnormal float) sends the whole call to the older path, wherever in the input it sits. So does a property name that is not a `str`. A value inside a list or a dict that the text cannot carry (a `datetime` inside a list, a `NaN` in a list, a dict key that is not a `str`) sends only the chunk of at most 100,000 rows that holds it. `new_edges` falls back for the whole call. A value of another type (a `set`, a numpy or pandas value, a subclass of the types above) also takes the older path. + +Nothing is lost on a fallback and the values are the same, but the stored type of a property that the type does not declare can differ: the older path stores a Python `int` as a `Long`, and the text path as an `Integer`. That is why a bad scalar sends the whole call to the older path. The stored type of a property then does not depend on which chunk held the bad value. + +A property whose value in every row is a `float32` numpy array or a Java `float[]` of one length crosses as one binary matrix for each property and chunk, and is stored as a `float[]`. A declared `ARRAY_OF_FLOATS` property takes the same path for lists and `float64` arrays of numbers too. See xref:how-to/vectors.adoc[Vectors]. The table is one laptop study of the `arcadedb-jvm` package: per-row time of this build against the build before it, 20,000 vertices and 20,000 edges for each shape in chunks of 5,000, one property on every row. Each cell is the median of the pair ratios (below 1 means faster) with its 95 % interval, from 11 to 12 clean ABBA pairs for each figure of `ab.py --noise`, with a clean A/A control. The `arcadedb` (native) package does not change. diff --git a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc index 71854a3..0c76914 100644 --- a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc +++ b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc @@ -249,7 +249,7 @@ A query vector for `vectorNeighbors` can be a numpy array or a list. This sectio `db.query` returns a result set you can iterate. What a row holds, how a Python value is converted and which rows are record objects is on xref:how-to/values-and-records.adoc[Values and records]. Two methods cover the usual needs: * `first()` returns the first row, or `None` when there is none, so `if db.query(...).first():` tests whether anything matched. -* `to_list()` materialises the whole result as a list of dicts. Use it when you need the whole result in memory; iterate when you stop early or the result is large. +* `to_list()` materialises the whole result as a list of dicts, one Python dict per row. Use it for a small result, or when you need the whole result in memory; iterate when you stop early. For a large result, see <>. * `iter_json_batches(batch_size=10_000)` yields the rows as lists of JSON-native dicts, `batch_size` rows per list (the last may be shorter), with the same values as `to_json_list()`. On the native build `batch_size` bounds memory: the result set holds one reply of the cursor (2048 rows) and one batch at a time, and reads the next reply only when the next batch is asked for, so a large result is never held whole. On the JVM build a batch is serialised on the Java side one at a time. `to_list()`, `to_json_list()`, `to_dataframe()`, `to_columns()`, `to_arrow()` and `to_polars()` still hold the whole result, because they return it whole. + One laptop study on the native build, 1,000,000 rows read with `batch_size=10_000`, 10 clean pairs of this change against the previous behaviour; each figure is the ratio of the new value to the old one, so below 1 is lower. Peak RSS: 0.286 (95% interval 0.284 to 0.289). Time to the first batch: 0.010 (95% interval 0.009 to 0.010). Wall time of the whole read: 0.958 (95% interval 0.952 to 0.966). This is one machine and one result shape, not a speed promise. @@ -275,6 +275,55 @@ On the JVM build, `Document.to_dict()`, `Result.to_dict()` and `get_property_nam The control times `json.dumps` of rows already fetched, which uses no query and no JPype call. Its ratios on the four workloads are 0.996, 0.987, 0.993 and 1.000, and the same build measured twice gave ratios whose intervals contain 1. The figures come from one Linux x86_64 laptop with two cores for the process, Python 3.12, the median of 21 timed passes per run, and builds alternated in both orders. They cover the JVM build and these four workloads only, not wider rows, other value types, `to_list()`, or the native build. +[[reading-large-results]] +=== Reading large results + +The row APIs build one Python dict per row: `to_list()`, `to_json_list()`, and a `for` loop over the result set. They suit small results. For a large result, read columns: + +* `to_columns()` returns a dict of column name to a numpy array, or to a Python list for text and for JSON-typed values. +* `to_arrow()` returns a `pyarrow.Table`. +* `to_dataframe()` returns a `pandas.DataFrame`. +* `to_polars()` returns a `polars.DataFrame`, read through `to_arrow()`. + +These four build no dict per row. Both packages read the result as typed column frames and decode them with the same code, so one query gives the same column types and values on `arcadedb` (native) and on `arcadedb-jvm`. On the native package, the first rows of a query (up to 256) are the one exception: they arrive with the query as JSON, because the row APIs read them. A column read puts those rows into the same column rules as the later frames. + +`to_columns()` needs numpy, `to_arrow()` needs numpy and pyarrow, `to_dataframe()` needs pandas, and `to_polars()` needs polars and pyarrow. `to_columns()` and `to_arrow()` return `None` when a library is missing. `to_dataframe(convert_types=False)` on `arcadedb-jvm` builds the frame from `to_list()` instead; on `arcadedb` the argument has no effect. + +`to_columns()`, `to_arrow()` and `to_polars()` take `batch_size`, the most rows in one frame (default 25,000), and `columns`, a list of names to read as a projection would. `to_dataframe()` takes neither and uses 25,000. `batch_size` must be at least 1 (`ValueError` otherwise). A value above 2,147,483,647 (2^31^ - 1) is clamped to 2,147,483,647. On the native package a frame also ends when the values it holds reach about 1 MiB. + +Memory: these calls return the whole result, so it must fit in memory. The frames are decoded as they are read and kept until the end of the result. For a result of several frames, `to_columns()` and `to_dataframe()` then copy them into one array for each column, so the frames and the merged arrays are held together for a moment. `to_arrow()` keeps the frames as the chunks of one column. If the result does not fit, read it in pieces with `iter_json_batches()`. + +==== What the column calls return + +* The type of a column belongs to the whole result, not to where a frame ends. A frame that lacks a property, or holds only nulls for it, takes the type of the frames that have values. `batch_size` never changes a column's type. +* Nulls: in `to_columns()` and `to_dataframe()`, an integer column with a null is `float64` with `NaN` (the pandas rule), a `DATETIME` column with a null holds `NaT`, and text and JSON-typed columns hold `None`. In `to_arrow()` and `to_polars()`, an integer column with a null stays `int64` with a null. +* An integer beyond 2^53^ next to a floating value has no exact double, so the column is not made a `float64` column. `to_columns()` gives a list of Python values, each an `int` or a `float` as stored. `to_arrow()` and `to_polars()` give a string column with the exact text of each value. `to_dataframe()` passes the same list to pandas, which infers `float64` for a list of `int` and `float` values, so the integer is rounded there. Use `to_columns()` or `to_arrow()` when such a value must stay exact. +* A vector column exists only when the property is declared as an array (`ARRAY_OF_FLOATS`). A `LIST` property of numbers is a list column, whatever its values. +* Dates and decimals: `DATETIME` is `datetime64[ms]` in `to_columns()` and `timestamp[ms]` in `to_arrow()`, and `datetime64[us]` in `to_dataframe()`. A `DECIMAL` stays a `Decimal` (an object array in `to_columns()`, a decimal column in Arrow). A `DATE` in `to_dataframe()` is a `datetime.date` in an object column. +* The arrays of `to_columns()` are writable. +* A result with no rows gives `{}` from `to_columns()`. With `columns=[...]`, it gives those columns as empty lists. `to_arrow()` gives an empty table, or with `columns=[...]` those columns empty and of the null type. + +A vector cell has one form for each call: + +[cols="2,3"] +|=== +| Call | Vector column (`float[]`) + +| `to_columns()` +| One `(count, dim)` `float32` matrix. A null vector is a row of `NaN`. + +| `to_dataframe()` +| One `float32` numpy array in each cell, and `None` for a null. + +| `to_arrow()` +| A fixed-size list of `float32`. A null vector is a null. + +| `to_polars()` +| `Array(Float32, n)`. +|=== + +A `double[]` property gives `float64` in each of these. If the vectors of a result do not all have one length, the column is a list of variable length: `list` in Arrow, a list of lists in `to_columns()`. + [[result-set-single-pass]] === A result set is read once @@ -295,6 +344,8 @@ A record id inside a row value (`SELECT @rid AS r`, a `LINK` property, a list of On the native build the record wrappers send their record id to the engine as a parameter, not in the statement text. The engine keeps a parsed statement in a cache keyed by its text (`arcadedb.sqlStatementCache`, 300 entries by default) and leaves a record id written in a `FROM` target as part of that text, so each record would be a cache entry of its own. `get_out_edges()`, `get_in_edges()`, `get_both_edges()`, `new_edge()`, `Edge.get_in()`, `Edge.get_out()`, `save()` and `delete()` send the id as a parameter, so every record shares one text. `db.lookup_by_rid()` keeps the id in the statement text: measured, the parameter path was 2 % slower for a lookup, so it stays as it is. +On the native build, `save()` of a new record and `Vertex.new_edge` read the record they create from the reply of the statement that creates it, so a create is one call into the library. `save()` of a saved record makes three calls: the transaction check, the `SELECT` of the stale-record check, and the `UPDATE`. The records, their values and the errors are the same as before. + `Vertex.new_edge()` works on a vertex object that you read in an earlier transaction, on both builds. The first edge of a vertex rewrites its record, so an object read before that commit is out of date. The native build goes by record id. The JVM build looks up the current source vertex by its record id inside the active transaction and passes the target as its record id, so the engine loads that one too. The vertex object you hold is not refreshed: call `db.lookup_by_rid()` again to see its current properties. [source,python,subs="attributes+"] diff --git a/docs/modules/ROOT/pages/how-to/vectors.adoc b/docs/modules/ROOT/pages/how-to/vectors.adoc index 23df5dc..8b9ea80 100644 --- a/docs/modules/ROOT/pages/how-to/vectors.adoc +++ b/docs/modules/ROOT/pages/how-to/vectors.adoc @@ -114,6 +114,8 @@ print(db.count_type("Chunk")) On the `arcadedb-jvm` package, `create_vertices` sends the values of a property declared `ARRAY_OF_FLOATS` as one binary float32 matrix when every row holds a list, a numpy array or a Java `float[]` or `double[]` of numbers of one length. A property that is not declared takes the same matrix when every row holds a `float32` numpy array or a Java `float[]` (what `to_java_float_array` returns on this package) of one length, and the stored value is a `float[]`, as before. A list or a `float64` array on an undeclared property keeps its doubles and the older path. So `float32` arrays and `to_java_float_array` values take the binary path on both packages, declared or not. +To read many vectors, use `to_columns()`, `to_dataframe()`, `to_arrow()` or `to_polars()`. `to_list()` builds one Python list of floats for each vector. A column is a vector column only when the property is declared `ARRAY_OF_FLOATS`; a `LIST` property of numbers is a list column. `to_columns()` returns a `(count, dim)` `float32` matrix, with a row of `NaN` for a null vector. `to_dataframe()` returns one `float32` numpy array in each cell, and `None` for a null. `to_arrow()` returns a fixed-size list of `float32`, and `to_polars()` an `Array(Float32, n)`. Vectors of different lengths in one result make a variable-length list column (`list` in Arrow). The rules for nulls, `batch_size` and memory are in xref:how-to/databases-and-queries.adoc#reading-large-results[Reading large results]. + == Search === SQL diff --git a/docs/modules/ROOT/pages/installation.adoc b/docs/modules/ROOT/pages/installation.adoc index 0f2e67d..05e55d2 100644 --- a/docs/modules/ROOT/pages/installation.adoc +++ b/docs/modules/ROOT/pages/installation.adoc @@ -160,7 +160,7 @@ Each Python process starts its own JVM, and each JVM sees the whole container li [[jvm-cds]] === Class-data sharing -The bundled JRE ships the JVM's default class-data-sharing archive, dumped with `-XX:+UseCompactObjectHeaders`, the flag that `start_jvm` sets. The JVM finds the archive by itself and maps the JDK's classes from it instead of loading and verifying them at every start. You do nothing to use it. The JVM ignores the archive, and starts as it did before, when your `jvm_args` change a setting that the archive depends on (`-XX:-UseCompactObjectHeaders`, `-Xshare:off`) or when the heap is larger than 32 GB (the default is half of the RAM, so a machine with more than 64 GB of RAM starts without the archive unless you set `heap_size`). The archive holds the JDK's classes only, not the engine's jars. +The bundled JRE ships the JVM's default class-data-sharing archive, dumped with `-XX:+UseCompactObjectHeaders`, the flag that `start_jvm` sets. The JVM finds the archive by itself and maps the JDK's classes from it instead of loading and verifying them at every start. You do nothing to use it. The JVM ignores the archive, and starts as it did before, when your `jvm_args` change a setting that the archive depends on (`-XX:-UseCompactObjectHeaders`, `-Xshare:off`) or when the heap is larger than 32 GB (the default is half of the RAM, so a machine with more than 64 GB of RAM starts without the archive unless you set `heap_size`). The archive holds the JDK's classes only, not the engine's jars. It is a file in the bundled JRE, so each JVM wheel is larger by the size of the archive. The native package has no JVM and no archive. [[jvm-aot]] === Faster start-up with the AOT cache From 36f106216e7bbd76079ae67513d429276ea6c9c2 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Tue, 10 Nov 2026 07:27:48 +0900 Subject: [PATCH 2/7] Docs for post1: fixes after checking each statement against the candidate wheels Read against the code of the post1 candidate (the wheels' source commit 1d91ee8 plus the tests-only commits after it): - insert_many and create_vertices: the one-pass check raises nothing to the caller. A bad row makes json_bulk_dumps raise ValueError or TypeError inside the package, and insert_many and create_vertices catch it and take the row or older path. Reworded the sentence that said the first bad row raises the same type and message. - Columnar fallbacks on the native package: the numpy datetime64 array and the pandas NaT or NA case belong to insert_columns lists. A DataFrame goes through pyarrow.Table.from_pandas, where they are nulls and stay columnar. - Retry of a NeedRetryException: only the synchronous columnar insert retries (DocumentBatcher.writeChunks, BulkOps.writeChunks). parallel=True and the native row path (BulkOps.insertRows, no transaction of its own) do not. - Vector column rule: ColumnBatcher and ColumnFrames decide by the value type (float[] or double[]), not by the property declaration. Reworded in databases-and-queries and vectors. - Class-data archive: classes_coh.jsa is 15,065,088 bytes, 3,869,721 inside the wheel; the native wheel has no JRE. Co-Authored-By: Claude Sonnet 5.5 --- docs/modules/ROOT/pages/how-to/bulk-loads.adoc | 6 +++--- docs/modules/ROOT/pages/how-to/databases-and-queries.adoc | 2 +- docs/modules/ROOT/pages/how-to/vectors.adoc | 2 +- docs/modules/ROOT/pages/installation.adoc | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index 22110a8..c25a527 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -65,7 +65,7 @@ Because the two packages have different defaults, the same program can keep diff On both packages, `insert_many` keeps rows with `datetime`, `date`, `Decimal`, `bytes`, numpy scalar, or pandas `NA` and `NaT` values on the bulk path. On the `arcadedb` (native) package, a row with a value the bulk text cannot carry (NaN or infinity, an integer beyond 64 bits, a date-time or `Decimal` inside a list or dict, or a `datetime.time`) goes in with its own statement, in the order given. -`insert_many` and `create_vertices` check their rows while they write the JSON text, in one pass over the rows, not in a loop before it. The first bad row of a batch raises the same exception type and message as before. The message names the bad value, not a row number. +`insert_many` and `create_vertices` check their rows while they write the JSON text, in one pass over the rows, not in a loop before it. The same rows fail the check as before, and they take the same fallback path as before. The check raises no error to your code. [[columnar-loads]] == Columnar loads @@ -74,7 +74,7 @@ On both packages, `insert_many` keeps rows with `datetime`, `date`, `Decimal`, ` On the `arcadedb` (native) package, a batch whose columns the library's buffers can carry crosses in one call, with no Python dict per row and no JSON text. These columns stay columnar: integers, floats, bools, text, date-times (`datetime`, `pandas.Timestamp`, and Arrow timestamps of any unit), dates, `Decimal`, and `bytes` or `bytearray`. The stored values are the ones the row path stores. -A column the buffers cannot carry makes the batch take the row path, which stores the same values through `insert_many`. These are a column that mixes types, a nested value, a numpy `datetime64` array, an integer beyond 64 bits, a string with a lone surrogate, a non-finite `Decimal`, and a pandas `NaT` or `NA` column. `ARCADEDB_NATIVE_BULK=0` in the environment turns the columnar path off. +A column the buffers cannot carry makes the batch take the row path, which stores the same values through `insert_many`. These are a column that mixes types, a nested value, an integer beyond 64 bits, a string with a lone surrogate, a non-finite `Decimal`, and, in `insert_columns`, a numpy `datetime64` array or a list that holds pandas `NaT` or `NA`. A DataFrame goes through Arrow first, where a `NaT` or `NA` is a null, so it stays columnar. `ARCADEDB_NATIVE_BULK=0` in the environment turns the columnar path off. On the `arcadedb-jvm` package, a column of only `str` and `None` crosses as one UTF-8 buffer with its row offsets, and a mask when a row is null. A column that mixes types, is empty, or holds more than 2 GiB of text crosses value by value. A column with a lone surrogate also crosses value by value, and is refused with the error it always raised, before anything is written. @@ -102,7 +102,7 @@ On both packages, `insert_dataframe` and `insert_arrow` on a vertex type load th * The load runs with the write-ahead log setting of the database. `graph_batch()` has its own `use_wal` argument. * Each batch of `batch_size` rows (10,000 by default) is one transaction. A failed batch is rolled back, earlier batches stay committed, and the error message says how many rows that is. -* A batch whose save or commit raises a `NeedRetryException` (an MVCC conflict with another writer, a lock timeout) is rolled back and written again from its first row. It is tried up to 10 more times, after a wait that starts at 1 second and doubles up to 10 seconds. After the last try the error is raised unchanged. The columnar insert does this for every type it loads, not only for vertices, when no transaction of yours is open. +* A batch whose save or commit raises a `NeedRetryException` (an MVCC conflict with another writer, a lock timeout) is rolled back and written again from its first row. It is tried up to 10 more times, after a wait that starts at 1 second and doubles up to 10 seconds. After the last try the error is raised unchanged. The columnar insert does this for every type it loads, not only for vertices, when no transaction of yours is open. A call with `parallel=True` does not retry, and on the native package neither does a batch that takes the row path. * A vertex load inside a transaction you opened is refused with `ArcadeDBError` before anything is written, because the load commits its own batches. A document load joins your transaction. == Change the commit size diff --git a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc index 0c76914..39af033 100644 --- a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc +++ b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc @@ -298,7 +298,7 @@ Memory: these calls return the whole result, so it must fit in memory. The frame * The type of a column belongs to the whole result, not to where a frame ends. A frame that lacks a property, or holds only nulls for it, takes the type of the frames that have values. `batch_size` never changes a column's type. * Nulls: in `to_columns()` and `to_dataframe()`, an integer column with a null is `float64` with `NaN` (the pandas rule), a `DATETIME` column with a null holds `NaT`, and text and JSON-typed columns hold `None`. In `to_arrow()` and `to_polars()`, an integer column with a null stays `int64` with a null. * An integer beyond 2^53^ next to a floating value has no exact double, so the column is not made a `float64` column. `to_columns()` gives a list of Python values, each an `int` or a `float` as stored. `to_arrow()` and `to_polars()` give a string column with the exact text of each value. `to_dataframe()` passes the same list to pandas, which infers `float64` for a list of `int` and `float` values, so the integer is rounded there. Use `to_columns()` or `to_arrow()` when such a value must stay exact. -* A vector column exists only when the property is declared as an array (`ARRAY_OF_FLOATS`). A `LIST` property of numbers is a list column, whatever its values. +* A vector column is a column whose values are all `float[]`, which is what a property declared `ARRAY_OF_FLOATS` holds. A `LIST` property of numbers is a list column, whatever its values. * Dates and decimals: `DATETIME` is `datetime64[ms]` in `to_columns()` and `timestamp[ms]` in `to_arrow()`, and `datetime64[us]` in `to_dataframe()`. A `DECIMAL` stays a `Decimal` (an object array in `to_columns()`, a decimal column in Arrow). A `DATE` in `to_dataframe()` is a `datetime.date` in an object column. * The arrays of `to_columns()` are writable. * A result with no rows gives `{}` from `to_columns()`. With `columns=[...]`, it gives those columns as empty lists. `to_arrow()` gives an empty table, or with `columns=[...]` those columns empty and of the null type. diff --git a/docs/modules/ROOT/pages/how-to/vectors.adoc b/docs/modules/ROOT/pages/how-to/vectors.adoc index 8b9ea80..2eef4df 100644 --- a/docs/modules/ROOT/pages/how-to/vectors.adoc +++ b/docs/modules/ROOT/pages/how-to/vectors.adoc @@ -114,7 +114,7 @@ print(db.count_type("Chunk")) On the `arcadedb-jvm` package, `create_vertices` sends the values of a property declared `ARRAY_OF_FLOATS` as one binary float32 matrix when every row holds a list, a numpy array or a Java `float[]` or `double[]` of numbers of one length. A property that is not declared takes the same matrix when every row holds a `float32` numpy array or a Java `float[]` (what `to_java_float_array` returns on this package) of one length, and the stored value is a `float[]`, as before. A list or a `float64` array on an undeclared property keeps its doubles and the older path. So `float32` arrays and `to_java_float_array` values take the binary path on both packages, declared or not. -To read many vectors, use `to_columns()`, `to_dataframe()`, `to_arrow()` or `to_polars()`. `to_list()` builds one Python list of floats for each vector. A column is a vector column only when the property is declared `ARRAY_OF_FLOATS`; a `LIST` property of numbers is a list column. `to_columns()` returns a `(count, dim)` `float32` matrix, with a row of `NaN` for a null vector. `to_dataframe()` returns one `float32` numpy array in each cell, and `None` for a null. `to_arrow()` returns a fixed-size list of `float32`, and `to_polars()` an `Array(Float32, n)`. Vectors of different lengths in one result make a variable-length list column (`list` in Arrow). The rules for nulls, `batch_size` and memory are in xref:how-to/databases-and-queries.adoc#reading-large-results[Reading large results]. +To read many vectors, use `to_columns()`, `to_dataframe()`, `to_arrow()` or `to_polars()`. `to_list()` builds one Python list of floats for each vector. A column is a vector column when its values are `float[]`, which is what a property declared `ARRAY_OF_FLOATS` holds; a `LIST` property of numbers is a list column. `to_columns()` returns a `(count, dim)` `float32` matrix, with a row of `NaN` for a null vector. `to_dataframe()` returns one `float32` numpy array in each cell, and `None` for a null. `to_arrow()` returns a fixed-size list of `float32`, and `to_polars()` an `Array(Float32, n)`. Vectors of different lengths in one result make a variable-length list column (`list` in Arrow). The rules for nulls, `batch_size` and memory are in xref:how-to/databases-and-queries.adoc#reading-large-results[Reading large results]. == Search diff --git a/docs/modules/ROOT/pages/installation.adoc b/docs/modules/ROOT/pages/installation.adoc index 05e55d2..3e64546 100644 --- a/docs/modules/ROOT/pages/installation.adoc +++ b/docs/modules/ROOT/pages/installation.adoc @@ -160,7 +160,7 @@ Each Python process starts its own JVM, and each JVM sees the whole container li [[jvm-cds]] === Class-data sharing -The bundled JRE ships the JVM's default class-data-sharing archive, dumped with `-XX:+UseCompactObjectHeaders`, the flag that `start_jvm` sets. The JVM finds the archive by itself and maps the JDK's classes from it instead of loading and verifying them at every start. You do nothing to use it. The JVM ignores the archive, and starts as it did before, when your `jvm_args` change a setting that the archive depends on (`-XX:-UseCompactObjectHeaders`, `-Xshare:off`) or when the heap is larger than 32 GB (the default is half of the RAM, so a machine with more than 64 GB of RAM starts without the archive unless you set `heap_size`). The archive holds the JDK's classes only, not the engine's jars. It is a file in the bundled JRE, so each JVM wheel is larger by the size of the archive. The native package has no JVM and no archive. +The bundled JRE ships the JVM's default class-data-sharing archive, dumped with `-XX:+UseCompactObjectHeaders`, the flag that `start_jvm` sets. The JVM finds the archive by itself and maps the JDK's classes from it instead of loading and verifying them at every start. You do nothing to use it. The JVM ignores the archive, and starts as it did before, when your `jvm_args` change a setting that the archive depends on (`-XX:-UseCompactObjectHeaders`, `-Xshare:off`) or when the heap is larger than 32 GB (the default is half of the RAM, so a machine with more than 64 GB of RAM starts without the archive unless you set `heap_size`). The archive holds the JDK's classes only, not the engine's jars. It is a file in the bundled JRE (about 15 MB, about 3.9 MB inside the wheel), so each JVM wheel is larger by that much. The native package has no JVM and no archive. [[jvm-aot]] === Faster start-up with the AOT cache From cbc2b0c669bae14f5e08ac00c7658201738de577 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Tue, 10 Nov 2026 13:21:18 +0900 Subject: [PATCH 3/7] Docs for post1: a killed create_vertices load on the JVM package can leave part of a call (kill probe, 240 cycles) Co-Authored-By: Claude Sonnet 5.5 --- docs/modules/ROOT/pages/how-to/bulk-loads.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index c25a527..c7f1bef 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -225,6 +225,7 @@ This section is about a `graph_batch()` load that stops before `close()` returns === What is in the database * The vertices and edge records that a returned call committed are there. A load that is killed loses the work after the last commit and nothing before it. +* On the `arcadedb-jvm` package, a killed `create_vertices` load can leave part of a call in the database. `graph_batch()` relaxes the write-ahead log for the load, so a transaction is not atomic after a kill. In a kill probe of 240 cycles (`kill -9` at a random time during a load of 400,000 rows in calls of 50,000) on the released 26.11.1 package, 236 stored a whole number of calls and 4 stored another count, and in those 4 the stored keys were not a prefix of the input: some earlier rows were missing while later rows were present. No stored value was wrong, no row was stored twice, and the database reopened in every cycle. After an interrupted vertex load, compare the stored keys with the input and reload the type. * The edges are not traversable. The engine writes the edge records first and links them to their vertices (the head pointers and the incoming lists) in `close()`. Until `close()` has run, `outE()`, `inE()`, `out()` and `MATCH` find none of them, but `SELECT count(*) FROM E` counts them. In the example: 70,000 edge records, 0 reachable. * `CHECK DATABASE` reports each of these edges as an `ORPHAN RECORD` and counts them in `unreachableEdgeRecords`. It also uses the words "was not connected from", "is not referenced back by" and "missing from that vertex's OUT/IN list" for the same state. Read the statement in the {arcadedb-docs-url}/reference/sql/sql-database-admin#sql-check-database[`CHECK DATABASE` reference]; this page does not repeat it. From bcd2b98f9c9cf850ca081e49af7a3e508ef8a153 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Tue, 10 Nov 2026 19:41:43 +0900 Subject: [PATCH 4/7] Tests: pin the docs statements on to_dataframe exactness (a later frame, an integer column with a null) The docs page 'What the column calls return' now says: to_dataframe() keeps an integer beyond 2**53 exact (an object column), also when the integer and the floats are in different frames of a large result; an integer column with a null is still float64 with NaN in to_columns() and to_dataframe() (rounded), and int64 with a null in to_arrow(). Two tests pin both sentences on the post2 candidate code. Co-Authored-By: Claude Sonnet 5.5 --- tests/test_column_frames.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/tests/test_column_frames.py b/tests/test_column_frames.py index 98de315..1ae0e1a 100644 --- a/tests/test_column_frames.py +++ b/tests/test_column_frames.py @@ -601,6 +601,35 @@ def test_to_dataframe_integers_up_to_2_53_next_to_floats_stay_float64(temp_db): assert df["x"].tolist() == [1.0, 1.5, 2.0**53, -(2.0**53)] +def test_to_dataframe_keeps_the_integer_when_the_floats_are_in_a_later_frame(temp_db): + # docs, "What the column calls return": the integer stays exact also when it and the floats are in different frames of a + # large result. to_dataframe() reads 25,000 rows in one frame, so rows 0 to 24,999 are integers and the last 10 are floats + big = 2**60 + 1 + want = [big + i for i in range(25_000)] + [0.5 + i for i in range(10)] + df = _mix_frame(temp_db, want) + assert df["x"].dtype == object + assert _same_values_and_types(df["x"].tolist(), want) + columns = temp_db.query("sql", "SELECT i, x FROM Mix ORDER BY i").to_columns() + assert isinstance(columns["x"], list) # the same column in to_columns(), not a float64 array + assert _same_values_and_types(columns["x"], want) + + +def test_an_integer_beyond_2_53_in_a_column_with_a_null_is_rounded_but_in_arrow( + temp_db, +): + # docs, "What the column calls return": the limit that stays. An integer column with a null is float64 with NaN in + # to_columns() and to_dataframe() (the pandas rule), which cannot hold 2**60 + 1; to_arrow() keeps int64 with a null + big = 2**60 + 1 + df = _mix_frame(temp_db, [big, None]) + assert df["x"].dtype == np.float64 and np.isnan(df["x"][1]) + assert int(df["x"][0]) == 2**60 # rounded to the nearest double + query = "SELECT i, x FROM Mix ORDER BY i" + columns = temp_db.query("sql", query).to_columns() + assert columns["x"].dtype == np.float64 and int(columns["x"][0]) == 2**60 + arrow = temp_db.query("sql", query).to_arrow()["x"] + assert arrow.type == pa.int64() and arrow.to_pylist() == [big, None] + + def test_to_dataframe_of_an_empty_result_is_an_empty_frame(temp_db): _mix_frame(temp_db, [2**60 + 1, 1.5]) df = temp_db.query("sql", "SELECT i, x FROM Mix WHERE i < 0").to_dataframe() From e6178e79397774da382b4851c7d26d70b1ef36b9 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Tue, 10 Nov 2026 19:41:43 +0900 Subject: [PATCH 5/7] Docs for post2: to_dataframe keeps an integer beyond 2**53 exact (#340), the limits that stay, and the scope of the kill probe - databases-and-queries: to_dataframe() gives an object column of the exact Python values, also across the frames of a large result; to_dataframe(convert_types=False) on the JVM wheel and the no-bridge fallback still round; an integer column with a null is float64 with NaN in to_columns() and to_dataframe() (rounded), exact in to_arrow() and to_polars(). Checked on the post1 wheel (float64) against the post2 python files on the same wheel (object). - bulk-loads: the kill probe ran on 26.11.1 and was not repeated on later packages. Co-Authored-By: Claude Sonnet 5.5 --- docs/modules/ROOT/pages/how-to/bulk-loads.adoc | 2 +- docs/modules/ROOT/pages/how-to/databases-and-queries.adoc | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index c7f1bef..67ea76b 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -225,7 +225,7 @@ This section is about a `graph_batch()` load that stops before `close()` returns === What is in the database * The vertices and edge records that a returned call committed are there. A load that is killed loses the work after the last commit and nothing before it. -* On the `arcadedb-jvm` package, a killed `create_vertices` load can leave part of a call in the database. `graph_batch()` relaxes the write-ahead log for the load, so a transaction is not atomic after a kill. In a kill probe of 240 cycles (`kill -9` at a random time during a load of 400,000 rows in calls of 50,000) on the released 26.11.1 package, 236 stored a whole number of calls and 4 stored another count, and in those 4 the stored keys were not a prefix of the input: some earlier rows were missing while later rows were present. No stored value was wrong, no row was stored twice, and the database reopened in every cycle. After an interrupted vertex load, compare the stored keys with the input and reload the type. +* On the `arcadedb-jvm` package, a killed `create_vertices` load can leave part of a call in the database. `graph_batch()` relaxes the write-ahead log for the load, so a transaction is not atomic after a kill. In a kill probe of 240 cycles (`kill -9` at a random time during a load of 400,000 rows in calls of 50,000) on the released 26.11.1 package, 236 stored a whole number of calls and 4 stored another count, and in those 4 the stored keys were not a prefix of the input: some earlier rows were missing while later rows were present. No stored value was wrong, no row was stored twice, and the database reopened in every cycle. After an interrupted vertex load, compare the stored keys with the input and reload the type. The probe ran on 26.11.1 and was not repeated on later packages. They send more kinds of value to the same engine call (`GraphBatch.createVertices`). * The edges are not traversable. The engine writes the edge records first and links them to their vertices (the head pointers and the incoming lists) in `close()`. Until `close()` has run, `outE()`, `inE()`, `out()` and `MATCH` find none of them, but `SELECT count(*) FROM E` counts them. In the example: 70,000 edge records, 0 reachable. * `CHECK DATABASE` reports each of these edges as an `ORPHAN RECORD` and counts them in `unreachableEdgeRecords`. It also uses the words "was not connected from", "is not referenced back by" and "missing from that vertex's OUT/IN list" for the same state. Read the statement in the {arcadedb-docs-url}/reference/sql/sql-database-admin#sql-check-database[`CHECK DATABASE` reference]; this page does not repeat it. diff --git a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc index 39af033..03ef5de 100644 --- a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc +++ b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc @@ -296,8 +296,8 @@ Memory: these calls return the whole result, so it must fit in memory. The frame ==== What the column calls return * The type of a column belongs to the whole result, not to where a frame ends. A frame that lacks a property, or holds only nulls for it, takes the type of the frames that have values. `batch_size` never changes a column's type. -* Nulls: in `to_columns()` and `to_dataframe()`, an integer column with a null is `float64` with `NaN` (the pandas rule), a `DATETIME` column with a null holds `NaT`, and text and JSON-typed columns hold `None`. In `to_arrow()` and `to_polars()`, an integer column with a null stays `int64` with a null. -* An integer beyond 2^53^ next to a floating value has no exact double, so the column is not made a `float64` column. `to_columns()` gives a list of Python values, each an `int` or a `float` as stored. `to_arrow()` and `to_polars()` give a string column with the exact text of each value. `to_dataframe()` passes the same list to pandas, which infers `float64` for a list of `int` and `float` values, so the integer is rounded there. Use `to_columns()` or `to_arrow()` when such a value must stay exact. +* Nulls: in `to_columns()` and `to_dataframe()`, an integer column with a null is `float64` with `NaN` (the pandas rule), a `DATETIME` column with a null holds `NaT`, and text and JSON-typed columns hold `None`. In `to_arrow()` and `to_polars()`, an integer column with a null stays `int64` with a null. A `float64` column cannot hold every integer beyond 2^53^, so `to_columns()` and `to_dataframe()` round such an integer in a column that has a null. `to_arrow()` and `to_polars()` keep it exact. +* An integer beyond 2^53^ next to a floating value has no exact double, so the column is not made a `float64` column. `to_columns()` gives a list of Python values, each an `int` or a `float` as stored. `to_arrow()` and `to_polars()` give a string column with the exact text of each value. `to_dataframe()` gives an `object` column of the same Python values, so the integer stays exact. This holds also when the integer and the floats are in different frames of a large result. Two reads still round the integer, because they build a `float64` column: `to_dataframe(convert_types=False)` on `arcadedb-jvm`, and the fallback of `to_dataframe()` that reads through `to_list()` when the Java bridge classes are missing. Use `to_columns()` or `to_arrow()` when such a value must stay exact on those two reads. * A vector column is a column whose values are all `float[]`, which is what a property declared `ARRAY_OF_FLOATS` holds. A `LIST` property of numbers is a list column, whatever its values. * Dates and decimals: `DATETIME` is `datetime64[ms]` in `to_columns()` and `timestamp[ms]` in `to_arrow()`, and `datetime64[us]` in `to_dataframe()`. A `DECIMAL` stays a `Decimal` (an object array in `to_columns()`, a decimal column in Arrow). A `DATE` in `to_dataframe()` is a `datetime.date` in an object column. * The arrays of `to_columns()` are writable. From 6d957d2e59bf0d18c0ba8ffae930aa50032f882d Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Tue, 10 Nov 2026 19:42:43 +0900 Subject: [PATCH 6/7] Tests: black layout of the two new to_dataframe tests (the repo's pre-commit black 26.10.1) Co-Authored-By: Claude Sonnet 5.5 --- tests/test_column_frames.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/test_column_frames.py b/tests/test_column_frames.py index 1ae0e1a..c5a6d13 100644 --- a/tests/test_column_frames.py +++ b/tests/test_column_frames.py @@ -609,8 +609,9 @@ def test_to_dataframe_keeps_the_integer_when_the_floats_are_in_a_later_frame(tem df = _mix_frame(temp_db, want) assert df["x"].dtype == object assert _same_values_and_types(df["x"].tolist(), want) + # the same column in to_columns() is a list of exact values, not a float64 array columns = temp_db.query("sql", "SELECT i, x FROM Mix ORDER BY i").to_columns() - assert isinstance(columns["x"], list) # the same column in to_columns(), not a float64 array + assert isinstance(columns["x"], list) assert _same_values_and_types(columns["x"], want) From af0e481e0aa7ed54df16e70d5c596cb494c39c03 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Tue, 10 Nov 2026 21:38:15 +0900 Subject: [PATCH 7/7] Docs: bulk-loads kill probe wording matches the result files The first bullet said a killed load loses nothing before the last commit, the next one says a killed create_vertices load on the JVM package can miss earlier rows: the first bullet now names that exception. The probe was also run on the post1 candidate build (2 of 240 cycles stored another count, both prefixes; released 26.11.1: 4 of 240, none a prefix; pooled by script from repros/ingest-crash-20261110 results and results-followup), so the sentence that it was not repeated on later packages was wrong. It has not been run on post2. Co-Authored-By: Claude Sonnet 5.5 --- docs/modules/ROOT/pages/how-to/bulk-loads.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index 67ea76b..0144526 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -224,8 +224,8 @@ This section is about a `graph_batch()` load that stops before `close()` returns === What is in the database -* The vertices and edge records that a returned call committed are there. A load that is killed loses the work after the last commit and nothing before it. -* On the `arcadedb-jvm` package, a killed `create_vertices` load can leave part of a call in the database. `graph_batch()` relaxes the write-ahead log for the load, so a transaction is not atomic after a kill. In a kill probe of 240 cycles (`kill -9` at a random time during a load of 400,000 rows in calls of 50,000) on the released 26.11.1 package, 236 stored a whole number of calls and 4 stored another count, and in those 4 the stored keys were not a prefix of the input: some earlier rows were missing while later rows were present. No stored value was wrong, no row was stored twice, and the database reopened in every cycle. After an interrupted vertex load, compare the stored keys with the input and reload the type. The probe ran on 26.11.1 and was not repeated on later packages. They send more kinds of value to the same engine call (`GraphBatch.createVertices`). +* The vertices and edge records that a returned call committed are there. A load that is killed loses the work after the last commit and nothing before it, with one exception: `create_vertices` on the `arcadedb-jvm` package (next point). +* On the `arcadedb-jvm` package, a killed `create_vertices` load can leave part of a call in the database. `graph_batch()` relaxes the write-ahead log for the load, so a transaction is not atomic after a kill. In a kill probe of 240 cycles (`kill -9` at a random time during a load of 400,000 rows in calls of 50,000) on the released 26.11.1 package, 236 stored a whole number of calls and 4 stored another count, and in those 4 the stored keys were not a prefix of the input: some earlier rows were missing while later rows were present. No stored value was wrong, no row was stored twice, and the database reopened in every cycle. After an interrupted vertex load, compare the stored keys with the input and reload the type. The same probe on the `26.11.1.post1` candidate build stored another count in 2 of 240 cycles, and in both the stored keys were a prefix of the input. It has not been run on `26.11.1.post2`. * The edges are not traversable. The engine writes the edge records first and links them to their vertices (the head pointers and the incoming lists) in `close()`. Until `close()` has run, `outE()`, `inE()`, `out()` and `MATCH` find none of them, but `SELECT count(*) FROM E` counts them. In the example: 70,000 edge records, 0 reachable. * `CHECK DATABASE` reports each of these edges as an `ORPHAN RECORD` and counts them in `unreachableEdgeRecords`. It also uses the words "was not connected from", "is not referenced back by" and "missing from that vertex's OUT/IN list" for the same state. Read the statement in the {arcadedb-docs-url}/reference/sql/sql-database-admin#sql-check-database[`CHECK DATABASE` reference]; this page does not repeat it.