diff --git a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc index 87fa82c..f2f2503 100644 --- a/docs/modules/ROOT/pages/how-to/bulk-loads.adoc +++ b/docs/modules/ROOT/pages/how-to/bulk-loads.adoc @@ -78,6 +78,25 @@ A column the buffers cannot carry makes the batch take the row path, which store On the `arcadedb-jvm` package, a column of only `str` and `None` crosses as one UTF-8 buffer with its row offsets, and a mask when a row is null. A column that mixes types, is empty, or holds more than 2 GiB of text crosses value by value. A column with a lone surrogate also crosses value by value, and is refused with the error it always raised, before anything is written. +On the `arcadedb-jvm` package, from 26.11.1.post3, a timestamp or date column of `insert_arrow` and `insert_dataframe` crosses as one `long[]` of epoch ticks for each batch, with one `boolean[]` that marks the null rows when the batch has a null. Java builds each `LocalDateTime` or `LocalDate` when it fills the document of that row. Before 26.11.1.post3 the bindings made one Java object for each cell, with a JPype call of its own. The stored values are the same: a timestamp keeps the precision of its Arrow unit (a nanosecond value is not cut to microseconds), a timestamp without a time zone is stored as the wall clock it shows, a timestamp with a time zone is stored as UTC, and a `date64` value is divided into days toward negative infinity. A list of Python `datetime` or `date` values that you pass to `insert_columns` yourself still converts value by value. The `arcadedb` (native) package does not change. + +The table is one laptop study of the `arcadedb-jvm` package: the time for each row of the 26.11.1.post3 wheel against the 26.11.1.post2 wheel, with 3 calls of 100,000 rows that carry a `DATETIME` and a `DATE` property next to four narrow ones. The ratio is the median of the pair ratios (below 1 means less time) with its 95% interval. The setup was Python 3.12, one fresh process for each run, the two builds alternated in both orders (ABBA), the first round dropped, 20 clean pairs for each figure, and an A/A control (the post2 wheel twice) whose median ratio was between 0.97 and 1.03 in every row. The figures come from one Linux x86_64 laptop with two cores for the process. The post3 wheels were the ones that CI built from the tagged commit. The wheels on PyPI were checked for the same engine as 26.11.1 and were not timed again. + +[cols="5,3,1"] +|=== +| What was timed | Ratio, post3 over post2 (95% interval) | Clean pairs + +| `insert_arrow`, an Arrow table with `timestamp[ms]` and `date32` columns +| 0.295 [0.283, 0.311] +| 20 + +| `insert_dataframe`, a pandas frame with `datetime64[ms]` and `datetime.date` columns +| 0.283 [0.278, 0.290] +| 20 +|=== + +`insert_columns` with Python lists of `datetime` and `date` values, which this change does not touch, gave 1.009 [0.983, 1.050] in the same study. The interval contains 1, so there is no claim in either direction. The study did not time memory. It read the peak resident size of the process for information only, and it was higher with the new path: 1.066 [1.014, 1.194] for `insert_arrow` and 1.202 [1.059, 1.299] for `insert_dataframe` (median of the pair ratios, 95% interval). The reason has not been examined, so the page makes no claim about memory. + [[parallel-inserts]] == Parallel and async inserts diff --git a/docs/modules/ROOT/pages/how-to/concurrency.adoc b/docs/modules/ROOT/pages/how-to/concurrency.adoc index 2508f46..c567b71 100644 --- a/docs/modules/ROOT/pages/how-to/concurrency.adoc +++ b/docs/modules/ROOT/pages/how-to/concurrency.adoc @@ -5,7 +5,7 @@ This page covers what is specific to the Python bindings: which calls work from == How to read the numbers -Every number on this page, except the ratios in <>, comes from probe scripts that were run on 2026-10-24 on one Linux x86_64 machine. The setup of each probe is next to its numbers. The builds in the table below are older than the current release, 26.11.1.post2, and the probes were not run again on it, so read each number as a measurement of the build that produced it. The scripts and their raw logs are not part of this repository yet. +Every number on this page, except the ratios in <>, comes from probe scripts that were run on 2026-10-24 on one Linux x86_64 machine. The setup of each probe is next to its numbers. The builds in the table below are older than the current release, 26.11.1.post3, and the probes were not run again on it, so read each number as a measurement of the build that produced it. The scripts and their raw logs are not part of this repository yet. [cols="1,3,2"] |=== diff --git a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc index 03ef5de..4c3215d 100644 --- a/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc +++ b/docs/modules/ROOT/pages/how-to/databases-and-queries.adoc @@ -275,6 +275,29 @@ On the JVM build, `Document.to_dict()`, `Result.to_dict()` and `get_property_nam The control times `json.dumps` of rows already fetched, which uses no query and no JPype call. Its ratios on the four workloads are 0.996, 0.987, 0.993 and 1.000, and the same build measured twice gave ratios whose intervals contain 1. The figures come from one Linux x86_64 laptop with two cores for the process, Python 3.12, the median of 21 timed passes per run, and builds alternated in both orders. They cover the JVM build and these four workloads only, not wider rows, other value types, `to_list()`, or the native build. +On the JVM build, `to_list()` and `iter_dicts()` read the rows from the JVM as JSON text, in batches, and the C `json` module parses each batch. A value that JSON cannot carry exactly (a `DECIMAL`, a date or date-time, a set, a record id, an embedded document, a non-finite float) is tagged in the text, and a hook restores it after the parse. From 26.11.1.post3 a batch whose text holds no tag is parsed without the hook, and the bindings do not read the side array that the hook needs. The rows and their Python types are the same. A batch that holds a tagged value takes the hook as before; the study checked that its values are the same and claims no speed for it. + +The table gives the time for each row of `iter_dicts()` over a table of `LONG`, `DOUBLE` and `STRING` properties, so that no value is tagged, for the 26.11.1.post3 wheel against the 26.11.1.post2 wheel. The query was issued before the timer started. The ratio is the median of the pair ratios (below 1 means less time) with its 95% interval. The setup was Python 3.12, one fresh process for each run, the two builds alternated in both orders (ABBA), the first round dropped, 20 clean pairs for each figure, and an A/A control (the post2 wheel twice) whose median ratio was between 0.97 and 1.03 in every row. The figures come from one Linux x86_64 laptop with two cores for the process. The post3 wheels were the ones that CI built from the tagged commit. The wheels on PyPI were checked for the same engine as 26.11.1 and were not timed again. + +[cols="5,3,1"] +|=== +| What was timed | Ratio, post3 over post2 (95% interval) | Clean pairs + +| `iter_dicts()` over a result of 1 row +| 0.749 [0.735, 0.777] +| 20 + +| `iter_dicts()` over a result of 100 rows +| 0.763 [0.733, 0.782] +| 20 + +| `iter_dicts()` over a result of 100,000 rows +| 0.755 [0.744, 0.770] +| 20 +|=== + +`to_list()` goes through the same parse, but its figures at 1 and 100 rows were too noisy to read at the round cap of the study, and the one at 100,000 rows did not resolve, so this page gives no figure for `to_list()`. The figures cover the JVM build and untagged rows only, not the native build. The study claims no gain in memory. + [[reading-large-results]] === Reading large results @@ -344,7 +367,7 @@ A record id inside a row value (`SELECT @rid AS r`, a `LINK` property, a list of On the native build the record wrappers send their record id to the engine as a parameter, not in the statement text. The engine keeps a parsed statement in a cache keyed by its text (`arcadedb.sqlStatementCache`, 300 entries by default) and leaves a record id written in a `FROM` target as part of that text, so each record would be a cache entry of its own. `get_out_edges()`, `get_in_edges()`, `get_both_edges()`, `new_edge()`, `Edge.get_in()`, `Edge.get_out()`, `save()` and `delete()` send the id as a parameter, so every record shares one text. `db.lookup_by_rid()` keeps the id in the statement text: measured, the parameter path was 2 % slower for a lookup, so it stays as it is. -On the native build, `save()` of a new record and `Vertex.new_edge` read the record they create from the reply of the statement that creates it, so a create is one call into the library. `save()` of a saved record makes three calls: the transaction check, the `SELECT` of the stale-record check, and the `UPDATE`. The records, their values and the errors are the same as before. +On the native build, `save()` of a new record and `Vertex.new_edge` read the record they create from the reply of the statement that creates it, so a create is one call into the library. `save()` of a saved record makes three calls: the transaction check, the `SELECT` of the stale-record check, and the `UPDATE`. With auto-transaction on (see xref:how-to/transactions.adoc[Transactions]) the transaction check cannot refuse, because the engine opens a transaction for a bare write, so from 26.11.1.post3 the check is left out and `save()` makes two calls. With auto-transaction off, the default, the check is still made, and a `save()` outside a transaction raises before it reads or writes anything. The records, their values and the errors are the same as before. `Vertex.new_edge()` works on a vertex object that you read in an earlier transaction, on both builds. The first edge of a vertex rewrites its record, so an object read before that commit is out of date. The native build goes by record id. The JVM build looks up the current source vertex by its record id inside the active transaction and passes the target as its record id, so the engine loads that one too. The vertex object you hold is not refreshed: call `db.lookup_by_rid()` again to see its current properties. diff --git a/docs/modules/ROOT/pages/how-to/dataframes-and-columns.adoc b/docs/modules/ROOT/pages/how-to/dataframes-and-columns.adoc index 43ba7f4..9cf0d11 100644 --- a/docs/modules/ROOT/pages/how-to/dataframes-and-columns.adoc +++ b/docs/modules/ROOT/pages/how-to/dataframes-and-columns.adoc @@ -337,7 +337,7 @@ A missing library does not fail the same way everywhere. `to_columns()` returns == Differences between the two builds -The calls above, the types and the results are the same on `arcadedb` and `arcadedb-jvm`, and so is how `to_columns()` and `to_arrow()` get the rows: both read the result from the engine as packed columns, in requests of at most `batch_size` rows, so `batch_size` sets how many rows cross at a time. One thing differs, the message of a missing pandas: the JVM package adds the `uv add` hint to the `ImportError` of `to_dataframe()`, and the native package raises the plain `ImportError`. +The calls above, the types and the results are the same on `arcadedb` and `arcadedb-jvm`, and so is how `to_columns()` and `to_arrow()` get the rows: both read the result from the engine as packed columns, in requests of at most `batch_size` rows, so `batch_size` sets how many rows cross at a time. One thing differs, the message of a missing pandas: the JVM package adds the `uv add` hint to the `ImportError` of `to_dataframe()`, and the native package raises the plain `ImportError`. Another difference is how `insert_arrow` and `insert_dataframe` pass a timestamp or date column to the engine, which is described on xref:how-to/bulk-loads.adoc#columnar-loads[Bulk loads and commit size]. The stored values are the same on both packages. == Related pages diff --git a/docs/modules/ROOT/pages/how-to/transactions.adoc b/docs/modules/ROOT/pages/how-to/transactions.adoc index e780d0a..2398097 100644 --- a/docs/modules/ROOT/pages/how-to/transactions.adoc +++ b/docs/modules/ROOT/pages/how-to/transactions.adoc @@ -1,5 +1,5 @@ = Transactions -:description: Commit and roll back with the context manager or with begin, commit and rollback, what an exception, a failed statement and a second begin do, what other threads see before a commit, auto-transaction, run_in_transaction, and the GraphBatch rule. +:description: Commit and roll back with the context manager or with begin, commit and rollback, what an exception, a failed statement and a second begin do, what other threads see before a commit, auto-transaction, what a transaction costs on the native build, run_in_transaction, and the GraphBatch rule. // Sources: src/arcadedb/core.py (begin, commit, rollback, transaction, run_in_transaction, set_auto_transaction), transactions.py and graph_batch.py. Every statement below is pinned by tests/test_transactions_behavior.py. @@ -123,6 +123,31 @@ db.set_auto_transaction(False) True False ---- +== What a transaction costs on the native build + +On the `arcadedb` (native) package, `begin()`, `commit()`, `rollback()`, the block `with db.transaction():` and `db.is_transaction_active()` each make a call into the library, and the library answers with the state of the transaction. From 26.11.1.post3 it answers with that state as one number. Before, it answered with a JSON text that the bindings copied, freed and decoded. What a transaction does, and what an error does to it, are the same as before. The `arcadedb-jvm` package does not change. + +The table is one laptop study of the `arcadedb` (native) package: the time for each operation of the 26.11.1.post3 wheel against the 26.11.1.post2 wheel, with 20,000 operations in each run. The ratio is the median of the pair ratios (below 1 means less time) with its 95% interval. Auto-transaction was off, the default, in all three rows. The setup was Python 3.12, one fresh process for each run, the two builds alternated in both orders (ABBA), the first round dropped, 20 clean pairs for each figure, and an A/A control (the post2 wheel twice) whose median ratio was between 0.97 and 1.03 in every row. The figures come from one Linux x86_64 laptop with two cores for the process. The post3 wheels were the ones that CI built from the tagged commit. The wheels on PyPI were checked for the same engine as 26.11.1 and were not timed again. + +[cols="5,3,1"] +|=== +| What was timed | Ratio, post3 over post2 (95% interval) | Clean pairs + +| `with db.transaction(): pass`, one begin and one commit with an empty body +| 0.370 [0.368, 0.374] +| 20 + +| `with db.transaction():` around one `INSERT` through `db.command` +| 0.875 [0.873, 0.878] +| 20 + +| `set()` and `save()` of one saved vertex, in transactions of 1,000 records +| 0.932 [0.928, 0.935] +| 20 +|=== + +In the third row each `save()` asks once for the transaction state, because auto-transaction was off. The study did not time `save()` with auto-transaction on, where it no longer asks (see xref:how-to/databases-and-queries.adoc#record-ids-in-rows[Record ids in rows]), so this page gives no speed for that case. The study claims no gain in memory. + == What a transaction sees A transaction sees its own rows at once. Another thread sees them only after the commit. This block starts a thread for each count: diff --git a/docs/modules/ROOT/pages/quickstart.adoc b/docs/modules/ROOT/pages/quickstart.adoc index f82bcfb..a5e32c0 100644 --- a/docs/modules/ROOT/pages/quickstart.adoc +++ b/docs/modules/ROOT/pages/quickstart.adoc @@ -20,7 +20,7 @@ This page builds a small database from scratch. You need one of the two packages See xref:which-package.adoc[Which package?] if you are not sure. If you are not sure, take `{native-pkg-name}`. -The scripts on this page ran on Python 3.12 against both packages, version 26.11.1.post2, and both printed the output shown. The code is the same for both packages. +The scripts on this page ran on Python 3.12 against both packages, version 26.11.1.post3, and both printed the output shown. The code is the same for both packages. == Embedded mode