Skip to content

Commit 12a366a

Browse files
tae898claude
andcommitted
Docs: how to pass time-series columns to the native append_samples, with the measured gain
The address path of append_samples (numeric NumPy columns and an integer timestamp array cross by address, not as .tolist() and JSON text) was not described anywhere. Add a section to the bulk-loads how-to (which dtypes cross, what keeps the text path, the opt-out, the shape the figure was measured on) and a docstring on the native append_samples. The gating figure (whole ingest of 2,000,000 points in chunks of 100,000, new build over main) and its 95% interval are generated from the study's score file, with its A/A and control intervals checked by the same script, not typed. Docs and docstring only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 400a01d commit 12a366a

2 files changed

Lines changed: 34 additions & 1 deletion

File tree

‎docs/modules/ROOT/pages/how-to/bulk-loads.adoc‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
= Bulk loads and commit size
2-
:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, how insert_many treats Python's cyclic garbage collector on arcadedb-jvm, and a known limitation: a graph batch does not reuse the space of deleted edges.
2+
:description: The default commit size of insert_many, graph_batch and create_vertices on the arcadedb (native) and arcadedb-jvm packages, what it means for a failed load, what to do after an interrupted graph batch, how to change it, how insert_many treats Python's cyclic garbage collector on arcadedb-jvm, how to pass time-series columns to append_samples on arcadedb (native), and a known limitation: a graph batch does not reuse the space of deleted edges.
33

44
`insert_many`, `graph_batch` and `GraphBatch.create_vertices` load data in transactions. The commit size is the number of rows in one transaction. It decides how much work the engine holds in memory before it commits, and how many rows stay in the database when a load fails.
55

@@ -77,6 +77,22 @@ A generator or any other lazy input is not paused. Your own code runs between th
7777

7878
The `arcadedb` (native) package does not change the collector. In our measurements, no collection ran inside a native `insert_many` call, so there is nothing to pause.
7979

80+
== Time-series samples with `append_samples` (`arcadedb`, native)
81+
82+
`db.async_executor().append_samples(type_name, timestamps, *columns)` loads a columnar batch into a TIMESERIES type: the timestamps, then one column for each tag and field in the order of the type's declaration. Call `wait_completion()` before you read the samples back. How to declare the type is in the main ArcadeDB docs: see {arcadedb-docs-url}/concepts/timeseries[Time series].
83+
84+
On the `arcadedb` (native) package, a NumPy column crosses to the engine by its address, with no JSON text, when it is one of these:
85+
86+
* The timestamps: a one-dimensional, non-empty integer or bool array (`int64` is the plain case), or a `datetime64[ns]` array without `NaT`.
87+
* A numeric column: a one-dimensional, non-empty array of any integer or bool kind, or of a float kind up to `float64` (`float32` widens to `float64` without a change of value). A `uint64` array, as timestamps or as a column, crosses only if its largest value fits in `int64`.
88+
* Each column must have as many values as there are timestamps.
89+
90+
Everything else keeps the text path, which stores the same values and only takes longer: a list or a tuple, a subclass of `ndarray`, a two-dimensional or empty array, a float array as the timestamps, a `datetime64` array of another unit or with `NaT`, a `long double` array, and a string or tag column. A call can mix both: the numeric columns cross by address and the tag column goes as text. To get the fast path, build the timestamps and the numeric columns as NumPy arrays with these dtypes and pass the arrays as they are, without `.tolist()`. An array that is not already contiguous `int64` or `float64` is converted first, which copies it in NumPy.
91+
92+
The measurement is a laptop A/B on the `arcadedb` (native) wheel, one fresh process per run, Python 3.12 and NumPy 2.5.4. The workload loaded 2,000,000 points of 100 hosts in chunks of 100,000: an `int64` timestamp array, one list of host names (the tag), and three `float64` arrays, with one `wait_completion()` at the end. The figure is the whole ingest time in seconds, including that wait, as the ratio of the build with this change to the build without it (below 1 means faster): 0.370 (95% interval 0.362 to 0.390, median of 11 clean paired runs). A document insert in the same runs, which does not use this path, took 1.020 (95% interval 1.016 to 1.022) of the time on the build with this change. The figure holds for this shape: your gain depends on how many numeric columns a call carries and how large the chunks are.
93+
94+
`ARCADEDB_NATIVE_TS_BIN=0` in the environment turns the address path off and returns to the text path. A native library that is older than this change falls back to the text path by itself.
95+
8096
== Graph batches next to deletes (fixed in ArcadeDB 26.11.1)
8197

8298
NOTE: *Fixed in ArcadeDB 26.11.1.* Before 26.11.1, a graph batch did not reuse the space of deleted edges (https://github.com/ArcadeData/arcadedb/issues/10746[ArcadeData/arcadedb#10746]). The engine wrote the edge records of each `graph_batch()` flush onto new pages and never looked at the free space that deletes left on older pages, so the edge bucket gained one 64 KiB page per flush while the live data stayed flat. From 26.11.1 (https://github.com/ArcadeData/arcadedb/pull/10836[ArcadeData/arcadedb#10836]) each edge goes where an ordinary insert would put it while an existing page has room, and only the rest of a flush goes onto new pages, so a large import stays as fast as before. The 26.11.1 packages ship this fix. The `26.11.1.devN` pre-releases embed engine 26.10.2 and still show the growth described below.

‎src/arcadedb/_native/extras.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -614,6 +614,23 @@ def command(
614614
def append_samples(
615615
self, type_name, timestamps, *column_values, primitive: bool = False
616616
):
617+
"""Columnar bulk append into a TIMESERIES type, the same call as on the JVM build.
618+
619+
Columns follow the type's declaration order (tags first, then fields); call wait_completion() before reading the samples back.
620+
621+
What to pass to get the fast path: NumPy arrays, not lists. A one-dimensional, non-empty int, uint (uint64 only inside the int64
622+
range) or bool array, or a datetime64[ns] array without NaT, as the timestamps, and an int, uint, bool or float array (float32 and
623+
float64; not long double) as a numeric column, each of the length of the timestamps, cross by address, with no JSON text (an array
624+
that is not already contiguous int64 or float64 is converted first, which copies it). A list, a tuple, an ndarray subclass, a 2-D
625+
or empty array, a float array as timestamps, another datetime64 unit, a string or tag column and a column of another length keep
626+
the text path, which stores the same values. One call can mix both.
627+
ARCADEDB_NATIVE_TS_BIN=0 turns the address path off.
628+
629+
Measured on this package (laptop A/B, 2,000,000 points of 100 hosts in chunks of 100,000: an int64 timestamp array, a list of host
630+
names and three float64 arrays; whole ingest in seconds including wait_completion(), new build over old, below 1 is faster):
631+
0.370 (95% interval 0.362 to 0.390, median of 11 clean paired runs). The figure is for that shape.
632+
"""
633+
617634
def plain(values):
618635
kind = getattr(getattr(values, "dtype", None), "kind", None)
619636
if (

0 commit comments

Comments
 (0)