Skip to content

JVM insert_arrow, insert_dataframe, insert_columns: a timestamp or date column crosses as one long[] of epoch ticks, not one JPype call per cell (refs #315) - #383

Closed
tae898 wants to merge 2 commits into
ArcadeData:mainfrom
humemai:perf/315-jvm-temporal-ingest-bulk
Closed

tae898 wants to merge 2 commits into
ArcadeData:mainfrom
humemai:perf/315-jvm-temporal-ingest-bulk

Conversation

@tae898

@tae898 tae898 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

On the JVM wheel, insert_arrow, insert_dataframe and insert_columns (called by the first two) made one LocalDateTime.ofEpochSecond or LocalDate.ofEpochDay JPype call per cell of every timestamp or date column, and then _column_to_java converted every element of that list again. A million-row timestamp column was two million JPype calls. The native wheel moved the same columns to one crossing in #333 (#316).

Now a timestamp or date column of a batch is the integer epoch values in one long[] (plus one boolean[] when a row is null), wrapped in a new bridge class TemporalColumn, built with one call per column and batch. DocumentBatcher builds each LocalDateTime or LocalDate as it fills the document of that row, the way StringColumn does for text. The old per-cell path is gone. The wheel carries its own bridge jar, so there is no fallback for an older one.

The Java side, in src/java/com/arcadedb/python/TemporalColumn.java:

public static TemporalColumn dateTimes(long[] ticks, long ticksPerSecond, boolean[] nulls)
public static TemporalColumn dates(long[] ticks, long ticksPerDay, boolean[] nulls)
public int rows()
public Object get(int i)   // LocalDateTime or LocalDate, or null for a null row

DocumentBatcher.checkColumns and fillRow take it beside long[], double[], boolean[], Object[], FloatVectors.Matrix and StringColumn. In Python, dataframe._temporal_column replaces _temporal_to_java, and core._column_to_java hands the column over as it is.

How the stored value is kept

  • A date-time is a UTC wall clock. A timestamp with a zone is already UTC epoch time and a timestamp without one is the clock it shows, so both read the integer, in the unit of the column (ticksPerSecond 1, 1,000, 1,000,000 or 1,000,000,000).
  • Precision: whole seconds are floorDiv(tick, ticksPerSecond) and the rest is floorMod(...) * (1e9 / ticksPerSecond) nanoseconds, so a nanosecond tick keeps its nanoseconds and a tick before the epoch is the last tick of 1969. This is the np.divmod arithmetic of the old code, now in Java.
  • Dates: date32 is days (ticksPerDay 1) and date64 is milliseconds (ticksPerDay 86,400,000), floored as the old numpy // was.
  • Nulls: a null row holds 0 in the array and a flag in the boolean[], which is only sent when there is a null. get(i) returns null for it.
  • A value past the years of java.time still raises before a row is written, as it did in Python: the factory builds the value of the smallest and of the largest tick of the non-null rows, and the rest lies between them.
  • A list of Python datetime objects passed to insert_columns is not touched: it still crosses value by value. A numpy datetime64 column is still refused there.

Related issues

Part of #315.

Performance changes

Not measured yet. The change removes the per-cell crossings, which the tests count, but I have not timed it. The speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2, as CONTRIBUTING asks.

How I checked it

The tests come first in the first commit and the change in the second, so a run of the first commit shows them failing on the old code.

  • tests/test_arrow_temporal_column_jvm.py (JVM wheel only, listed in NATIVE_NOT_APPLICABLE): two tests count what one load does and fail on the old code. One asserts no java.time call and no convert_python_to_java per cell for a 600-row batch of seven temporal columns (the old code makes one of each per cell). The other asserts that each of the seven columns reaches the bridge as one TemporalColumn per batch (500, 500 and 300 rows). The rest compare the bridge column with the per-cell reading it replaced (every unit, random ticks, negative ticks, nulls), pin the refusal of a bad unit, and pin that a timestamp[s] of 2**62 is refused with nothing written. No test asserts on time.
  • tests/test_dataframe_ingest.py::TestTemporalColumns (both wheels): every timestamp unit with no zone, with UTC and with a zone, to the nanosecond through a DATETIME_NANOS property read back with format('yyyy-MM-dd HH:mm:ss.SSSSSSSSS'); date32 and date64 before 1970, declared and undeclared; a load in batches of six timestamp and two date columns with nulls into a document type, a document type with parallel=True and a vertex type; a pandas frame (with a NaT and a zone) and a Polars frame. The expected values come from the integers, never from a read of the database.
  • Run on this branch, light checks only: uvx pre-commit run --files on the changed files (black, isort, ruff, prettier on the Java and the adoc, whitespace), bandit, pyflakes, and a cross-check of the expected wall clocks against numpy datetime64.
  • Not run locally, so CI has to prove them: the Java compiles (no JDK 25 on this machine for it), and every test above, on both wheels. A draft pull request skips the platform legs, so they start when it is marked ready.

CI: to be filled in once the legs have run.

Docs

docs/modules/ROOT/pages/how-to/bulk-loads.adoc: one paragraph on how a timestamp or date column crosses on arcadedb-jvm. No other page or the README describes it, and no documented behavior changes.

Checklist

  • A regression test fails on the old code and passes on the new (the first needs the CI run of the first commit to be seen)
  • The test suite that the change can affect passes (CI)
  • uvx pre-commit run --files <changed files> passes

🤖 Generated with Claude Code

tae898 and others added 2 commits November 14, 2026 02:47
…taframe on the JVM wheel (refs ArcadeData#315)

Round trips of every Arrow temporal type through a real load on both wheels (every timestamp unit with and without a zone, to the nanosecond; date32 and date64 before 1970; nulls; batches; vertex and parallel loads; pandas and Polars frames), and JVM-only tests that count what one load does: no java.time call and no convert_python_to_java per cell, and each timestamp or date column reaching the bridge as one TemporalColumn per batch. The counting tests fail on the code before, which builds one LocalDateTime or LocalDate per cell in Python and converts every element again.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…te column crosses as one long[] of epoch ticks, not one JPype call per cell (refs ArcadeData#315)

_temporal_to_java made one LocalDateTime.ofEpochSecond or LocalDate.ofEpochDay call per cell, and insert_columns converted every element of that list again with convert_python_to_java: a million-row timestamp column was two million JPype calls. The column is now the integer epoch values in one long[] (plus one boolean[] when a row is null) wrapped in the new bridge class TemporalColumn (dateTimes(long[] ticks, long ticksPerSecond, boolean[] nulls) and dates(long[] ticks, long ticksPerDay, boolean[] nulls)), and DocumentBatcher builds each LocalDateTime or LocalDate as it fills the document of that row, as StringColumn does for text. The reading rules are the ones of the code before: UTC wall clock, a zoned timestamp is already UTC epoch time, a naive one is the clock it shows, the unit's precision is kept (a nanosecond tick is not cut to microseconds), a tick before the epoch floors, date64 is milliseconds divided by 86,400,000 with a floor. The factory checks the smallest and the largest tick, so a value past the years of java.time still raises before a row is written. The per-cell path is retired: the wheel carries its own bridge jar, so no fallback for an older one. Not measured: the speed claim waits for a timed A/B against 26.11.1.post2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898

tae898 commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator Author

This change shipped in 26.11.1.post3 (on PyPI since 2026-11-14) through the release line, which #387 merged into main with a merge commit. The release line carries the same change as its own commits, so I close this draft. The measurements are on #315.

@tae898 tae898 closed this Nov 14, 2026
tae898 added a commit that referenced this pull request Nov 14, 2026
…, and the stale statements (refs #315) (#399)

JVM iter_dicts parses a batch with no tagged value without the object hook (#382); JVM insert_arrow and insert_dataframe carry a timestamp or date column as one long[] of epoch ticks (#383); native transaction calls return the state as one integer (#384); native save() of a saved record with auto-transaction on skips the transaction check (#385). The figures are copied by script from the scored gate study and each is checked against the public comment on #315. The n_update_one figure ran with auto-transaction off, so the page says off (the comment says on). The save() paragraph said three calls without the auto-transaction case. The quickstart was re-run on both published 26.11.1.post3 packages on Python 3.12 and printed the output shown.

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jvm The arcadedb-jvm package (JPype build) performance Speed or memory of the bindings, with a measurement python Pull requests that update python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant