Repository navigation
JVM insert_arrow, insert_dataframe, insert_columns: a timestamp or date column crosses as one long[] of epoch ticks, not one JPype call per cell (refs #315) - #383
Closed
tae898 wants to merge 2 commits into
Conversation
…taframe on the JVM wheel (refs ArcadeData#315) Round trips of every Arrow temporal type through a real load on both wheels (every timestamp unit with and without a zone, to the nanosecond; date32 and date64 before 1970; nulls; batches; vertex and parallel loads; pandas and Polars frames), and JVM-only tests that count what one load does: no java.time call and no convert_python_to_java per cell, and each timestamp or date column reaching the bridge as one TemporalColumn per batch. The counting tests fail on the code before, which builds one LocalDateTime or LocalDate per cell in Python and converts every element again. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…te column crosses as one long[] of epoch ticks, not one JPype call per cell (refs ArcadeData#315) _temporal_to_java made one LocalDateTime.ofEpochSecond or LocalDate.ofEpochDay call per cell, and insert_columns converted every element of that list again with convert_python_to_java: a million-row timestamp column was two million JPype calls. The column is now the integer epoch values in one long[] (plus one boolean[] when a row is null) wrapped in the new bridge class TemporalColumn (dateTimes(long[] ticks, long ticksPerSecond, boolean[] nulls) and dates(long[] ticks, long ticksPerDay, boolean[] nulls)), and DocumentBatcher builds each LocalDateTime or LocalDate as it fills the document of that row, as StringColumn does for text. The reading rules are the ones of the code before: UTC wall clock, a zoned timestamp is already UTC epoch time, a naive one is the clock it shows, the unit's precision is kept (a nanosecond tick is not cut to microseconds), a tick before the epoch floors, date64 is milliseconds divided by 86,400,000 with a floor. The factory checks the smallest and the largest tick, so a value past the years of java.time still raises before a row is written. The per-cell path is retired: the wheel carries its own bridge jar, so no fallback for an older one. Not measured: the speed claim waits for a timed A/B against 26.11.1.post2. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
Collaborator
Author
tae898
added a commit
that referenced
this pull request
Nov 14, 2026
…, and the stale statements (refs #315) (#399) JVM iter_dicts parses a batch with no tagged value without the object hook (#382); JVM insert_arrow and insert_dataframe carry a timestamp or date column as one long[] of epoch ticks (#383); native transaction calls return the state as one integer (#384); native save() of a saved record with auto-transaction on skips the transaction check (#385). The figures are copied by script from the scored gate study and each is checked against the public comment on #315. The n_update_one figure ran with auto-transaction off, so the page says off (the comment says on). The save() paragraph said three calls without the auto-transaction case. The quickstart was re-run on both published 26.11.1.post3 packages on Python 3.12 and printed the output shown. Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
On the JVM wheel,
insert_arrow,insert_dataframeandinsert_columns(called by the first two) made oneLocalDateTime.ofEpochSecondorLocalDate.ofEpochDayJPype call per cell of every timestamp or date column, and then_column_to_javaconverted every element of that list again. A million-row timestamp column was two million JPype calls. The native wheel moved the same columns to one crossing in #333 (#316).Now a timestamp or date column of a batch is the integer epoch values in one
long[](plus oneboolean[]when a row is null), wrapped in a new bridge classTemporalColumn, built with one call per column and batch.DocumentBatcherbuilds eachLocalDateTimeorLocalDateas it fills the document of that row, the wayStringColumndoes for text. The old per-cell path is gone. The wheel carries its own bridge jar, so there is no fallback for an older one.The Java side, in
src/java/com/arcadedb/python/TemporalColumn.java:DocumentBatcher.checkColumnsandfillRowtake it besidelong[],double[],boolean[],Object[],FloatVectors.MatrixandStringColumn. In Python,dataframe._temporal_columnreplaces_temporal_to_java, andcore._column_to_javahands the column over as it is.How the stored value is kept
ticksPerSecond1, 1,000, 1,000,000 or 1,000,000,000).floorDiv(tick, ticksPerSecond)and the rest isfloorMod(...) * (1e9 / ticksPerSecond)nanoseconds, so a nanosecond tick keeps its nanoseconds and a tick before the epoch is the last tick of 1969. This is thenp.divmodarithmetic of the old code, now in Java.ticksPerDay1) and date64 is milliseconds (ticksPerDay86,400,000), floored as the old numpy//was.boolean[], which is only sent when there is a null.get(i)returns null for it.java.timestill raises before a row is written, as it did in Python: the factory builds the value of the smallest and of the largest tick of the non-null rows, and the rest lies between them.datetimeobjects passed toinsert_columnsis not touched: it still crosses value by value. A numpydatetime64column is still refused there.Related issues
Part of #315.
Performance changes
Not measured yet. The change removes the per-cell crossings, which the tests count, but I have not timed it. The speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2, as CONTRIBUTING asks.
How I checked it
The tests come first in the first commit and the change in the second, so a run of the first commit shows them failing on the old code.
tests/test_arrow_temporal_column_jvm.py(JVM wheel only, listed inNATIVE_NOT_APPLICABLE): two tests count what one load does and fail on the old code. One asserts nojava.timecall and noconvert_python_to_javaper cell for a 600-row batch of seven temporal columns (the old code makes one of each per cell). The other asserts that each of the seven columns reaches the bridge as oneTemporalColumnper batch (500, 500 and 300 rows). The rest compare the bridge column with the per-cell reading it replaced (every unit, random ticks, negative ticks, nulls), pin the refusal of a bad unit, and pin that atimestamp[s]of 2**62 is refused with nothing written. No test asserts on time.tests/test_dataframe_ingest.py::TestTemporalColumns(both wheels): every timestamp unit with no zone, with UTC and with a zone, to the nanosecond through aDATETIME_NANOSproperty read back withformat('yyyy-MM-dd HH:mm:ss.SSSSSSSSS'); date32 and date64 before 1970, declared and undeclared; a load in batches of six timestamp and two date columns with nulls into a document type, a document type withparallel=Trueand a vertex type; a pandas frame (with aNaTand a zone) and a Polars frame. The expected values come from the integers, never from a read of the database.uvx pre-commit run --fileson the changed files (black, isort, ruff, prettier on the Java and the adoc, whitespace), bandit, pyflakes, and a cross-check of the expected wall clocks against numpydatetime64.CI: to be filled in once the legs have run.
Docs
docs/modules/ROOT/pages/how-to/bulk-loads.adoc: one paragraph on how a timestamp or date column crosses onarcadedb-jvm. No other page or the README describes it, and no documented behavior changes.Checklist
uvx pre-commit run --files <changed files>passes🤖 Generated with Claude Code