Repository navigation
Merge the 26.11.1.post2 release line into main - #351
Merged
Merged
Conversation
…ouble, as on the native wheel (refs #314) (#320) RowBatcher.keepNonFinite now widens a Float to a Double and a float[] to a double[] before the engine's JSONObject writes it, so the JSON text is the exact float64 widening of the float32 (0.10000000149011612, not the shortest float32 text 0.1). NaN and infinity keep their tag. tests/test_cross_wheel_golden.py is one fixed table with committed golden literals for to_list, to_json_list and to_columns (FLOAT, DOUBLE, float32 float[] of 3 and 9, DECIMAL, LONG beyond 2**53, DATE, DATETIME, NULL, embedded list), run unchanged on both wheels. Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> (cherry picked from commit 551dee9) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…r-row steps, one _kind per column (refs #312) (#321) * Native columnar readers: infer Arrow types first, skip per-key and per-row steps, one _kind per column (refs #312) Bindings-only, Python in arcadedb._native, no image rebuild. Output is meant to be identical to before: - columns.to_arrow tries pa.array(values) first and keeps the old per-value rules for a column that inference cannot type (ArrowInvalid, ArrowTypeError) or types differently (temporal types, unsigned ints). - columns._names tests each row's keys against the known ones in C instead of a Python step per key. - columns.to_columns calls _kind once per column instead of twice. - NativeResultSet._fill can hand a reply's rows straight to _drain_dicts, which no longer builds a tuple and a generator step per row. The float32-tail view in codec.loads_tagged is not done: a vector must stay a list of floats for the row API. tests/test_native_columns_reference.py keeps the old rules as a private reference and compares tables, columns, names and the drain on mixed, null-heavy, ragged and vector inputs. No speed is claimed here. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * Native columns: a failed inference of any kind falls to the old rules; tests compare NaN tables by value (refs #312) pa.array([2**63]) raises OverflowError, not ArrowInvalid, so an int above int64 next to floats would have raised where the old explicit-type call fell back to exact text. _inferred now leaves every failed inference to the rules, so the outcome of such a column is decided by the old code. Four reference cases cover it. Table.equals is False for a NaN next to the same NaN, so the comparison is the schema plus the repr of the values. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> (cherry picked from commit 6c6a5a1) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…py, free it once (#323) The reply was copied as bytes (string_at), cut for the float tail, decoded to str, then parsed. It is now decoded straight from the library's memory with PyUnicode_FromStringAndSize, only the last 40 bytes and the float tail are copied as bytes, and the free is in the finally of the same call. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…325) A parallel call skipped the columnar fast path and went through one dict per row and one JSON text per chunk of 10,000 rows. The columnar entry arcadedb_n_insert_colp now takes "parallel": true in its header: the image builds the records from the column buffers and hands them to the async writers (db.async().createRecord with a failure token), as DocumentBatcher.insertColumnsParallel does on the JVM wheel. The Python side waits for the writers and reads the failures exactly as insert_many(parallel=True) does (_finish_parallel, shared). Rows the buffers cannot carry still take the row path. The native library is rebuilt from the unchanged engine commit. Tests (native only): a spy proves one colp crossing and no insert_parallel chunk; the stored records equal those of insert_many(parallel=True); nulls, vertex types, rejected records, an edge type, a missing type, the fall back and the off switch. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ing of the statement (#317) The create path read the row it always reads with a second crossing (arcadedb_n_next_bin after arcadedb_n_cmd). It now uses the form of arcadedb_n_exec whose reply carries the first rows, and so do the two UPDATE forms of save(). The stale-record check stays: an UPDATE's own reply comes after the write. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… columnar entry (refs #325) A document load with parallel=True left the Arrow columns as Python values and went to insert_many(parallel=True). It now keeps the columns as they are for the columnar entry, which hands the records to the async writers; a batch the buffers cannot carry still takes the row path. A test spies on the entry points and compares the stored records with the row path of the old code. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… pass, not in a loop before it (refs #318) json_bulk_dumps and json_bulk_dumps_tagged walked every value of every row in Python before json.dumps walked the rows again. The rows are now written once and the checks come from what that pass shows: the text (a surrogate escape, an integer token of 2**63 and above) and a census of the types of the keys and values in C. The exact per-row walk (_check_rows) is the reference and runs only when a value may be bad, and when json.dumps itself refuses the rows, so the exception type and message for the first bad row in row order are the old loop's. Fuzzed against the old code: 60000 cases, no difference. JVM GraphBatch.create_vertices: classify_vertex_rows decides the shapes by type census instead of a json_bulk_scalar_ok call per value; a scalar the text would change (NaN, integer beyond 64 bits, lone surrogate, subnormal) now sends its own chunk through the property-matrix path, as a tagged chunk already did. The untagged path now also refuses a subnormal, as the tagged one did. NativeGraphBatch: the dead _create_vertices_json_bulk is deleted, and the row type check is a C census. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…olars, read by the decoder the JVM wheel uses (refs #312) The native library answers a read of a cursor as one column frame of about 1 MiB (arcadedb_n_next_col, ColumnFrames.java), in the layout that ColumnBatcher writes for the JVM wheel: int64, float64, bool bytes, epoch milliseconds or microseconds, float32 vectors, Arrow strings (int32 offsets and a UTF-8 blob), DECIMAL as text, and a tagged JSON column for what no fixed layout carries without changing a value (nested values, records, a mixed column, an integer beyond 2**53 next to a float). arcadedb._columnar holds the one decoder and the merge of frames that both wheels call; results.py keeps only its JPype transport and the text of its json columns. The first reply of a query is still JSON for the row APIs, and the cursor keeps those rows (at most 256, until its next read by row) so that a columnar read replays them into its first frame. A result that the first reply held whole has no cursor: its dicts go through the old rules once, as one more chunk. The old JSON read path stays for to_list, to_json_list and row iteration in this commit. Tests: tests/test_column_frames.py (the decoder on hand-built frames, and the stored table through every columnar API, both wheels), tests/test_native_column_frames_reference.py (the old route copied into the test as the reference, native only). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… flags the wheel starts with (#313) jlink makes a JRE without a class-data-sharing archive. scripts/dump_cds_archive.sh dumps the JVM's default archive as the last step of the JRE build (Docker and the native macOS and Windows builds), with -XX:+UseCompactObjectHeaders as jvm.py sets it, and checks it with -Xshare:on. The AOT cache stays opt-in. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…b plus int32 offsets (refs #324) A string column used to cross JPype one element at a time: convert_python_to_java per value and an Object[] of Java Strings built element by element, all alive at once. A column of nothing but str and None is now one byte[] (the UTF-8 text of all rows), one int[] of row + 1 offsets and, when there is a null, one boolean[], wrapped in the new bridge class StringColumn; DocumentBatcher.insertColumns and insertColumnsParallel decode each row when they build its document, so the Java Strings are short-lived. The blob crosses through bytes into JArray(JByte), a copy; nothing relies on a JPype view of Python memory. Kept as before: a mixed column (an int, bytes, a date-time) and an empty column convert per element; a column with a lone surrogate (no UTF-8 form) and one beyond an int offset keep the per-element path. insert_dataframe and insert_arrow reach the change through insert_columns. Stored values are the same on both wheels (tests compare with the per-element path and with the values given). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… after black wrapped it Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ed query (refs #318) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… keeps the rest for the next call With batch_size=64 the first frame took 64 of the 256 replayed rows and the cursor then dropped the other 192 (test_to_arrow_multi_batch: 108 rows of 300). The library now puts back the unread tail of the head after each frame. Also: the reference test named a property "by" (an SQL keyword), and the dataframe test expected None where pandas 3 shows nan for a missing string. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ath, with mutation proofs (refs #336) One check (tests/ingest_integrity.py) compares the stored type with the rows that were sent: the count read three ways, an order-independent digest of every column read two ways, every key exactly once, and all of it again after close and reopen. test_ingest_integrity.py runs it for insert_many (list and generator), insert_columns, insert_arrow, insert_dataframe, create_vertices, async create_record and command, append_samples, each sequential and parallel where it exists, at every chunk and commit boundary, for edges (endpoint pairs and weights), several writer threads with a reader thread, a bad row mid-batch with the documented outcome exact, rejected and raising async callbacks, and close straight after the last call. Every family also runs with a rows-dropped, duplicated, altered or uncommitted fault and must report it. The native and JVM seams are in their own files. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…per property on both wheels (refs #337) JVM: a Java float[] was no vector candidate, so classify_vertex_rows gave up on the first row and the whole load went through _to_java_property_matrix (convert_python_to_java per value, one Object[] per row, one Object[][] per call): 59 to 61 percent of the wall of the e2 and l3d loads in the untimed ingest profile of 26.11.1. A float32 numpy array on an undeclared property took the same way. take_vector_columns (type_conversion) now picks the columns: declared ARRAY_OF_FLOATS properties as before plus Java float[] and double[], and any other property whose value in every row is a float32 array or a Java float[]; the stored value of those is a float[] on both paths. The rest rows are rebuilt from columns by C iterators when the rows share their keys. Native: the float32 arrays of a row went through json.dumps with a Python hook (_gb_default, _default) per row, after a _prep walk of every value, one ["B", offset, count] tag per row parsed back in Java, and np.concatenate of one small array per row (29 to 31 percent prep in the same profile). encode_vertex_args writes the other properties with one json.dumps (no hook when they are plain scalars) and the matrices are the front of the side buffer, a ctypes view with no copy, described by one "vecs" entry that NativeExtras.withVectors turns into one float[] per row. The old per-row tag stays for edge properties and for a column that does not qualify. Tests: both wheels pin the stored bits and the split; the JVM wheel counts the JPype crossings and the property-matrix fallbacks; the native wheel counts the ctypes crossings and the codec's per-row hooks. The existing native test that expected a tag per vertex row now expects the matrix. The Java is not compiled locally: CI builds the native image. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…loss (review of #328) The failure token is now made by the caller (arcadedb_n_async "token") and its id travels in the colp header, so the caller holds the id whatever happens to the crossing. An interrupt in it (or between the chunks of the row path) waits for the writers and reads the tokens before it propagates; a wait that raises still reads every token (the entries of an unread token lived forever); the error of a failed handover says how many rows were handed over and that some may have been stored. insert_table(parallel=True) no longer says the earlier batches were written in the caller's transaction (the writers commit them, a rollback does not remove them). Tests: duplicate keys (stored + reported >= n, stored == distinct), the writers' batch boundary, nothing lands after the call, an interrupt in the crossing and in the row path, a failing wait, the in-transaction message. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…(no json.dumps __context__); test the stored type across chunks (refs #318) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… belong to the whole result, bounded frames The first reply of a query stays what the caller already holds as dicts and is written as one more frame by Python (colframes.py) by the rules of the library's encoder, so the cursor keeps no rows (the replay kept up to 256 live Result objects per open cursor and read them after the snapshot the JSON reply had taken). The JSON reply names the properties that were float[] or double[] (fv, dv): a vector column is a declared array, not a list that happens to fit float32, at any result size. columns.py, with its size-dependent guess, is gone. Frames: the row limit is clamped to a C int (a batch_size of 2**31 or more wrapped to zero and looped), a frame ends by the larger of the values read and rows x columns x 9 bytes (a schemaless result with a property per row allocated gigabytes against a 1 MiB estimate), an empty frame from an open cursor is an error, a NULL reply and a library older than the wheel are errors with a message. Types: float32 next to float64 or another dimension unify to a list of float (never text), a frame of only nulls takes the type of the others, an integer beyond 2**53 next to a float stays exact in a json column on both wheels (ColumnBatcher too), an empty result with columns= returns those columns on both wheels, to_columns arrays are writable, to_dataframe vectors are numpy float32 cells and datetimes datetime64[us] on both wheels (documented). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…'s open batch (CI of the first candidate) The first candidate run showed stored 1000 of 2950 distinct keys: a writer that meets a duplicate key rolls back its whole open batch (up to commit_every rows), so the valid rows of that batch are abandoned and reported too. stored == distinct was the wrong invariant. Assert stored + reported >= n, stored <= distinct, only sent ids, none stored twice. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ay columnar (refs #316) One column of a date-time, date, Decimal or bytes value used to make columnar.pack return None, so every column of the batch went through one dict per row and a JSON text with tagged values. BulkOps.insertColumns gets four kinds that read the same objects Json.decodeTag builds for those tagged values, so the stored value is the same: T (int64 ticks of the UTC wall clock since the epoch, dim = ticks per second so an Arrow timestamp keeps its unit, LocalDateTime), D (int32 epoch days, LocalDate), n (decimal text in the string layout, BigDecimal) and x (bytes in the string layout, byte[]). columnar.py lays out lists of datetime (also pandas.Timestamp), date, Decimal, bytes and bytearray and Arrow timestamp, date32, date64, decimal and binary arrays; a time-zone-aware value is its UTC wall clock and a naive one is taken as it is, the rule of _tag_datetime. The DataFrame and Arrow loader keeps those Arrow columns for the entry and turns them into tagged values only when a batch falls back to the row path. The native library is rebuilt from the unchanged engine commit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… the crossing (review of #333) A date64 past int32 days wrapped in the int32 cast (the row path raised) and a timestamp[s] past the years of java.time threw DateTimeException in the middle of a batch (BulkOps T kind, LocalDateTime.ofEpochSecond). Both now make pack() return None, so the call takes the row path and raises what it raised before, with nothing stored. Only a timestamp[s] can pass the java.time range: an int64 of milliseconds is at most 9.2e15 seconds against 3.15e16. Tests: the pack-level limits, the sync call (nothing stored, no transaction left open), the parallel call (may have been stored), and a date-time chunk with NaN stores the NaN that INSERT ... CONTENT :row stores (both end in MutableDocument.putProperty). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ound the whole batch (refs #337) codec._prep is counted by name, so it also counted the calls the batch makes outside create_vertices: creating the batch (extras._enc of its options runs the original _prep, whose recursion goes through the patched name, once per option) and close() (encode_gb_args({}) once). That is 2 calls, matching the Free leg; they do not depend on the rows. The create_vertices path itself calls neither _prep nor a hook (take_vector_columns, _plain_rows and one json.dumps). The first test now takes the hook counts and the crossings around the create_vertices call (gb_default and default exactly 0, prep at most one per crossing), and a new test runs 250 rows in three chunks and 2,500 rows in three chunks with the same bounds: the old path called the hooks per row. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… form (#317) A CI run showed that an INSERT or UPDATE error is raised by the call in both forms, not when the row is read. The test pins that the two forms agree, and a SELECT run as a command pins the read-time case for both. The helper's docstring states the timing. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…n to_dataframe A dimension that changes inside a frame makes a json column (list<double>), so the same change between two frames now gives list<double> too (it gave list<float>); float32 next to float64 of one size is fixed_size_list<double>. In to_dataframe a vector cell is a numpy float32 array or None for a null, on both wheels (a NaN row, which to_columns keeps in its matrix, read as a vector of NaN where the old route had a missing value); to_arrow and to_polars already kept the null. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ds changed or that were mis-built - review fixes: the date64 case built an int64 beyond its own range (DAY_MS * 2**40); use 2**33, still past int32 days. A non-finite Decimal (NaN, -NaN, sNaN, Infinity, -Infinity) is refused in lists, with None, and in object columns. - nan-decimal fall back: the NaN row goes to the statement as before; the finite row around it is a run of carriable rows and crosses on the columnar entry now (a finite Decimal is carried), so one colp call is expected, not none. - pack refusals: bytes alone are carried (kind x) now; refuse bytes with text and bytes with bytearray instead. - temporal lists: a BINARY property reads back as the signed values of a Java byte[] or as bytes; compare after & 0xFF. - mixed chunk: the rows around an uncarriable row cross on the columnar entry (the date-time column is carried), one colp call per run. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lumnar insert (refs #319) Both wheels loaded a vertex type through GraphBatch.create_vertices with one dict per row and a JSON text per chunk, and discarded the RIDs. The Java column inserters already create vertices (BulkOps.insertColumns, DocumentBatcher.insertColumns): newVertex, set per property, save, in a transaction, the loop GraphBatch.createVertices runs, so the stored records, the index entries and the bucket of each vertex are the same. insert_table now sends a vertex type the way it sends a document type; the GraphBatch is no longer opened for it. The rule kept from GraphBatch: a vertex load refuses to run inside a transaction the caller opened (a document load still joins it), with the engine's wording, before anything is written. parallel=True on a vertex type is still refused as before. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…on, on both wheels (review of #335) GraphBatch.createVerticesWithRetryInternal rolled a vertex chunk back and wrote it again (back-off from 1 s doubling to 10 s, 10 retries) on any NeedRetryException: an MVCC ConcurrentModificationException, a lock timeout. The vertex load through the columnar insert is one plain transaction per chunk, so two threads on one vertex type with a NOTUNIQUE index would get the error with earlier chunks committed. BulkOps.insertColumns (native wheel) and DocumentBatcher.insertColumns and insertManyJson (JVM wheel) now write a chunk in a transaction of their own, roll it back on a NeedRetryException and write it again from its first row, bounded like the engine's and rethrowing the error unchanged after the last attempt. A transaction the caller opened is untouched: rows join it, nothing is retried. Tests (both wheels): 2 and 4 threads, 20 batches each on one vertex type, count == total; a duplicate in the third of five batches keeps the first two, leaves no index entry of its own, states the committed count, and the fixed tail loads. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…osses as one matrix on the JVM wheel A FixedSizeList<float32> (or list, or large list) column into an ARRAY_OF_FLOATS property went chunk.to_pylist() (one Python float per element), then core._column_to_java converted every row with convert_python_to_java (an ArrayList of 96 boxed doubles per row, 100,000 conversions for ten batches of 10,000 rows) and the engine narrowed each list back to a float[]. That was the cost the released route did not have (GraphBatch took the rows differently): in the untimed profile of the l3d shape the conversion was the largest part of the call, the Python heap peaked at 31.5 MiB for a 3.7 MiB buffer, and the boxed doubles are what the JVM heap held. _column_values now takes a float32 vector column of one length, with no null row and no null element, as one 2-D float32 numpy matrix (a view of Arrow's buffer; flatten follows a slice's offset), and _column_to_java already sent a 2-D float32 numpy column as one float[] inside the bridge's FloatVectors.Matrix, so the documents get a row each Java-side. Taken only for a property declared ARRAY_OF_FLOATS and only on the JVM backend (float_vectors=not native and declared == ARRAY_OF_FLOATS): the native wheel keeps Arrow arrays and is not touched. Everything else keeps to_pylist and so keeps what that route stored or refused: a null row stays null, a null element is refused, vectors of different length are stored as given or refused by a vector index, float64 and integer lists are narrowed by the engine (and refused past the float range) which a cast here would not do, undeclared and LIST properties keep a list. tests/test_arrow_vector_column_jvm.py (JVM only, registered in conftest.py): 18 tests pin what the old route stored (count and an order-independent digest through SQL and through the result API, both again after close and reopen, special float32 values bit for bit, a null row, a null element, different lengths, a wrong dimension against a vector index, float64 and integer lists, undeclared and LIST properties, empty vectors, chunked tables and slices); 5 count what the load does and fail on the code before (per-row conversions, Matrix crossings, Python heap of one batch). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…d retries a crashed file once (#295, #347) The Windows native parity legs of the 26.11.1.post1 release run lost one test process each (test_free_threading.py on 3.11, test_find_nearest_query_forms.py on 3.14): no traceback, no XML, no exit code, so nothing told an access violation from a runner kill. It did not reproduce in 360 isolated runs. The workflow and harness are read at the tag, so this has to ship in a new tag. Harness (native/wheel/parity_wheel.py, decision in the new parity_crash.py): - a file whose process ends with no junit result is CRASH(rc=0xC0000005,3221225477) (NTSTATUS hex and decimal; -11,SIGSEGV on POSIX), the exit code of every attempt goes to the tsv (two new columns, attempts and exit) and to <out>.meta.json - --retry-crash (off by default; test-native.yml passes it when RUNNER_OS is Windows, so Linux and macOS stay strict) runs a crashed file once more in a fresh process: a retry that produces a result is FLAKY-CRASH(rc=<first>), a second crash is CRASH(rc=<first>;retry rc=<second>) and fails the leg. A failing test, TIMEOUT and BADXML are never retried; a leg retries at most 3 files, so a wheel that crashes everywhere cannot pass on retries. The log of the first attempt is kept as <file>.attempt1.log - the XML of an attempt is removed first, so a stale one cannot pass for a result Crash trace: tests/conftest.py disabled faulthandler on Windows because HotSpot raises handled access violations (the JVM build). The native wheel has no HotSpot, so a native run on Windows with ARCADEDB_TEST_FAULTHANDLER_FILE set (the harness sets it per attempt, on Windows only) enables faulthandler writing to that file; the JVM build and every other platform are unchanged (test_faulthandler_is_off_on_windows_only still holds). The files ride in the existing parity artifact (parity-wheel-xml/), empty ones are deleted. Loud: the gate awk (test job and pyver job) lets a FLAKY-CRASH row pass and prints it into the job summary; a new crash-report job lists every FLAKY-CRASH and CRASH of the run and, on a tag push or a dispatch only, posts ONE comment on issue 295 (run URL, attempt, Python, wheel, exit codes). It holds issues: write alone; the token failing, missing, a fork or a pull request only print a warning, it never fails the run. release-native.yml grants issues: write to its call of test-native.yml, because a called workflow cannot hold more permission than its caller (without that line the release run would refuse to start). Tests (tests/test_parity_crash_retry.py, tests/test_parity_gate_awk.py, no skips): the decision with a fake runner, real children that os._exit with a high code or die of a signal, the harness against a fixture repository with real pytest children, the faulthandler switch in a child that really faults, the report and its permission rules, the workflow wiring, and the real gate awk on small tables (the awk file is not collected on Windows). 12 mutations of the new code each fail at least one test; against the tree of release/26.11.1 the awk tests fail and the retry tests cannot import. Not exercised: Windows itself (NTSTATUS exit codes, the faulthandler trace on a real access violation in libarcadedb.dll) and the caller permission path of a tag run. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…erates locked housekeeping files (review of aa223d0) Review findings on the crash handling, all fixed here: - A process that wrote its XML and then died in shutdown was only a ::warning:: line. It is now a DIRTY-EXIT record (exit code hex and decimal, file, attempt) in meta.json, the job summary and the issue 295 comment. It is still not retried and does not fail the leg: the status stays what the XML says, so the gate is unchanged. Probed with a fixture conftest whose pytest_unconfigure calls os._exit. - The report named <file>.attempt1.log for every crash, but that file exists only after a retry. A crash that was not retried (Linux, macOS, a spent budget) now names <file>.log. - The claim that a stale XML cannot pass for a result had no test. A stale XML pre-seeded for a file that then crashes stays CRASH. If the stale XML cannot be removed (locked), an XML older than the attempt's start is not taken as its result. - Removing an empty trace, keeping the first attempt's log, writing a log and removing a stale XML go through housekeeping(): one retry after a short pause on OSError, then a ::warning:: and on. A virus-scanner lock on a just-closed file can no longer fail the harness or hide a result. Tests added to tests/test_parity_crash_retry.py and tests/test_parity_gate_awk.py; 9 further mutations of the new code each fail at least one test (20 in all). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…-EXIT event, correct log name in the #295 table, stale-XML guard with a test, tolerant log housekeeping; from the independent review)
…rm, black 26.10.1 (CI dispatch on the post2 candidate) The dispatch found two failures in the new tests, and a gate I had not run: - bandit (test-jvm.yml: uvx bandit==1.9.4 -c pyproject.toml -r src tests --severity-level low --confidence-level low) flagged B105 on the fake GH_TOKEN values. The repo's convention is a `# nosec B105` marker on the string's line (tests/conftest.py TEST_PASSWORD): the made-up token carries one, the no-token case now builds the environment without the key (the real case) and needs none. Bandit is clean locally: "No issues identified", 271 nosec-disabled. - test_exit_codes_are_written_as_ntstatus_and_decimal_or_as_a_signal expected "-6,SIGABRT": signal.SIGABRT is 22 on Windows. The expectation, and the SIGSEGV one in the real-fault test, are now computed from the signal module. Other Windows-reachable assumptions re-read: the new test files read the workflow, summary and trace files as UTF-8 explicitly; nothing else needed a change. - black 26.10.1 (the pre-commit hook; the base versions of conftest.py and parity_wheel.py were black-clean) would reformat all six touched files: applied. isort --profile black, scripts/check_no_embedded_name.py and scripts/check_test_skips.py are clean. 130 tests pass, 0 skips; the 20 mutations of the new code are each still caught (anchors updated to the black layout). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… on the made-up token, portable signal numbers, black layout; the first CI dispatch found both in the new tests)
…pre-commit hook the CI runs; found by running the static gates on the integration branch before dispatch) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ad moved them onto the closing line of a wrapped call; the CI bandit job failed on the first dispatch of the final commit) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…hed Python 3.11 in CI: SIGSEGV in a JVM thread after tracemalloc.stop()) The full-matrix CI of the post2 integration branch died with "Fatal Python error: Segmentation fault" in test_the_python_heap_of_one_batch_stays_below_twice_the_flat_buffer on Python 3.11 (Linux and Windows); 3.12 to 3.14 passed. Reproduced locally on 3.11.15 and traced to the test method, not to the production path: - fresh processes, faulthandler on, exit codes: insert_arrow of the new route under tracemalloc.start()/stop() around the call: 3 of 12, 1 of 8 and 1 of 2 died with SIGSEGV (139), the single test 1 of 1; the same load with tracemalloc started and never stopped: 0 of 12; the same load with no tracemalloc: 0 of 8 and 0 of 1; the old to_pylist route under tracemalloc start/stop: 0 of 8; the released 26.11.1 GraphBatch route under tracemalloc: 0 of 8 (plain 0 of 8). - no ArcadeDB code in the loop: jpype.JArray(JFloat) of a 10,000 x 96 float32 numpy array plus FloatVectors.Matrix, tracemalloc start/stop around it: 6 of 12 died; never stopped: 0 of 12; no tracemalloc: 0 of 12. The crash is in a thread with no Python frame (a JVM thread), after tracemalloc.stop(), which is a known 3.11 weakness (stop swaps the allocators while a thread outside the GIL still allocates); the other Pythons passed. So the production path (_float32_matrix, _column_to_java) is not at fault and stays. The memory-shape assertion now looks at what the conversion hands over: a spy on dataframe._column_values must see ONE 2-D float32 numpy array of shape (10000, 96) that is a view (not a copy, not per-row lists). On the old dataframe.py it fails (isinstance of the list of lists); the two call-count tests are unchanged. Result: the file passes (23) on 3.11, 3.12 and 3.14, and the new test alone passes in 12 of 12 fresh 3.11 processes. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The release line (tag 26.11.1.post2 = 1454809, which contains the post1 tag 26.11.1.post1 = dbe1670) branched from the 26.11.1 tag and carries the post1 candidates (the native column frames #331, the other native and JVM load and read changes of #327 to #339, and the cherry-picks of main's #320 and #321), the post2 additions (Windows parity harness #295/#347, #340, the JVM Arrow vector load change) and their test fixes. Main did not have them. The merge uses a merge commit so that the release tag stays an ancestor of main. Four files conflicted, each because main and the release line changed the same lines. Resolutions, each checked against the history of both sides: - src/arcadedb/_native/columns.py (modify/delete): deleted. Main changed it only with #321 (6c6a5a1). The release line holds the same change as the cherry-pick 946dfda (identical blobs of columns.py and of test_native_columns_reference.py at both commits) and then deleted the module with the native column frames (#331, d606cce). Nothing else imports it; the merged database.py reads columns through the frames code. - src/arcadedb/_native/database.py: kept both. Main's facets lines in _fill and its facets property (#299, #301), then the release line's self._note_vectors(reply) before the return and its _note_vectors method. - tests/conftest.py: the release side of the native-only block (the comment of test_native_columns_reference.py now says what the file compares, plus the two new native-only files), and both entries in WINDOWS_NOT_APPLICABLE (main's test_dev_engine_pin.py and the release line's test_parity_gate_awk.py). - tests/test_native_columns_reference.py (add/add): the release side whole. Main's file tested the deleted module. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… modules that read the columns now The page came with main and its first comment line named _native/columns.py (#321) as the source of to_columns, to_arrow, to_dataframe and to_polars. The release line deleted that module with the native column frames (#331), so the merge of the release line leaves the comment pointing at a file that is gone. It now names _columnar.py and _native/colframes.py. The comment does not render; no statement of the page changed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898
added
performance
…n frames do, not the dict drain they replaced The CI of the merge PR (#351, parity against the installed wheel, linux-x86_64, Python 3.12) failed one test of 3401: tests/test_dataframe_columns_behavior.py::test_how_the_rows_reach_the_columns_differs_between_the_builds, `assert [] == [2]`. The test came with main (#282) and pinned the native path before the column frames (#331): every row read into Python dicts (`_drain_dicts`) and `batch_size` unused. After #331 the native readers (`to_columns`, `to_arrow`, `to_polars`, `to_dataframe`) read the rows still in the cursor as column frames through `lib.call_frame`, at most `batch_size` rows per request (25,000 for `to_dataframe`); only the rows the result already holds in Python (the first reply of 256 rows, rows taken by iteration) are dicts. `_drain_dicts` is still used by `to_list` and `to_json_list`, not by the column readers. On a 600-row result both builds ask the engine for exactly `batch_size` rows per request (native 4 requests of 100 for the 344 rows past the first reply, JVM 7). The test is replaced by one that holds on both builds: it watches the request limit at the seam of the build it runs on (the Java bridge's nextColumnBatch, or lib.call_frame) and asserts the same thing, every request carries `batch_size`, and a smaller `batch_size` makes more requests, with the ids intact. It fails with `batch_size` ignored on both builds (checked by mutating `_read_frames` on the native wheel of the PR run and `_read_columns` on the JVM overlay). The "Differences between the two builds" section of dataframes-and-columns.adoc said the native build reads every row into dicts and does not use `batch_size`; it now says both builds read in requests of at most `batch_size` rows, and the one difference left is the message of a missing pandas. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Collaborator
Author
|
The full-matrix runs of the three workflows on I merge this pull request with a merge commit, not a squash, when these three runs and the checks of the pull request itself are all green. I will write the result here. |
This was referenced Nov 10, 2026
Collaborator
Author
|
The three full-matrix runs on |
This was referenced Nov 10, 2026
tae898
added a commit
that referenced
this pull request
Nov 10, 2026
…ls (#364) The Windows examples leg of PR #351 crashed in the JVM (EXCEPTION_ACCESS_VIOLATION in jvector VamanaDiversityProvider.isDiverse) and both files HotSpot wrote were lost: hs_err_pid<pid>.log in the per-user log directory (-XX:ErrorFile, src/arcadedb/jvm.py, _logdir.py) and hs_err_pid<pid>.mdmp in the working directory (examples/). The existing upload only took examples/*.log. Two steps on failure(), one artifact each per matrix leg, 14 days, if-no-files-found: ignore. They are separate because upload-artifact keeps one root directory per artifact and on Windows the log directory (C:) and the workspace (D:) have no common ancestor. include-hidden-files is on for the Linux directory (~/.local); the patterns name only hs_err_pid files. Refs #352 Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request must be merged with a MERGE COMMIT, not squashed and not rebased. It brings the release line of
26.11.1.post2intomain, and the release tag26.11.1.post2(1454809, which contains the tag26.11.1.post1) has to stay an ancestor ofmain; a squash would break that ancestry. The repository allows merge commits.What does this PR do?
I merge
14548090045494d8ce1d1595aea813e84ee7fa67(the tag26.11.1.post2) intomainwithgit merge --no-ff. The release line branched from the26.11.1tag. 68 of its commits (47 without the merge commits) are not onmain, and 27 commits ofmainare not on the release line. Againstmainthe merge changes 74 files (12659 insertions(+), 1669 deletions(-)).The post1 candidates came in through
release/26.11.1as #326 to #339. They givemainthe native column frames forto_arrow,to_columns,to_dataframeandto_polars(#331), the native reply read from its address in one copy and freed once (#327), the row of a created or updated record read in the crossing of the statement (#329), the row checks ofinsert_manyandcreate_verticesdone in the onejson.dumpspass (#330), a class-data-sharing archive for the JRE of the JVM wheel (#332), astrcolumn ofinsert_columnscrossing as one UTF-8 blob on the JVM wheel (#334), vertex loads ofinsert_dataframeandinsert_arrowthrough the columnar insert with the parallel (#328) and temporal (#333) loads under it (#335), the ingest integrity tests (#338), andfloat[]and float32 arrays crossing as one matrix per property increate_vertices(#339). The cherry-picks of #320 and #321 are already onmain: their changes are the same on both sides.The post2 additions are the Windows parity harness that records crash exit codes, keeps a trace and retries a crashed file once (#295, #347),
to_dataframekeeping an integer beyond 2**53 next to a float exact (#340), a float32 vector column ofinsert_arrowandinsert_dataframecrossing as one matrix on the JVM wheel (the speed fix of #348, whose memory report stays open), and the test fixes that the CI dispatches of the post2 candidate found.How I resolved the four conflicts
Each one is a collision of a change of
mainwith a change that the release line made to the same lines. I read both sides and the history of both before choosing.src/arcadedb/_native/columns.py(modified onmain, deleted on the release line): deleted.mainchanged it only with Native columnar readers: infer Arrow types first, skip per-key and per-row steps, one _kind per column (refs #312) #321 (6c6a5a1). The release line has the same change as the cherry-pick946dfda:columns.pyandtests/test_native_columns_reference.pyare byte-identical at both commits. It then deleted the module with the native column frames (Native: column frames for to_arrow, to_columns, to_dataframe and to_polars, read by the decoder the JVM wheel uses (refs #312) #331,d606cce), so nothing of Native columnar readers: infer Arrow types first, skip per-key and per-row steps, one _kind per column (refs #312) #321 is lost. No other file imports it, and the mergeddatabase.pyreads columns through the frames code.src/arcadedb/_native/database.py: I kept both.main's facets lines in_filland itsfacetsproperty (Expose the facets of a result set in both packages (closes #290) #299, Facet kinds from Python: HIERARCHY, DISTINCT, STATS, AFTER and facet.searchValues (closes #291) #301) stay, and the release line'sself._note_vectors(reply)before the return and its_note_vectorsmethod stay. The two are independent, and__init__and_lazyof the merged class set both groups of attributes. Only the native CI exercises this file.tests/conftest.py: the release side of the native-only block (the comment oftest_native_columns_reference.pysays what the file compares now, plus the two new native-only files), and both entries inWINDOWS_NOT_APPLICABLE,test_dev_engine_pin.pyfrommainandtest_parity_gate_awk.pyfrom the release line.tests/test_conftest_windows_table.pyofmainchecks that table against the parity runner and passes.tests/test_native_columns_reference.py(added on both sides): the release side whole.main's file tested the deleted module, and the release line's file tests the drain of a native result set against row-by-row iteration.A second commit fixes one thing the merge leaves behind: the source comment of
docs/modules/ROOT/pages/how-to/dataframes-and-columns.adoc, a page ofmain, named_native/columns.py. It names_columnar.pyand_native/colframes.pynow. The comment does not render, and no statement of the page changed.What I verified, and what I did not
Static gates, run the way CI runs them on the merged tree: pre-commit (black 26.10.1, isort 9.0.2, shfmt, actionlint, the old-name check) on the 73 files the merge changes, Bandit 1.9.4 on
src testsat low/low and onexamplesat medium/high,compile()of all tracked.pyfiles on Python 3.11 and 3.12, and pyflakes compared with both parents (no message that neither parent has). All clean.The full JVM test suite of the merged tree, 3295 tests: 3293 passed, 0 skipped, 2 failed, and
scripts/check_test_skips.pyreports OK. I could not build a wheel from this branch, so the environment is the JVM wheel of the lastmainrun (run 42658639612, engine 26.12.1 snapshot) with the bridge jar of the release line's wheel (the merge changes nothing undersrc/javaagainst the release line) and everysrc/arcadedb/**/*.pyof this branch put in front of it withPYTHONPATH; the run printed the overlay path asarcadedb.__file__. The two failures aretests/test_jvm_cds.py: the archive is added by the release line's build scripts and the oldmainwheel has none. Against the post2 wheel (run 42789420384) with the same overlay, that file passes, together withtests/test_column_frames.pyandtests/test_conftest_windows_table.py(74 passed). After the docs fix, the docs page tests andtests/test_dataframe_columns_behavior.pypassed again (53 tests).I did not run the tests that need the native library (
tests/test_native_*.py, so alsotests/test_native_columns_reference.py, and the parity suite), the Windows and macOS legs, or the Antora build of the docs. TheDocsworkflow of this pull request builds the site. The full-matrix runs oftest-jvm.yml,test-native.ymlandtest-python-examples.ymlon this branch are the decisive check. They wait for the release runs of26.11.1.post2to finish, so that they do not compete with them for the runner cap that the organisation shares with upstream, and I will link them in a comment.Related issues
No closing keyword: the merge touches #295, #340, #347 and #348, which belong to the release and stay open until it is done.
Performance changes
None in this pull request. The merge changes no code beyond the conflict resolutions, and the timed results are in the pull requests listed above.
Docs
The one page this merge leaves stale is fixed (see above). The Antora pages and
README.mdotherwise come from the two lines unchanged.Checklist
uvx pre-commit run --files <changed files>passes🤖 Generated with Claude Code