Skip to content

Native tests: skips come from explicit markers and NotOnNativeYet, never from failure text - #535

Merged
tae898 merged 1 commit into
native-previewfrom
fix/native-explicit-skip-markers
Oct 25, 2026
Merged

tae898 merged 1 commit into
native-previewfrom
fix/native-explicit-skip-markers

Conversation

@tae898

@tae898 tae898 commented Oct 25, 2026

Copy link
Copy Markdown

The native test hook in bindings/python/tests/conftest.py turned a failed test into a skip when the exception text contained any of about 15 generic markers (assert (None is not None), KeyError: 'flush', DID NOT RAISE KeyboardInterrupt, cause is <class 'NoneType'>, getClass, bridge jar, ...) or when the test name was on a list. Those texts are also what genuine bugs look like, and the hook hid three real regressions in PR #522 today. This PR removes the text and name matching. A native test is now skipped only by an explicit signal: NotOnNativeYet raised by the native code (reported as NOTYET), or an entry in NATIVE_NA_JVM_TESTS with a reason (skipped at collection, reported as NA-JVM: <why>).

Inventory

I ran the full bindings suite (72 files; the 9 NATIVE_NOT_APPLICABLE files are not collected) with ARCADEDB_BACKEND=native against the native-lib/nnext image (engine 26.10.2), with the text matching switched to log-only so every failure showed its real traceback. Before this PR the hook was hiding 154 results (151 by failure text, 3 by test name), 19 more were the explicit NotOnNativeYet skips, and 10 more were listed as NA-JVM by name. After:

category tests what happens now
(a) needs the JVM by design 85 listed in NATIVE_NA_JVM_TESTS with a reason, skipped at collection (NA-JVM: ...)
(b) feature missing on purpose 19 unchanged: the native code raises NotOnNativeYet (NOTYET: ...)
(c) was hidden, runs and passes on native 79 no skip: 73 that the hook hid, plus 6 that were listed NA-JVM by name
(d) real native bug 1 xfail(strict=True) naming BACKLOG-bindings.md P-04, red when fixed
(d) real native bug, fixed here 1 P-05, the 6 tests above that were NA-JVM by name run because of it

Final native run on this branch: 998 passed, 85 NA-JVM, 19 NOTYET, 1 xfail (strict), 1 module-level importorskip skip (test_dataframe_ingest.py, polars is not in my venv; CI installs it), 0 failed.

Real bugs found

  • P-04 (native, open): the native glue opens its GraphBatch with GraphBatch.builder(db) (native-spike/.../NativeExtras.java:243), which takes no part in the engine's single-batch guard (LocalDatabase.batchStarted, Two concurrent GraphBatch instances on the same database silently lose edges, and nothing prevents them ArcadeData/arcadedb#5666); the JPype path goes through db.batch() and is guarded. Native lets a second db.graph_batch() open while one is open, and two concurrent batches silently lose edges. It was hidden behind the text-matched skip of test_graph_batch_close_refused_in_the_callers_transaction_can_be_retried. The new test_a_second_graph_batch_is_refused_while_one_is_open pins it (xfail strict on native); the old test skips the guard assertion on native only. The fix is a one-line Java glue change plus a native-image rebuild, so it is not in this PR.
  • P-05 (native, fixed here): to_java_int_array and to_java_byte_array had no native branch and raised JVMNotRunning on the native wheel, while to_java_float_array returns a float32 ndarray there. They now return an int32 and an int8 ndarray with the same range checks. The sparse-vector SQL tests used jtypes.JArray(jtypes.JInt) directly; they call arcadedb.to_java_int_array now, so the sparse minScore / index-type / neighbours / INT8 rescoring tests run on native (they were hidden skips).
  • Not bugs, but tests that over-asserted a JVM detail and now run on both backends: test_resultset_read_errors.py (60 tests asserted __cause__ is a jpype.JException; native has no Java exception object, so it checks the message prefix and that the engine exception class is named in it) and the _is_the_engines_refusal helper in test_graph_batch.py (7 tests).

Reclassified tests

(a) NA-JVM, now listed explicitly with a reason (85 tests, 67 entries)
  • test_conversion_fixes.py::test_java_instants_convert_without_losing_microseconds (6): builds a java.* object through JPype
  • test_docs_examples.py::test_docs_installation_jvm_examples (1): documents start_jvm() and jvm_kwargs
  • test_graph_api.py::test_document_wrap_static_method (1): wraps the Java object behind a result (_java_result)
  • test_graph_batch.py::test_graph_batch_create_vertices_keyboard_interrupt_rolls_back (1): patches GraphBatch._create_vertices_json_bulk, the JPype path's bulk call
  • test_graph_batch.py::test_graph_batch_outside_the_callers_transactions (1): sets GraphBatch.TEST_BEFORE_VERTEX_COMMIT_HOOK, a Java static, through JPype
  • test_graph_batch.py::test_graph_batch_create_vertices_refuses_before_any_commit_attempt (1): sets GraphBatch.TEST_BEFORE_VERTEX_COMMIT_HOOK, a Java static, through JPype
  • test_graph_batch.py::test_graph_batch_leaves_the_callers_wal_setting_alone (1): reads the Java database behind the wrapper (db._java_db)
  • test_legacy_temporal_implementations.py::test_the_engine_hands_back_the_configured_class_holding_the_utc_instant (2): reads the Java database behind the wrapper (db._java_db)
  • test_parameter_binding.py::test_each_parameter_shape_reaches_one_java_overload (1): inspects the Java Database overloads through JPype
  • test_parameter_binding.py::test_async_parameters_reach_one_java_overload (1): inspects the Java DatabaseAsyncExecutor overloads through JPype
  • test_parameter_binding.py::TestScalarParametersTakeTheShortPath::test_a_java_array_value_is_not_converted_and_binds_as_before (1): builds a Java array through JPype (jpype.JArray)
  • test_parameter_binding.py::TestScalarParametersTakeTheShortPath::test_non_scalars_keep_the_general_path (1): counts JPype parameter conversions
  • test_parameter_binding.py::TestScalarParametersTakeTheShortPath::test_a_subclass_of_a_scalar_is_not_assumed_to_be_one (1): counts JPype parameter conversions
  • test_percall_calls.py::test_float_array_conversion_gives_a_float_array (5): asserts the converted value is a Java float[] (getClass() == "[F")
  • test_resultset.py::test_to_dict_one_crossing_matches_the_per_property_path (1): needs the bridge jar's RowAccess class
  • test_resultset.py::TestSmallResultsCostOneBridgeCall::test_to_list_makes_one_call_for_a_short_result (1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)
  • test_resultset.py::TestSmallResultsCostOneBridgeCall::test_to_list_of_an_empty_result_makes_one_call (1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)
  • test_resultset.py::TestSmallResultsCostOneBridgeCall::test_to_json_list_makes_one_call_for_a_short_result (1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)
  • test_resultset.py::TestSmallResultsCostOneBridgeCall::test_a_full_batch_still_ends_on_the_next_call (1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)
  • test_resultset.py::TestSmallResultsCostOneBridgeCall::test_a_drained_set_reads_as_empty_and_is_not_an_error (1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)
  • test_resultset.py::TestSmallResultsCostOneBridgeCall::test_python_does_not_cross_the_bridge_to_close_a_drained_set (1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)
  • test_resultset.py::TestSmallResultsCostOneBridgeCall::test_the_bridge_closes_what_it_drains_and_only_that (2): counts or intercepts the JPype bridge crossings (an internal of the JPype path)
  • test_run_in_transaction_retry.py::test_every_need_retry_exception_is_retried (4): throws Java exceptions (com.arcadedb.exception.*) from the test
  • test_run_in_transaction_retry.py::test_the_budget_is_spent_before_the_conflict_reaches_the_caller (1): throws Java exceptions (com.arcadedb.exception.*) from the test
  • test_run_in_transaction_retry.py::test_the_wait_the_refusing_side_asked_for_is_honored (1): throws Java exceptions (com.arcadedb.exception.*) from the test
  • test_run_in_transaction_retry.py::test_the_wait_is_capped_by_retry_after_max_wait (1): throws Java exceptions (com.arcadedb.exception.*) from the test
  • test_run_in_transaction_retry.py::test_a_block_that_committed_part_of_its_work_is_not_replayed (1): throws a Java ConcurrentModificationException from the test (native: test_run_in_transaction_partial_commit.py)
  • test_run_in_transaction_retry.py::test_contended_commits_spend_their_budget_before_failing (1): reads the Java database behind the wrapper (db._java_db)
  • test_server_wire_protocols.py::test_wire_listeners_stay_on_loopback_by_default (1): reads the server's GlobalConfiguration through the JVM
  • test_server_wire_protocols.py::test_host_applies_to_every_wire_listener (1): reads the server's GlobalConfiguration through the JVM
  • test_server_wire_protocols.py::test_a_protocol_host_overrides_the_server_host_for_that_protocol_only (1): reads the server's GlobalConfiguration through the JVM
  • test_server_wire_protocols.py::test_the_host_reaches_exactly_the_listener_bind_settings (1): reads the engine's GlobalConfiguration through the JVM
  • test_sigint.py::test_ctrl_c_raises_keyboard_interrupt_and_runs_cleanup (1): a Java call blocked in JPype's SIGINT handler (tests: test_ctrl_c_raises_keyboard_interrupt_and_runs_cleanup[java] is flaky (JPype aborts on SIGINT inside Thread.sleep) #179)
  • test_sigint.py::test_interrupt_true_keeps_the_jvm_handling_sigint (1): start_jvm(interrupt=True): the JVM handles SIGINT
  • test_sql_constraints_and_maintenance.py::test_backup_check_and_rebuild_index_statements (1): sets GlobalConfiguration.SERVER_BACKUP_DIRECTORY through the JVM
  • test_transaction_config.py::test_set_wal_flush_is_database_wide (1): reads the Java database behind the wrapper (db._java_db)
  • test_type_conversion.py::test_offset_datetime_conversion (1): builds a java.* object through JPype
  • test_type_conversion.py::test_scalar_list_crosses_as_one_array_with_the_same_types (1): asserts the converted value is a java.util.ArrayList
  • test_type_conversion.py::TestPrimitiveArrayFormats::test_int_array (1): builds a Java array through JPype (jpype.JArray)
  • test_type_conversion.py::TestPrimitiveArrayFormats::test_long_array (1): builds a Java array through JPype (jpype.JArray)
  • test_typed_rows.py::test_java_values_sql_cannot_produce (1): builds a java.* object through JPype
  • test_typed_rows.py::test_value_python_cannot_hold_raises_as_before (1): builds a java.* object through JPype
  • test_typed_rows.py::test_object_array_raises_as_before (1): builds a Java array through JPype (jpype.JArray)
  • test_typed_rows.py::test_iter_dicts_reads_one_row_at_a_time (1): builds a Java InternalResultSet
  • test_typed_rows.py::test_vertex_edges_match_the_engine_iteration (5): the oracle is the Java Vertex.getEdges iteration
  • test_typed_rows.py::test_vector_search_hits_match_the_per_hit_path (1): the oracle is the Java vector index (findNeighborsFromVector)
  • test_typed_rows.py::test_a_batch_of_wide_rows_ends_at_a_few_megabytes (1): calls the bridge's typed_rows.nextRows on the Java result set
  • test_typed_rows.py::test_the_batch_says_whether_the_result_set_is_drained (1): calls the bridge's typed_rows.nextRows on the Java result set
  • test_vector.py::TestLSMVectorIndex::test_create_vector_index_with_pq_params (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector.py::TestLSMVectorIndex::test_create_vector_index_with_custom_id_property (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_approximate_product (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_approximate_typeindex (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_looks_up_all_hits_in_one_bridge_call (1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)
  • test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_uses_database_rid_conversion (1): intercepts Database.to_java_rid, the JPype path's RID conversion
  • test_vector_params_verification.py::TestVectorParams::test_quantization_param (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector_params_verification.py::TestVectorParams::test_store_vectors_in_graph_param (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector_params_verification.py::TestVectorParams::test_encoding_param (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector_params_verification.py::TestVectorParams::test_add_hierarchy_param (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector_params_verification.py::TestVectorParams::test_per_index_cache_params (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector_params_verification.py::TestVectorParams::test_quantization_none (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector_params_verification.py::TestVectorParams::test_quantization_binary (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector_params_verification.py::TestVectorParams::test_quantization_product (1): reads the Java index behind the wrapper (_java_index: getClass, metadata)
  • test_vector_params_verification.py::TestVectorParams::test_wrapper_default_matches_engine_default (1): reads the engine default through java.lang.String and GlobalConfiguration
  • test_vector_params_verification.py::TestVectorParams::test_omitted_max_connections_reaches_the_index (1): reads the engine default through java.lang.String and GlobalConfiguration
  • test_vector_params_verification.py::TestVectorParams::test_jvm_heap_check (1): reads java.lang.Runtime through JPype
  • test_vector_sql.py::TestVectorSearchOptionsSql::test_where_map_and_condition (1): picks the expected behaviour by com.arcadedb.Constants.getRawVersion() through JPype
  • test_vector_sql.py::TestVectorSearchOptionsSql::test_where_combines_with_filter (1): picks the expected behaviour by com.arcadedb.Constants.getRawVersion() through JPype
(b) NOTYET, unchanged (19 tests)
  • test_light_edge_and_view_known_issues.py::test_one_hop_count_over_light_edges_an_older_version_loaded
  • test_light_edge_and_view_known_issues.py::test_other_ways_to_count_light_edges_an_older_version_loaded
  • test_percall_calls.py::test_a_lone_null_or_lone_java_array_keeps_the_plain_path
  • test_percall_calls.py::test_named_scalars_bind_as_the_plain_path_binds_them
  • test_percall_calls.py::test_other_values_take_the_plain_path
  • test_percall_calls.py::test_positional_scalars_bind_as_the_plain_path_binds_them
  • test_vector.py::TestLSMVectorIndex::test_lsm_index_stats
  • test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_approximate_returns_k
  • test_vector.py::test_warm_up_loads_the_persisted_graph_before_the_first_search
  • test_vector_sql.py::TestVectorSQL::test_create_index_with_native_int8_encoding_sql
  • test_vector_sql.py::TestVectorSQL::test_create_index_with_rich_metadata_sql
  • test_vector_sql.py::TestVectorSearchOptionsSql::test_add_hierarchy_defaults_differ_between_sql_and_python
  • test_vector_sql.py::TestVectorSearchOptionsSql::test_graph_build_settings_reach_the_index
(c) hidden before, run and pass on native now (73 tests; plus the 6 to_java_* tests that were NA-JVM by name)
  • test_graph_batch.py::test_graph_batch_create_vertices_leaves_the_callers_transaction (3)
  • test_graph_batch.py::test_graph_batch_flush_and_close_leave_the_callers_transaction (3)
  • test_graph_batch.py::test_graph_batch_close_refused_in_the_callers_transaction_can_be_retried (1)
  • test_resultset_read_errors.py::test_the_engine_raises_these_errors_while_rows_are_read (1)
  • test_resultset_read_errors.py::test_every_reader_raises_arcadedb_error (52)
  • test_resultset_read_errors.py::test_first_raises_arcadedb_error (2)
  • test_resultset_read_errors.py::test_one_raises_the_engine_error_not_value_error (4)
  • test_resultset_read_errors.py::test_a_command_result_raises_arcadedb_error (1)
  • test_sparse_quantization_compact.py::test_sparse_weight_precision_and_compact (2)
  • test_sparse_quantization_compact.py::test_int8_scores_are_rescored_to_the_exact_dot_product (1)
  • test_vector_sql.py::test_sparse_vector_neighbors_sql (1)
  • test_vector_sql.py::test_min_score_drops_low_scores (1)
  • test_vector_sql.py::test_index_type_enum_creates_a_sparse_index (1)
  • test_conversion_fixes.py::test_to_java_int_array_refuses_values_beyond_32_bits_in_an_array, test_to_java_int_array_keeps_values_that_fit; test_vector.py::test_to_java_byte_array_accepts_bytes_like_fast_paths, test_to_java_int_array, test_to_java_int_array_accepts_numpy_directly, test_to_java_int_array_round_trips_through_a_sparse_vector

What stays the same

  • NATIVE_NOT_APPLICABLE and WINDOWS_NOT_APPLICABLE are still literal dicts (parity_wheel.py reads them with ast.literal_eval).
  • The native-preview CI gate (parity_wheel.py + the awk gate) fails on any failure, error, crash or timeout and does not fail on NOTYET / NA-JVM / other skips, which it counts by the message prefix. The skips this PR produces keep the NA-JVM: and NOTYET: prefixes, so the counts stay comparable. scripts/check_test_skips.py (main CI, JPype backend) is unaffected: the registry is read only when ARCADEDB_BACKEND=native, and the JPype backend gets no new skips.
  • A strict xfail shows up as an other skip in the parity table (junit pytest.xfail) and fails the run with XPASS the moment the bug is fixed.

Checked

Not covered (named, not hidden)

The 85 NA-JVM tests inspect Java objects directly, so a few of them guard behaviour that native also has and that has no native test yet: the wire-listener loopback default (test_wire_listeners_stay_on_loopback_by_default reads GlobalConfiguration through JPype), the GraphBatch KeyboardInterrupt rollback, the 12 vector-index parameter checks that read _java_index metadata, and the run_in_transaction retry budget on contended commits. Native versions of those need a config / metadata reader on the native side; I did not add them here.

The native image used is nnext (engine 26.10.2), not a fresh build of this branch, and the native-preview workflow itself was not run.

🤖 Generated with Claude Code

…tOnNativeYet, never from failure text

The report hook turned a failed native test into a skip when its exception text contained one of ~15 generic markers
("assert (None is not None)", "KeyError: 'flush'", "DID NOT RAISE KeyboardInterrupt", "cause is <class 'NoneType'>", "getClass", ...)
or when its name was on a list. Those texts are also what genuine bugs look like, and the hook hid three real regressions in PR #522.

Now only an explicit signal skips a test: NotOnNativeYet raised by the native code (NOTYET), or an entry in NATIVE_NA_JVM_TESTS keyed
by file::[Class::]function with a reason (NA-JVM, skipped at collection). Of the 154 items the text and name matching hid, 73 now run
and pass on native (60 resultset read errors, 7 graph batch, 6 sparse vector), 81 are listed as NA-JVM with a reason, and one real bug
is pinned as xfail(strict): BACKLOG-bindings.md P-04.

Also found and fixed: to_java_int_array and to_java_byte_array had no native branch and raised JVMNotRunning on the native wheel
(to_java_float_array has one); six tests that were listed as NA-JVM by name now run on native.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 marked this pull request as ready for review October 25, 2026 09:12
tae898 added a commit to ArcadeData/arcadedb-python that referenced this pull request Oct 25, 2026
…ver from failure text

Port of humemai/arcadedb-embedded-python#535, adapted to this repository's conftest. The native
build used to turn a failing test into an NA-JVM or NOTYET skip when its failure text matched a
list of strings ("JVMNotRunning", "assert (None is not None)", "KeyError: 'flush'"), which is also
what a real bug looks like. Now:

- NATIVE_NA_JVM_TESTS is keyed <file>::[<Class>::]<test> and lists only tests that reach into the
  JVM on purpose, with the reason. Rows were added for the tests this repository has that the
  upstream table did not know (temporal writers, XML limits, to_java_rid, db._java_db patching).
- pytest_runtest_makereport reports NOTYET only for NotOnNativeYet, raised by the native code.
- to_java_int_array and to_java_byte_array return NumPy arrays on the native build, so the sparse
  vector tests no longer import jpype.types.
- tests/README.md says how a test may be skipped on the native build.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 7441563 into native-preview Oct 25, 2026
49 of 51 checks passed
@tae898
tae898 deleted the fix/native-explicit-skip-markers branch October 25, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant