Repository navigation
Native tests: skips come from explicit markers and NotOnNativeYet, never from failure text - #535
Merged
Merged
Conversation
…tOnNativeYet, never from failure text
The report hook turned a failed native test into a skip when its exception text contained one of ~15 generic markers
("assert (None is not None)", "KeyError: 'flush'", "DID NOT RAISE KeyboardInterrupt", "cause is <class 'NoneType'>", "getClass", ...)
or when its name was on a list. Those texts are also what genuine bugs look like, and the hook hid three real regressions in PR #522.
Now only an explicit signal skips a test: NotOnNativeYet raised by the native code (NOTYET), or an entry in NATIVE_NA_JVM_TESTS keyed
by file::[Class::]function with a reason (NA-JVM, skipped at collection). Of the 154 items the text and name matching hid, 73 now run
and pass on native (60 resultset read errors, 7 graph batch, 6 sparse vector), 81 are listed as NA-JVM with a reason, and one real bug
is pinned as xfail(strict): BACKLOG-bindings.md P-04.
Also found and fixed: to_java_int_array and to_java_byte_array had no native branch and raised JVMNotRunning on the native wheel
(to_java_float_array has one); six tests that were listed as NA-JVM by name now run on native.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898
marked this pull request as ready for review
October 25, 2026 09:12
tae898
added a commit
to ArcadeData/arcadedb-python
that referenced
this pull request
Oct 25, 2026
…ver from failure text Port of humemai/arcadedb-embedded-python#535, adapted to this repository's conftest. The native build used to turn a failing test into an NA-JVM or NOTYET skip when its failure text matched a list of strings ("JVMNotRunning", "assert (None is not None)", "KeyError: 'flush'"), which is also what a real bug looks like. Now: - NATIVE_NA_JVM_TESTS is keyed <file>::[<Class>::]<test> and lists only tests that reach into the JVM on purpose, with the reason. Rows were added for the tests this repository has that the upstream table did not know (temporal writers, XML limits, to_java_rid, db._java_db patching). - pytest_runtest_makereport reports NOTYET only for NotOnNativeYet, raised by the native code. - to_java_int_array and to_java_byte_array return NumPy arrays on the native build, so the sparse vector tests no longer import jpype.types. - tests/README.md says how a test may be skipped on the native build. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The native test hook in
bindings/python/tests/conftest.pyturned a failed test into a skip when the exception text contained any of about 15 generic markers (assert (None is not None),KeyError: 'flush',DID NOT RAISE KeyboardInterrupt,cause is <class 'NoneType'>,getClass,bridge jar, ...) or when the test name was on a list. Those texts are also what genuine bugs look like, and the hook hid three real regressions in PR #522 today. This PR removes the text and name matching. A native test is now skipped only by an explicit signal:NotOnNativeYetraised by the native code (reported asNOTYET), or an entry inNATIVE_NA_JVM_TESTSwith a reason (skipped at collection, reported asNA-JVM: <why>).Inventory
I ran the full bindings suite (72 files; the 9
NATIVE_NOT_APPLICABLEfiles are not collected) withARCADEDB_BACKEND=nativeagainst thenative-lib/nnextimage (engine 26.10.2), with the text matching switched to log-only so every failure showed its real traceback. Before this PR the hook was hiding 154 results (151 by failure text, 3 by test name), 19 more were the explicitNotOnNativeYetskips, and 10 more were listed as NA-JVM by name. After:NATIVE_NA_JVM_TESTSwith a reason, skipped at collection (NA-JVM: ...)NotOnNativeYet(NOTYET: ...)xfail(strict=True)naming BACKLOG-bindings.md P-04, red when fixedFinal native run on this branch: 998 passed, 85 NA-JVM, 19 NOTYET, 1 xfail (strict), 1 module-level
importorskipskip (test_dataframe_ingest.py, polars is not in my venv; CI installs it), 0 failed.Real bugs found
GraphBatch.builder(db)(native-spike/.../NativeExtras.java:243), which takes no part in the engine's single-batch guard (LocalDatabase.batchStarted, Two concurrent GraphBatch instances on the same database silently lose edges, and nothing prevents them ArcadeData/arcadedb#5666); the JPype path goes throughdb.batch()and is guarded. Native lets a seconddb.graph_batch()open while one is open, and two concurrent batches silently lose edges. It was hidden behind the text-matched skip oftest_graph_batch_close_refused_in_the_callers_transaction_can_be_retried. The newtest_a_second_graph_batch_is_refused_while_one_is_openpins it (xfail strict on native); the old test skips the guard assertion on native only. The fix is a one-line Java glue change plus a native-image rebuild, so it is not in this PR.to_java_int_arrayandto_java_byte_arrayhad no native branch and raisedJVMNotRunningon the native wheel, whileto_java_float_arrayreturns a float32 ndarray there. They now return an int32 and an int8 ndarray with the same range checks. The sparse-vector SQL tests usedjtypes.JArray(jtypes.JInt)directly; they callarcadedb.to_java_int_arraynow, so the sparseminScore/ index-type / neighbours / INT8 rescoring tests run on native (they were hidden skips).test_resultset_read_errors.py(60 tests asserted__cause__is ajpype.JException; native has no Java exception object, so it checks the message prefix and that the engine exception class is named in it) and the_is_the_engines_refusalhelper intest_graph_batch.py(7 tests).Reclassified tests
(a) NA-JVM, now listed explicitly with a reason (85 tests, 67 entries)
test_conversion_fixes.py::test_java_instants_convert_without_losing_microseconds(6): builds a java.* object through JPypetest_docs_examples.py::test_docs_installation_jvm_examples(1): documents start_jvm() and jvm_kwargstest_graph_api.py::test_document_wrap_static_method(1): wraps the Java object behind a result (_java_result)test_graph_batch.py::test_graph_batch_create_vertices_keyboard_interrupt_rolls_back(1): patches GraphBatch._create_vertices_json_bulk, the JPype path's bulk calltest_graph_batch.py::test_graph_batch_outside_the_callers_transactions(1): sets GraphBatch.TEST_BEFORE_VERTEX_COMMIT_HOOK, a Java static, through JPypetest_graph_batch.py::test_graph_batch_create_vertices_refuses_before_any_commit_attempt(1): sets GraphBatch.TEST_BEFORE_VERTEX_COMMIT_HOOK, a Java static, through JPypetest_graph_batch.py::test_graph_batch_leaves_the_callers_wal_setting_alone(1): reads the Java database behind the wrapper (db._java_db)test_legacy_temporal_implementations.py::test_the_engine_hands_back_the_configured_class_holding_the_utc_instant(2): reads the Java database behind the wrapper (db._java_db)test_parameter_binding.py::test_each_parameter_shape_reaches_one_java_overload(1): inspects the Java Database overloads through JPypetest_parameter_binding.py::test_async_parameters_reach_one_java_overload(1): inspects the Java DatabaseAsyncExecutor overloads through JPypetest_parameter_binding.py::TestScalarParametersTakeTheShortPath::test_a_java_array_value_is_not_converted_and_binds_as_before(1): builds a Java array through JPype (jpype.JArray)test_parameter_binding.py::TestScalarParametersTakeTheShortPath::test_non_scalars_keep_the_general_path(1): counts JPype parameter conversionstest_parameter_binding.py::TestScalarParametersTakeTheShortPath::test_a_subclass_of_a_scalar_is_not_assumed_to_be_one(1): counts JPype parameter conversionstest_percall_calls.py::test_float_array_conversion_gives_a_float_array(5): asserts the converted value is a Java float[] (getClass() == "[F")test_resultset.py::test_to_dict_one_crossing_matches_the_per_property_path(1): needs the bridge jar's RowAccess classtest_resultset.py::TestSmallResultsCostOneBridgeCall::test_to_list_makes_one_call_for_a_short_result(1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)test_resultset.py::TestSmallResultsCostOneBridgeCall::test_to_list_of_an_empty_result_makes_one_call(1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)test_resultset.py::TestSmallResultsCostOneBridgeCall::test_to_json_list_makes_one_call_for_a_short_result(1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)test_resultset.py::TestSmallResultsCostOneBridgeCall::test_a_full_batch_still_ends_on_the_next_call(1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)test_resultset.py::TestSmallResultsCostOneBridgeCall::test_a_drained_set_reads_as_empty_and_is_not_an_error(1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)test_resultset.py::TestSmallResultsCostOneBridgeCall::test_python_does_not_cross_the_bridge_to_close_a_drained_set(1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)test_resultset.py::TestSmallResultsCostOneBridgeCall::test_the_bridge_closes_what_it_drains_and_only_that(2): counts or intercepts the JPype bridge crossings (an internal of the JPype path)test_run_in_transaction_retry.py::test_every_need_retry_exception_is_retried(4): throws Java exceptions (com.arcadedb.exception.*) from the testtest_run_in_transaction_retry.py::test_the_budget_is_spent_before_the_conflict_reaches_the_caller(1): throws Java exceptions (com.arcadedb.exception.*) from the testtest_run_in_transaction_retry.py::test_the_wait_the_refusing_side_asked_for_is_honored(1): throws Java exceptions (com.arcadedb.exception.*) from the testtest_run_in_transaction_retry.py::test_the_wait_is_capped_by_retry_after_max_wait(1): throws Java exceptions (com.arcadedb.exception.*) from the testtest_run_in_transaction_retry.py::test_a_block_that_committed_part_of_its_work_is_not_replayed(1): throws a Java ConcurrentModificationException from the test (native: test_run_in_transaction_partial_commit.py)test_run_in_transaction_retry.py::test_contended_commits_spend_their_budget_before_failing(1): reads the Java database behind the wrapper (db._java_db)test_server_wire_protocols.py::test_wire_listeners_stay_on_loopback_by_default(1): reads the server's GlobalConfiguration through the JVMtest_server_wire_protocols.py::test_host_applies_to_every_wire_listener(1): reads the server's GlobalConfiguration through the JVMtest_server_wire_protocols.py::test_a_protocol_host_overrides_the_server_host_for_that_protocol_only(1): reads the server's GlobalConfiguration through the JVMtest_server_wire_protocols.py::test_the_host_reaches_exactly_the_listener_bind_settings(1): reads the engine's GlobalConfiguration through the JVMtest_sigint.py::test_ctrl_c_raises_keyboard_interrupt_and_runs_cleanup(1): a Java call blocked in JPype's SIGINT handler (tests: test_ctrl_c_raises_keyboard_interrupt_and_runs_cleanup[java] is flaky (JPype aborts on SIGINT inside Thread.sleep) #179)test_sigint.py::test_interrupt_true_keeps_the_jvm_handling_sigint(1): start_jvm(interrupt=True): the JVM handles SIGINTtest_sql_constraints_and_maintenance.py::test_backup_check_and_rebuild_index_statements(1): sets GlobalConfiguration.SERVER_BACKUP_DIRECTORY through the JVMtest_transaction_config.py::test_set_wal_flush_is_database_wide(1): reads the Java database behind the wrapper (db._java_db)test_type_conversion.py::test_offset_datetime_conversion(1): builds a java.* object through JPypetest_type_conversion.py::test_scalar_list_crosses_as_one_array_with_the_same_types(1): asserts the converted value is a java.util.ArrayListtest_type_conversion.py::TestPrimitiveArrayFormats::test_int_array(1): builds a Java array through JPype (jpype.JArray)test_type_conversion.py::TestPrimitiveArrayFormats::test_long_array(1): builds a Java array through JPype (jpype.JArray)test_typed_rows.py::test_java_values_sql_cannot_produce(1): builds a java.* object through JPypetest_typed_rows.py::test_value_python_cannot_hold_raises_as_before(1): builds a java.* object through JPypetest_typed_rows.py::test_object_array_raises_as_before(1): builds a Java array through JPype (jpype.JArray)test_typed_rows.py::test_iter_dicts_reads_one_row_at_a_time(1): builds a Java InternalResultSettest_typed_rows.py::test_vertex_edges_match_the_engine_iteration(5): the oracle is the Java Vertex.getEdges iterationtest_typed_rows.py::test_vector_search_hits_match_the_per_hit_path(1): the oracle is the Java vector index (findNeighborsFromVector)test_typed_rows.py::test_a_batch_of_wide_rows_ends_at_a_few_megabytes(1): calls the bridge's typed_rows.nextRows on the Java result settest_typed_rows.py::test_the_batch_says_whether_the_result_set_is_drained(1): calls the bridge's typed_rows.nextRows on the Java result settest_vector.py::TestLSMVectorIndex::test_create_vector_index_with_pq_params(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector.py::TestLSMVectorIndex::test_create_vector_index_with_custom_id_property(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_approximate_product(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_approximate_typeindex(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_looks_up_all_hits_in_one_bridge_call(1): counts or intercepts the JPype bridge crossings (an internal of the JPype path)test_vector.py::TestLSMVectorIndex::test_lsm_vector_search_uses_database_rid_conversion(1): intercepts Database.to_java_rid, the JPype path's RID conversiontest_vector_params_verification.py::TestVectorParams::test_quantization_param(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector_params_verification.py::TestVectorParams::test_store_vectors_in_graph_param(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector_params_verification.py::TestVectorParams::test_encoding_param(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector_params_verification.py::TestVectorParams::test_add_hierarchy_param(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector_params_verification.py::TestVectorParams::test_per_index_cache_params(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector_params_verification.py::TestVectorParams::test_quantization_none(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector_params_verification.py::TestVectorParams::test_quantization_binary(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector_params_verification.py::TestVectorParams::test_quantization_product(1): reads the Java index behind the wrapper (_java_index: getClass, metadata)test_vector_params_verification.py::TestVectorParams::test_wrapper_default_matches_engine_default(1): reads the engine default through java.lang.String and GlobalConfigurationtest_vector_params_verification.py::TestVectorParams::test_omitted_max_connections_reaches_the_index(1): reads the engine default through java.lang.String and GlobalConfigurationtest_vector_params_verification.py::TestVectorParams::test_jvm_heap_check(1): reads java.lang.Runtime through JPypetest_vector_sql.py::TestVectorSearchOptionsSql::test_where_map_and_condition(1): picks the expected behaviour by com.arcadedb.Constants.getRawVersion() through JPypetest_vector_sql.py::TestVectorSearchOptionsSql::test_where_combines_with_filter(1): picks the expected behaviour by com.arcadedb.Constants.getRawVersion() through JPype(b) NOTYET, unchanged (19 tests)
test_light_edge_and_view_known_issues.py::test_one_hop_count_over_light_edges_an_older_version_loadedtest_light_edge_and_view_known_issues.py::test_other_ways_to_count_light_edges_an_older_version_loadedtest_percall_calls.py::test_a_lone_null_or_lone_java_array_keeps_the_plain_pathtest_percall_calls.py::test_named_scalars_bind_as_the_plain_path_binds_themtest_percall_calls.py::test_other_values_take_the_plain_pathtest_percall_calls.py::test_positional_scalars_bind_as_the_plain_path_binds_themtest_vector.py::TestLSMVectorIndex::test_lsm_index_statstest_vector.py::TestLSMVectorIndex::test_lsm_vector_search_approximate_returns_ktest_vector.py::test_warm_up_loads_the_persisted_graph_before_the_first_searchtest_vector_sql.py::TestVectorSQL::test_create_index_with_native_int8_encoding_sqltest_vector_sql.py::TestVectorSQL::test_create_index_with_rich_metadata_sqltest_vector_sql.py::TestVectorSearchOptionsSql::test_add_hierarchy_defaults_differ_between_sql_and_pythontest_vector_sql.py::TestVectorSearchOptionsSql::test_graph_build_settings_reach_the_index(c) hidden before, run and pass on native now (73 tests; plus the 6 to_java_* tests that were NA-JVM by name)
test_graph_batch.py::test_graph_batch_create_vertices_leaves_the_callers_transaction(3)test_graph_batch.py::test_graph_batch_flush_and_close_leave_the_callers_transaction(3)test_graph_batch.py::test_graph_batch_close_refused_in_the_callers_transaction_can_be_retried(1)test_resultset_read_errors.py::test_the_engine_raises_these_errors_while_rows_are_read(1)test_resultset_read_errors.py::test_every_reader_raises_arcadedb_error(52)test_resultset_read_errors.py::test_first_raises_arcadedb_error(2)test_resultset_read_errors.py::test_one_raises_the_engine_error_not_value_error(4)test_resultset_read_errors.py::test_a_command_result_raises_arcadedb_error(1)test_sparse_quantization_compact.py::test_sparse_weight_precision_and_compact(2)test_sparse_quantization_compact.py::test_int8_scores_are_rescored_to_the_exact_dot_product(1)test_vector_sql.py::test_sparse_vector_neighbors_sql(1)test_vector_sql.py::test_min_score_drops_low_scores(1)test_vector_sql.py::test_index_type_enum_creates_a_sparse_index(1)test_conversion_fixes.py::test_to_java_int_array_refuses_values_beyond_32_bits_in_an_array,test_to_java_int_array_keeps_values_that_fit;test_vector.py::test_to_java_byte_array_accepts_bytes_like_fast_paths,test_to_java_int_array,test_to_java_int_array_accepts_numpy_directly,test_to_java_int_array_round_trips_through_a_sparse_vectorWhat stays the same
NATIVE_NOT_APPLICABLEandWINDOWS_NOT_APPLICABLEare still literal dicts (parity_wheel.pyreads them withast.literal_eval).parity_wheel.py+ the awk gate) fails on any failure, error, crash or timeout and does not fail on NOTYET / NA-JVM / other skips, which it counts by the message prefix. The skips this PR produces keep theNA-JVM:andNOTYET:prefixes, so the counts stay comparable.scripts/check_test_skips.py(main CI, JPype backend) is unaffected: the registry is read only whenARCADEDB_BACKEND=native, and the JPype backend gets no new skips.otherskip in the parity table (junitpytest.xfail) and fails the run with XPASS the moment the bug is fixed.Checked
test_vector.py,test_conversion_fixes.py,test_type_conversion.py: 236 passed.perf/native-tx-crossings, which contains Native: run_in_transaction refuses to replay a block that already committed part of its work (P-03) #527's commit) with this commit cherry-picked (clean, theconftest.pyhunks auto-merge): 1011 passed, 85 NA-JVM, 19 NOTYET, 1 xfail, 0 failed.test_interrupt_hygiene.py16 passed and 1 explicitskipif,test_run_in_transaction_partial_commit.py3 passed,test_native_tx_crossings.py11 passed.fix/p03-native-run-in-tx-partial-commit) with this commit cherry-picked: 1001 passed, 0 failed;test_interrupt_hygiene.py17 passed,test_run_in_transaction_partial_commit.py3 passed.Not covered (named, not hidden)
The 85 NA-JVM tests inspect Java objects directly, so a few of them guard behaviour that native also has and that has no native test yet: the wire-listener loopback default (
test_wire_listeners_stay_on_loopback_by_defaultreadsGlobalConfigurationthrough JPype), the GraphBatch KeyboardInterrupt rollback, the 12 vector-index parameter checks that read_java_indexmetadata, and the run_in_transaction retry budget on contended commits. Native versions of those need a config / metadata reader on the native side; I did not add them here.The native image used is
nnext(engine 26.10.2), not a fresh build of this branch, and the native-preview workflow itself was not run.🤖 Generated with Claude Code