Skip to content

Skip duplicate tDataset hashes and metadata on concurrent bulk inserts - #319

Merged
knoxsp merged 1 commit into
masterfrom
fix/dataset-insert-ignore-race
Oct 23, 2026
Merged

knoxsp merged 1 commit into
masterfrom
fix/dataset-insert-ignore-race

Conversation

@knoxsp

@knoxsp knoxsp commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Replaces the dataset part of #301, without removing sqlite support (sqlite stays a supported backend, including for the test suite).

Problem

Two concurrent imports (e.g. add_network) that produce datasets with identical content, and so the same hash, can both miss the existing-data lookup in _bulk_insert_data before either has committed, then both insert it. The second plain INSERT fails with an IntegrityError on tDataset.hash and aborts the whole batch with an opaque error.

Fix

  • _bulk_insert_data inserts the new datasets with db.bulk_insert_ignore, so a duplicate hash is skipped. The hash is re-queried afterwards regardless of which import won.
  • _insert_metadata does the same for metadata. Without this the failure just moves one step on: the losing import then inserts metadata for a dataset the winner already created and hits the (dataset_id, key) primary key of tMetadata. Fix tDataset unique-hash race condition, drop sqlite support #301 only changed the dataset insert, so it would still have failed there. The hash covers the metadata, so the rows the winner wrote are the ones the loser would have written.
  • Insert by column name (Dataset.__table__, rows filtered to its columns): the rows also carry the metadata dict, and the Dataset model exposes value as value_ref, so passing the ORM model with these rows does not work.
  • Removes the commented-out LOCK TABLES code this replaces.

bulk_insert_ignore already supports mysql, postgresql and sqlite on master, so nothing in the database layer, hydra.ini, session store or test config changes.

Testing

New tests/test_dataset_insert_race.py simulates the race by having the first existing-data lookup report nothing for a dataset that is already stored. It fails on master with IntegrityError: UNIQUE constraint failed: tDataset.hash and passes with this change. Ran on the default sqlite backend with test_data, test_datatypes, test_scenario, test_network and test_groups: 155 passed. test_scenario.py::test_bulk_insert_data needs MongoDB and was not run locally. This was a single-connection simulation, so real concurrent behaviour on MySQL is untested here.

Supersedes #301.

Two concurrent imports that produce datasets with identical content (and so
the same hash) can both miss the existing-data lookup in _bulk_insert_data
before either has committed, then both insert it. The second plain INSERT
failed with an IntegrityError on tDataset.hash and aborted the whole batch.

Insert the datasets with bulk_insert_ignore so a duplicate hash is skipped
(the hash is re-queried afterwards regardless of which import won), and do
the same for the metadata, which would otherwise fail on its (dataset_id,
key) primary key for a dataset the other import already created. Insert by
column name: the rows also carry the metadata dict, and the Dataset model
exposes 'value' as value_ref.

Also drop the commented-out LOCK TABLES code this replaces.
@knoxsp
knoxsp merged commit 928d82f into master Oct 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants