Repository navigation
Skip duplicate tDataset hashes and metadata on concurrent bulk inserts - #319
Merged
Merged
Conversation
Two concurrent imports that produce datasets with identical content (and so the same hash) can both miss the existing-data lookup in _bulk_insert_data before either has committed, then both insert it. The second plain INSERT failed with an IntegrityError on tDataset.hash and aborted the whole batch. Insert the datasets with bulk_insert_ignore so a duplicate hash is skipped (the hash is re-queried afterwards regardless of which import won), and do the same for the metadata, which would otherwise fail on its (dataset_id, key) primary key for a dataset the other import already created. Insert by column name: the rows also carry the metadata dict, and the Dataset model exposes 'value' as value_ref. Also drop the commented-out LOCK TABLES code this replaces.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the dataset part of #301, without removing sqlite support (sqlite stays a supported backend, including for the test suite).
Problem
Two concurrent imports (e.g.
add_network) that produce datasets with identical content, and so the same hash, can both miss the existing-data lookup in_bulk_insert_databefore either has committed, then both insert it. The second plain INSERT fails with anIntegrityErrorontDataset.hashand aborts the whole batch with an opaque error.Fix
_bulk_insert_datainserts the new datasets withdb.bulk_insert_ignore, so a duplicate hash is skipped. The hash is re-queried afterwards regardless of which import won._insert_metadatadoes the same for metadata. Without this the failure just moves one step on: the losing import then inserts metadata for a dataset the winner already created and hits the(dataset_id, key)primary key oftMetadata. Fix tDataset unique-hash race condition, drop sqlite support #301 only changed the dataset insert, so it would still have failed there. The hash covers the metadata, so the rows the winner wrote are the ones the loser would have written.Dataset.__table__, rows filtered to its columns): the rows also carry the metadata dict, and theDatasetmodel exposesvalueasvalue_ref, so passing the ORM model with these rows does not work.LOCK TABLEScode this replaces.bulk_insert_ignorealready supports mysql, postgresql and sqlite on master, so nothing in the database layer,hydra.ini, session store or test config changes.Testing
New
tests/test_dataset_insert_race.pysimulates the race by having the first existing-data lookup report nothing for a dataset that is already stored. It fails on master withIntegrityError: UNIQUE constraint failed: tDataset.hashand passes with this change. Ran on the default sqlite backend with test_data, test_datatypes, test_scenario, test_network and test_groups: 155 passed.test_scenario.py::test_bulk_insert_dataneeds MongoDB and was not run locally. This was a single-connection simulation, so real concurrent behaviour on MySQL is untested here.Supersedes #301.