Repository navigation
fix(harvester): skip via ctx cds id - #1051
TahaKhan998 wants to merge 1 commit into
Conversation
| # Also use the CDS id taken from INSPIRE external_system_identifiers | ||
| # during transform (_inspire_ctx). Needed when metadata.identifiers | ||
| # never got the cds scheme (those records were wrongly created). | ||
| ctx_cds_id = (entry.get("_inspire_ctx") or {}).get("cds_id") |
There was a problem hiding this comment.
shouldn't CDS id already be in identifiers (already transformed record, in CDS-RDM format), retrieved in legacy_ids variable?
I suggest investigating why it is not already there, since transformer should retrieve it, instead of adding extra context check. the line above is already specialised in getting the CDS id.
There was a problem hiding this comment.
yeah after looking more into it, those cds ids werent on the inspire records at harvest time (external_system_identifiers), which is why they got created. the next day inspire was updated and the cds ids showed up there, so if we harvested them again now the mapper would put them on metadata.identifiers and theyd be skipped as still on old cds.

|
discussed this with Micha: |
Should we just close this pr in that case? |
some records got created even though they still exist on legacy cds. the skip check only looked at metadata.identifiers, and for these that was empty, so it thought there was no legacy cds id. the id was already sitting on _inspire_ctx.cds_id from inspire, we just never used it for that check. now we do, and if old cds still returns 200 we skip like before.