Skip to content

fix(harvester): skip via ctx cds id - #1051

Open
TahaKhan998 wants to merge 1 commit into
CERNDocumentServer:masterfrom
TahaKhan998:fix/harvester-skip-ctx-cds-id
Open

TahaKhan998 wants to merge 1 commit into
CERNDocumentServer:masterfrom
TahaKhan998:fix/harvester-skip-ctx-cds-id

Conversation

@TahaKhan998

Copy link
Copy Markdown

some records got created even though they still exist on legacy cds. the skip check only looked at metadata.identifiers, and for these that was empty, so it thought there was no legacy cds id. the id was already sitting on _inspire_ctx.cds_id from inspire, we just never used it for that check. now we do, and if old cds still returns 200 we skip like before.

# Also use the CDS id taken from INSPIRE external_system_identifiers
# during transform (_inspire_ctx). Needed when metadata.identifiers
# never got the cds scheme (those records were wrongly created).
ctx_cds_id = (entry.get("_inspire_ctx") or {}).get("cds_id")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't CDS id already be in identifiers (already transformed record, in CDS-RDM format), retrieved in legacy_ids variable?
I suggest investigating why it is not already there, since transformer should retrieve it, instead of adding extra context check. the line above is already specialised in getting the CDS id.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah after looking more into it, those cds ids werent on the inspire records at harvest time (external_system_identifiers), which is why they got created. the next day inspire was updated and the cds ids showed up there, so if we harvested them again now the mapper would put them on metadata.identifiers and theyd be skipped as still on old cds.
image

@TahaKhan998 TahaKhan998 removed their assignment Nov 9, 2026
@kpsherva

Copy link
Copy Markdown
Contributor

discussed this with Micha:
this is a spacial case of the records. First they do not have CDS ID - since CDS is not the original source, then they get imported in the legacy system (they obtain the CDS recid) and then lets say a day after, INSPIRE also adds this new CDS ID to their records (to gather all the identifiers this record can be represented with)

@TahaKhan998

Copy link
Copy Markdown
Author

discussed this with Micha: this is a spacial case of the records. First they do not have CDS ID - since CDS is not the original source, then they get imported in the legacy system (they obtain the CDS recid) and then lets say a day after, INSPIRE also adds this new CDS ID to their records (to gather all the identifiers this record can be represented with)

Should we just close this pr in that case?

@TahaKhan998 TahaKhan998 self-assigned this Nov 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants