Since #6, extract_edb.py generates a synthetic @norm/requires_dist row containing a normalised value (quotes, spaces, parens, hyphens vs. underscores, etc.) for each explicit requires_dist value in METADATA. However, because the original unnormalised requires_dist rows still exist in the EDB and still get copied into the IDB, even if the normalised rows agree, we can wind up incorrectly concluding there is a difference.
I propose removing the unnormalised rows from the IDB as follows:
- Change the
id field of the synthetic @norm/requires_dist rows from F<n> to S_NORM[F<m>], where F<m> is the id of the corresponding requires_dist row
- Add a rule to
metadata-normalization.souffle that uses remove_content() to prevent any F<m> fact being copied into the IDB when a corresponding S_NORM[F<m>] fact exists.
This gives maximum control over what makes it into the IDB, makes the provenance explicit, and can apply straightforwardly to any similar future cases of normalisation-in-the-extractor.
Since #6,
extract_edb.pygenerates a synthetic@norm/requires_distrow containing a normalised value (quotes, spaces, parens, hyphens vs. underscores, etc.) for each explicitrequires_distvalue inMETADATA. However, because the original unnormalisedrequires_distrows still exist in the EDB and still get copied into the IDB, even if the normalised rows agree, we can wind up incorrectly concluding there is a difference.I propose removing the unnormalised rows from the IDB as follows:
idfield of the synthetic@norm/requires_distrows fromF<n>toS_NORM[F<m>], whereF<m>is theidof the correspondingrequires_distrowmetadata-normalization.soufflethat usesremove_content()to prevent anyF<m>fact being copied into the IDB when a correspondingS_NORM[F<m>]fact exists.This gives maximum control over what makes it into the IDB, makes the provenance explicit, and can apply straightforwardly to any similar future cases of normalisation-in-the-extractor.