Track the provenance of parsed values - #975
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #975 +/- ##
==========================================
Coverage 100.00% 100.00%
==========================================
Files 27 27
Lines 8777 9043 +266
==========================================
+ Hits 8777 9043 +266 ☔ View full report in Codecov by Harness. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What does this PR do?
A parse combines defaults, config files, environment variables and command line arguments, but afterwards it was not known where each value came from. Now every value's source is recorded, and shown in errors and by a new
provenanceflag of the print config argument:Sources are kept per leaf in a
__provenance__slot of eachNamespace, also for namespaces in lists and dicts, so equality and dumps are unaffected and there is no new public API. A context variable holds the source being applied. Code that processes already parsed values, e.g. type adaptation or merging, runs without one, so values keep their sources. Line numbers are only for YAML, computed lazily.Also fixed: list adaptation deep copied the whole list for every item, making parsing quadratic, e.g. a minute for 10000 numbers. Parsing is now within noise of
main, and faster for lists.Behavior changes: errors due to a value end with a
Source:line, thecommentsflag is always accepted, and assigning__class__to convert anargparse.Namespaceno longer works.Not addressed: single source for plain lists like
list[int], errors for nested config values point to the top level key, and repeated--x+=re-adapting previous items.Before submitting