Repository navigation
fix(parsers): warn on malformed FHIR resources, strip UTF-8 BOM, reject non-finite GPX coordinates - #367
Merged
Conversation
…ct non-finite GPX coordinates FHIR (#313): the flatteners indexed nested elements assuming R4 list/dict shapes, so an object-valued address/coding, a list-valued valueQuantity or R5 Encounter.class, or a list-valued resourceType raised KeyError, AttributeError or TypeError. Add _as_list/_as_dict coercion helpers and use them wherever a flattener or the Bundle entry loop reaches into a nested element. A resource whose resourceType is not a string is skipped with a warning, and any remaining flattener error becomes a per-resource "<type> resource <id> skipped: malformed (<Error>)" warning instead of aborting the parse. Well-formed resources flatten exactly as before. BOM (#314): Parser.read_text decoded with "utf-8", so a leading UTF-8 BOM became part of the text: HL7 saw "\ufeffMSH" and found no messages, FHIR json.loads rejected the input, and EDIFACT ignored the UNA delimiters. read_text now defaults to "utf-8-sig" for bytes, binary file objects and paths, and strips a leading BOM from str input and text file objects. This covers the HL7 v2, FHIR and EDIFACT parsers; the XML parsers are unchanged. GPX (#320): float() accepts "nan" and "inf", so such points were kept with no warning. _point now rejects coordinates that are non-finite or outside |lat| <= 90, |lon| <= 180 through the existing skip-with-warning path. Tests for all three live in tests/parsers/test_bom_and_malformed.py and include each issue's reproduction. Closes #313 Closes #314 Closes #320
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
FreshData benchmark report —
|
| fixture | n_rows | n_cols | p50 s | p95 s | peak MB | repair % | false-repair % | preserve % | trust | monotonic | export % |
|---|
Authored-code reduction (Metric 6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three parser fixes so malformed input is recorded in
ParseResult.warningsinstead of raising or being silently kept.KeyError/AttributeError/TypeErroron an object-valuedaddress/coding, a list-valuedvalueQuantity, an R5 list-valuedEncounter.class, or a list-valuedresourceType. New_as_list/_as_dicthelpers handle nested elements and the Bundleentryloop. A resource with a non-stringresourceTypeis skipped with a warning. Any other flattener error becomes a per-resource warning (<type> resource '<id>' skipped: malformed (<Error>)) instead of stopping the parse. Well-formed resources flatten exactly as before, confirmed by comparing old and new flatteners over ~15k generated field shapes.Parser.read_textdecoded withutf-8, so a BOM stayed in the text: HL7 found noMSH, FHIR JSON was rejected, and EDIFACT ignored customUNAdelimiters.read_textnow defaults toutf-8-sigfor bytes, binary file objects and paths, and strips a leading BOM fromstrinput and text file objects. XML parsers (GPX, SDMX) are unchanged.float()accepts"nan"/"inf", so such points were kept._pointnow rejects coordinates that are non-finite or outside |lat| <= 90, |lon| <= 180, using the existing skip-with-warning path.Behaviour changes
resourceTypeare not counted inresource_types/total_resources.read_textdefaultencodingis now"utf-8-sig".lat="91") are now skipped with a warning.Tests
New module
tests/parsers/test_bom_and_malformed.py(62 tests):read_textsource type (str, bytes, bytearray, BytesIO, StringIO, Path); the issue repro; HL7v2, FHIR and EDIFACT (custom UNA delimiters) parsed from BOM input of every source type._as_list/_as_dictunit tests; object/list/scalar-valued fields; non-stringresourceType; a failing flattener becoming a per-resource warning; non-list Bundleentry; a full well-formed bundle compared field by field against expected rows.Verification
ruff check .: passedmypy src/freshdata: no issues in 202 source filespytest -m "not online and not large": py3.12 4301 passed, py3.9 4297 passedCloses #313
Closes #314
Closes #320