Skip to content

fix(parsers): warn on malformed FHIR resources, strip UTF-8 BOM, reject non-finite GPX coordinates - #367

Merged
kevincostner17 merged 1 commit into
mainfrom
fix/parsers-malformed-input
Sep 15, 2026
Merged

kevincostner17 merged 1 commit into
mainfrom
fix/parsers-malformed-input

Conversation

@kevincostner17

Copy link
Copy Markdown
Contributor

Summary

Three parser fixes so malformed input is recorded in ParseResult.warnings instead of raising or being silently kept.

  • FHIR: malformed resource shapes (FHIR parser raises on shape-malformed resources instead of recording warnings #313). The flatteners assumed R4 list/object shapes and raised KeyError/AttributeError/TypeError on an object-valued address/coding, a list-valued valueQuantity, an R5 list-valued Encounter.class, or a list-valued resourceType. New _as_list/_as_dict helpers handle nested elements and the Bundle entry loop. A resource with a non-string resourceType is skipped with a warning. Any other flattener error becomes a per-resource warning (<type> resource '<id>' skipped: malformed (<Error>)) instead of stopping the parse. Well-formed resources flatten exactly as before, confirmed by comparing old and new flatteners over ~15k generated field shapes.
  • UTF-8 BOM (Parsers do not strip a UTF-8 BOM: HL7 header missed, FHIR JSON rejected #314). Parser.read_text decoded with utf-8, so a BOM stayed in the text: HL7 found no MSH, FHIR JSON was rejected, and EDIFACT ignored custom UNA delimiters. read_text now defaults to utf-8-sig for bytes, binary file objects and paths, and strips a leading BOM from str input and text file objects. XML parsers (GPX, SDMX) are unchanged.
  • GPX: non-finite coordinates (GPX parser keeps lat='nan' / lon='inf' points instead of skipping them with a warning #320). float() accepts "nan"/"inf", so such points were kept. _point now rejects coordinates that are non-finite or outside |lat| <= 90, |lon| <= 180, using the existing skip-with-warning path.

Behaviour changes

  • FHIR: an object where a list is expected is read as one item; a list where an object is expected uses its first element if that is an object. Resources with a non-string resourceType are not counted in resource_types/total_resources.
  • read_text default encoding is now "utf-8-sig".
  • GPX: finite but out-of-range coordinates (e.g. lat="91") are now skipped with a warning.

Tests

New module tests/parsers/test_bom_and_malformed.py (62 tests):

Verification

  • ruff check .: passed
  • mypy src/freshdata: no issues in 202 source files
  • pytest -m "not online and not large": py3.12 4301 passed, py3.9 4297 passed

Closes #313
Closes #314
Closes #320

…ct non-finite GPX coordinates

FHIR (#313): the flatteners indexed nested elements assuming R4 list/dict
shapes, so an object-valued address/coding, a list-valued valueQuantity or
R5 Encounter.class, or a list-valued resourceType raised KeyError,
AttributeError or TypeError. Add _as_list/_as_dict coercion helpers and use
them wherever a flattener or the Bundle entry loop reaches into a nested
element. A resource whose resourceType is not a string is skipped with a
warning, and any remaining flattener error becomes a per-resource
"<type> resource <id> skipped: malformed (<Error>)" warning instead of
aborting the parse. Well-formed resources flatten exactly as before.

BOM (#314): Parser.read_text decoded with "utf-8", so a leading UTF-8 BOM
became part of the text: HL7 saw "\ufeffMSH" and found no messages, FHIR
json.loads rejected the input, and EDIFACT ignored the UNA delimiters.
read_text now defaults to "utf-8-sig" for bytes, binary file objects and
paths, and strips a leading BOM from str input and text file objects. This
covers the HL7 v2, FHIR and EDIFACT parsers; the XML parsers are unchanged.

GPX (#320): float() accepts "nan" and "inf", so such points were kept with
no warning. _point now rejects coordinates that are non-finite or outside
|lat| <= 90, |lon| <= 180 through the existing skip-with-warning path.

Tests for all three live in tests/parsers/test_bom_and_malformed.py and
include each issue's reproduction.

Closes #313
Closes #314
Closes #320
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: fe1fbbbd-a759-40cb-8009-f3aa02d093b4


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

FreshData benchmark report — performance

  • freshdata: ?
  • python: ?
  • platform: ?
fixture n_rows n_cols p50 s p95 s peak MB repair % false-repair % preserve % trust monotonic export %

Authored-code reduction (Metric 6)

@kevincostner17
kevincostner17 merged commit c245b7d into main Sep 15, 2026
21 checks passed
kevincostner17 added a commit that referenced this pull request Sep 15, 2026
Add Unreleased Added, Changed and Fixed entries for merged PRs #351,
#352, #353, #354, #356, #357, #358, #360, #361, #362, #363, #364, #365,
#367, #368, #369, #370, #371, #372, #374, #375, #377, #378 and #379.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant