Reads delimited and fixed-width record files and maps them to FHIR JSON resources. Column mapping, code translation and resource shaping are declared in a JSON data contract, so onboarding a new source extract is a configuration change, not a patch.
cargo build --release # target/release/fhir-utils
cargo build --release --target x86_64-unknown-linux-musl # fully static binary
cargo test # unit, integration, end-to-endCommand |
Purpose |
|
validate a data contract |
|
print the resolved contract, definitions inlined |
|
convert every input under |
|
convert one file |
|
print the normalized internal records as JSON lines |
Flag |
Meaning |
Required |
|
contract file for |
yes for |
|
base directory holding |
one of |
|
directory holding the contract; may repeat, later ones are fallback layers |
with |
|
expected SHA-256 of the contract; a mismatch stops the run |
no |
|
output directory, created when missing |
yes for |
|
fail on a task error, an empty group key or a row that builds nothing |
no |
|
one bundle per group key: |
no |
|
per-file read, tasks, build and write timings on standard error |
no |
-d and -f are mutually exclusive; -f without -c reads the contract from
MAPPING_CONFIG_DIRECTORY, and -d requires both
<base>/input and <base>/config. Exit code 0 on success, 1 on an invalid
contract, a missing path or a row conversion failure.
Inputs, code maps and join sources ending in .gz are decompressed transparently.
transform takes the convert flags without -o and prints one record per row on
standard output, which shows what the tasks produced before resource conversion.
Source extract input/patients.csv with headers that do not match the internal model:
MRN,LastName,FirstName,Sex,DOB,SSN 1234567,Smith,Ann,F,1980-01-02,123-45-6789 7654321,Jones,Bob,M,1975-11-30,
Code map config/sex.csv, two columns, header names fixed:
source_value,target_value F,female M,male
Contract config/data-contract.json:
{
"general": {
"timeZone": "America/New_York",
"tenantId": "hospital-a",
"streamType": "live"
},
"fileDefinitions": {
"patients": {
"fileType": "csv",
"resourceType": "Patient",
"groupByKey": "MRN",
"tasks": [
{
"task": "rename_columns",
"column_map": {
"MRN": "mrn",
"LastName": "nameLast",
"FirstName": "nameFirst",
"Sex": "gender",
"DOB": "birthDate",
"SSN": "ssn"
}
},
{ "task": "map_codes", "code_map": { "gender": "sex.csv" } }
]
}
}
}Run:
fhir-utils validate -f config/data-contract.json
fhir-utils convert -f input/patients.csv -c config -o out
fhir-utils convert --strict -f input/patients.csv -c config -o outResult:
out/1234567/1234567-Patient-patients-00001.json out/7654321/7654321-Patient-patients-00001.json
The key patients matches the file name patients.csv, groupByKey names the source
column because grouping happens before tasks, and sex.csv resolves against the
configuration directory. Target column names come from the record fields below; a name
outside that list travels through the pipeline but never reaches the resource.
Key |
Description |
Required |
|
IANA zone used to localize naive datetimes |
yes |
|
tenant identifier injected into every row |
yes |
|
|
yes |
|
identifier system for PI, MR, AN, VN, RI, RXN |
no |
|
values read as null |
no |
|
match file definition keys as regular expressions |
no |
|
code system names to canonical URLs: object or JSON file name |
no |
|
|
no |
|
one of the resource keys |
yes |
|
source column that groups the output |
yes |
|
column or list of columns deriving a stable resource id |
no |
|
line count to drop from the top, or zero-based line numbers |
no |
|
column names, names with comments, or names with widths |
no |
|
ordered tasks appended after the default chain |
no |
-
Every
generalkey is injected into each row as a constant, including keys beyond the ones listed above, so a contract can carry its own values into tasks. -
A key matches when it is a substring of the input file name without its extension; longest key first, then alphabetically.
-
Fixed-width files need a width form of
headers; otherwise the first remaining line supplies the column names. -
valueDelimitermay be several characters; quoted fields need a one-character delimiter. -
Input files matching no definition are skipped and named in the summary.
-
Repeated column names keep the last column of the row.
Patient, AllergyIntolerance, Condition, Encounter, Immunization, Observation,
Location, Organization, Practitioner, Procedure, MedicationUse,
MedicationAdministration, MedicationRequest, MedicationStatement,
DocumentReference, DiagnosticReport, Unstructured, Basic.
Keys match regardless of case; patient reads as Patient.
Task output must use these names; anything else is ignored by the converter. Every row
also carries filePath and rowNum. ssn is normalized to nine digits without
dashes and repeated-digit values are rejected. a → b renames a to b.
Model |
Fields |
Patient |
|
AllergyIntolerance |
|
Condition |
|
Encounter |
|
Immunization |
|
Location |
|
MedicationUse |
|
Observation |
|
Organization |
|
Practitioner |
|
Procedure |
|
Basic |
|
Unstructured |
|
Default chain, in order: add_row_num, set_nan_to_none,
remove_whitespace_from_columns, copy_columns into groupByKey, then add_constant
per general key plus filePath and configResourceType. User tasks run after it.
Task |
Parameters |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
+ |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
none |
|
|
|
|
|
none |
|
|
|
|
|
|
-
Tasks run in order over one chunk at a time; a failing task is logged and the chain continues.
-
A
commentkey on a task is accepted and ignored. -
List valued cells are strings joined with
|. -
condition_mapandcode_maptake an inline object or a CSV file withsource_valueandtarget_valuecolumns; anullor empty target clears the cell. -
A
.jsonmap file holds one flat object or many named tables addressed asfile.json#table; a many-table file used without#tablefails naming the tables. An object target maps a code to a{code, system, display, text}concept whose parts land in<column>System,<column>Displayand<column>Text.defaultworks likedefault. -
A value the code map does not list passes through unchanged, which is reported once per file and column; add a
defaultentry to control it. -
Unknown tasks and unknown or missing parameters are rejected when the contract loads, as is an unknown key inside a file definition or a missing
fileDefinitionsblock. -
A task target that neither a later task nor the resource reads is reported by
validateand warned about duringconvert;--strictmakes it fail the run. It catches a misspelled field name, which otherwise drops the value silently.
-
Path:
<out>/<groupByKey>/<groupByKey>-<resourceType>-<safeFileId>-<nnnnn>.json. -
--bundlewrites<out>/<groupByKey>/<groupByKey>-Bundle.jsoninstead: one bundle per group key withfullUrlper entry andmeta.lastUpdated;batchandtransactionentries carry aPUTrequest, which pairs withidKeyfor idempotent reloads. -
The counter runs per group key, resource type and source file.
-
JSON is pretty printed with sorted keys.
-
Empty strings are pruned from every resource, and objects or arrays left empty by that collapse with them; numbers, booleans and explicit null values survive.
-
safeFileIdkeeps[A-Za-z0-9-]and maps every other character to_. -
Group key directories keep letters, digits, spaces and dots; path separators, control characters and
\ : * ? " < > |become, and an all-dot key becomes. -
Re-running overwrites matching file names and leaves earlier files in place.
-
The summary counts files, rows read, resources written and rows that produced none.
-
A resource repeating an earlier one’s type and id in the same run is merged into it and counted in the summary: missing fields fill in, arrays union, and a conflicting scalar keeps the first value with a warning. Generated ids never repeat, so this only affects mapped or
idKey-derived ids. A remote output keeps the first occurrence unmerged, as it cannot be reread. -
A row whose record carries no usable code produces no resource; those rows are counted in the summary, listed by row number on standard error, and fatal with
--strict. -
Resource ids are generated per run unless the contract maps
resourceInternalId, so a reload creates new ids; theurn:id:extIDidentifier and theurn:id:source-file-idextension, which carries<file>:<rowNum>, stay stable and are the keys to match on. -
idKeynames one column or a list of columns whose values, prefixed with the resource type, seed a name-based UUID that fills an emptyresourceInternalIdafter the tasks run, so a reload keeps the same ids. A list composes a hierarchical seed such as patient plus code; a row missing any part keeps a generated id. -
The summary goes to standard output; warnings and errors go to standard error.
-
Warnings print by default;
RUST_LOG=infoadds a line per file naming the definition that matched it and, once converted, its row and resource counts.
Date parsing is shared by format_date, compare_to_date and the resource builders.
Ambiguous numeric forms read month first.
Form |
Examples |
ISO |
|
Compact |
|
Month first |
|
Named month |
|
Separators |
|
-
format_datereformats parsed values and passes the rest through; unparsed values are reported once per file and column. -
A bare year between 1900 and 2200, or a year-month such as
1985-06, is valid for date fields and keeps its precision. -
Naive datetimes are localized with
general.timeZone; an offset embedded in the value wins over the contract zone.
-
Short system names render as canonical URLs:
ICD9,ICD10,ICD10PCS,LOINC,SNOMED,CPT,CVX,NDC,RXNORM,MESH,NCI,UMLS,HCPCS,ATC,CDT,UCUM; matching ignores case and common alternate spellings. -
general.codeSystemsadds or overrides entries: an inline object of name to URL, or the name of a JSON file resolved through the pluggable opener. -
A name outside the table renders as
urn:id:<value>; full URIs pass through.
FHIR requires some fields the source extract may not carry. When a column is absent the converter fills these so the output loads into a server:
Resource |
Field |
Default |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
data absent |
|
|
|
Map the real column whenever the source has one; the defaults keep output valid, they do not carry clinical meaning.
Fatal errors print error E<code>: <message>. Codes are stable and grouped by
phase; message text may change between releases.
Code |
Meaning |
|
invalid time zone |
|
invalid stream type |
|
unknown resource type |
|
fixed width without name/width headers |
|
unknown task |
|
unknown task parameter |
|
missing task parameter |
|
contract or configuration error |
|
no file definition matched |
|
input parsing failed |
|
file system error |
|
row or output conversion failed |
|
JSON serialization failed |
|
writing output failed |
Class |
Behaviour |
Contract or validation error |
run stops naming the key or the accepted values |
Missing file definition |
file skipped |
Task failure |
warning, pipeline continues; fatal with |
Task target no field reads |
warning at load; fatal with |
Rows without a group key |
warning once per file; fatal with |
Rows producing no resource |
counted in the summary and listed; fatal with |
Reference to a resource the run did not write |
warning listing up to five; fatal with |
Resource carrying only its type, id and meta |
warning listing up to five; fatal with |
Unlisted code map value |
warning once per file and column, value passes through |
Input that is not UTF-8 |
run stops naming the file and the re-encoding command |
Row conversion error |
run stops with the group key and file |
Variable |
Meaning |
Default |
|
rows per chunk; |
|
|
contract directory used when |
|
|
contract file name |
|
Settings are read once per process. Contracts, external file definitions, mapping files
and join sources open through one pluggable opener: local paths and file:// URIs work
out of the box, other schemes need a backend registered with opener::set_opener. The
opener is scoped to the calling thread, so each worker thread installs its own.
Output resources are written through the same opener when the output carries a URI
scheme; a plain path writes to the local filesystem as before.
The optimized-streaming build feature adds http:// and https:// sources,
including a remote contract directory; other schemes still need a registered backend.
Repeating -c layers configuration directories: the contract and every map or
definition resolve from the first layer that has them, so a user layer ships only
its overrides on top of a shared base. --contract-digest sha256hex verifies the
contract bytes before the run, making a distributed contract reproducible.