Skip to content

Latest commit

 

History

History
90 lines (74 loc) · 3.85 KB

beam.md

File metadata and controls

90 lines (74 loc) · 3.85 KB

Apache Beam Redaction Transform

OpenMed supports bounded Beam redaction pipelines and a Beam-free synthetic harness. For v3.1, use Python 3.11+ and the tested versions below. Plain pip cannot resolve Beam's Pillow cap. Its cryptography and httplib2 caps also need these security overrides; never downgrade or disable dependency installation.

uv venv --python 3.11 .venv
printf '%s\n' 'Pillow==13.0.0' 'cryptography==50.0.0' 'httplib2==0.32.0' \
  > beam-security-overrides.txt
uv pip install --no-config --override beam-security-overrides.txt \
  "openmed[beam]==3.2.0" "apache-beam==2.75.0"

DirectRunner tests pass with these versions. pip check still reports Beam's three upstream bounds; investigate any other conflict. See the dependency policy.

Contract

The transform accepts either a string element or a mapping with a configured text field. Mapping records keep their outer shape and only the text field is transformed. The default bounds are 10,000 records, 10 MiB each of serialized input and output, 1 MiB per input or output record, and three redaction attempts. Records must contain bounded, JSON-compatible values and string mapping keys. Cyclic, hostile, or oversized inputs fail with stable value-free errors before redaction.

import apache_beam as beam

from openmed.interop.beam import BeamRedactionSpec, BeamRedactionTransform

spec = BeamRedactionSpec(
    text_field="note",
    policy="hipaa_safe_harbor",
    max_records=10_000,
)

with beam.Pipeline() as pipeline:
    redacted = (
        pipeline
        | beam.Create([{"record_id": "synthetic-1", "note": "synthetic note"}])
        | BeamRedactionTransform(spec)
    )

Workers create their model loader locally. The default OpenMed deidentifier is configured for cache-only loading with credential discovery disabled, mapping and audit retention off, and the safety sweep enabled, so constructing the transform does not require network access. Pre-stage the model on each worker or inject a local deidentifier for an air-gapped deployment and for tests. Additional deidentifier options are snapshotted into a bounded, serializable configuration; worker loader, policy, method, and safety controls cannot be overridden through that mapping.

extra_kwargs are bounded and copied at specification construction. They may not override config, loader, method, or policy, so they cannot replace the worker-local loader or the cache-only default configuration. Reports expose only the option count and a fingerprint of option keys, never raw keys or values. Options must be data-only values: nulls, booleans, finite numbers, strings, bytes, and bounded nested lists, tuples, or string-keyed dictionaries. The snapshot accepts at most 128 top-level options, 4,096 nested values, and 4 MiB of aggregate key, string, and byte data.

Direct synthetic harness

run_synthetic_harness applies the same schema validation, canonical JSON serialization, bounds, and capped retry loop without starting a Beam runner:

from openmed.interop.beam import BeamRedactionSpec, run_synthetic_harness

result = run_synthetic_harness(
    [{"note": "synthetic note"}],
    spec=BeamRedactionSpec(text_field="note"),
    deidentifier=my_local_deidentifier,
)
print(result.redacted_records)
print(result.report())

The report contains only schema metadata, SHA-256 fingerprints, and counters for records, attempts, retries, bytes, and redacted spans. Input values, output values, model exceptions, record identifiers, and deidentifier return objects are never copied into reports, logs, or contract exceptions. Retry backoff is bounded and disabled by default for deterministic direct runs. Redacted records and batches also have bounded expansion budgets, preventing a redactor from turning a valid input bound into unbounded worker output.