Skip to content

Latest commit

 

History

History
258 lines (204 loc) · 10 KB

analyze-text.md

File metadata and controls

258 lines (204 loc) · 10 KB

Analyze Text Helper

openmed.analyze_text is the top-level orchestrator that most users start with. It validates input, loads a token-classification or native span model, segments sentences, and normalizes the output so you can copy dict/JSON/HTML/CSV payloads straight into downstream systems.

GLiFormer and GLiNER2 (v3.2)

OpenMed 3.1's token-classification pipeline cannot load GLiFormer. Starting in v3.2, native GLiNER, GLiFormer, and GLiNER2 / GLiNER2.5 checkpoints share the same analyze_text result and formatting interface. Install the matching extra:

pip install "openmed[gliformer]"  # GLiFormer 0.1.3
pip install "openmed[gliner2]"    # gliner2[local] 2.1+, including GLiNER2.5
from openmed import OpenMedConfig, analyze_text

text = "No asthma. A synthetic note documents diabetes and asthma."
result = analyze_text(
    text,
    model_id="OpenMed/GLiFormer-BioNER-Disease-Base-v1",
    threshold=0.7,
    config=OpenMedConfig(device="cpu"),
    assert_context=True,
)
for entity in result.entities:
    assert text[entity.start:entity.end] == entity.text
    print(entity.text, entity.label, entity.confidence)

This registered checkpoint defaults to labels=["disease"]. Extraction uses the model's native labels; returned OpenMed entities normalize disease to DISEASE. Disease mentions remain mentions even when negated: the optional context stage adds assertion metadata, and does not establish a diagnosis.

Use explicit labels for general-purpose models:

result = analyze_text(
    "A synthetic note documents asthma.",
    model_id="fastino/gliner2.5-base-v1",
    labels=["disease"],
    threshold=0.7,
    config=OpenMedConfig(device="cpu"),
)

Recognized family names and local checkpoint configurations select the native loader without a generated index or a Hub metadata request. For a custom Hub ID whose name does not identify its architecture, pass family="gliformer", family="gliner2", or family="gliner". Local exports with renamed directories are detected from gliner_config.json / config.json; supply labels=["disease"] when the exported directory no longer identifies the registered disease model.

threshold controls native extraction (default 0.5); confidence_threshold is a separate final output filter. Sentence chunking and absolute offsets, output_format, clinical context, and in-memory result caching still apply. Native pipelines accept a string or a list of strings and process chunks in order; num_workers must be zero. aggregation_strategy and fast-tokenizer selection apply to Transformers models. Native loading accepts revision, cache_dir, token, device, and local_files_only; unsupported loading options raise an error rather than being silently ignored. max_length controls the high-level chunk-size hint, while each native runtime owns token limits.

Weights are downloaded on first use and kept in the configured local cache. For offline use, pre-download the checkpoint and any required tokenizer assets, then use a local directory or OpenMedConfig(local_only=True, device="cpu"). An uncached dependency fails offline; inference always uses the local model. See the zero-shot toolkit for the lower-level API and upstream GLiFormer usage.

Quick reference

from openmed import analyze_text

result = analyze_text(
    text="Patient started on imatinib for chronic myeloid leukemia.",
    model_name="disease_detection_superclinical",
    aggregation_strategy="simple",
    output_format="dict",
    include_confidence=True,
    confidence_threshold=0.55,
    group_entities=True,
    metadata={"source": "clinic-note-42"},
)
print(result.model)
print(result.entities[:3])
payload = result.to_dict()

Key arguments

  • model_name: registry alias, full Hugging Face id, or local model directory. Use openmed.list_models() if you need auto-discovery.
  • model_id: alias for model_name, supported for API-style callers that use model-id terminology.
  • aggregation_strategy: forwarded to the HF pipeline. simple (default) yields grouped tokens; None keeps raw tokens.
  • output_format: "dict" (default, returns AnalyzeResult), "json", "html", or "csv".
  • include_confidence & confidence_threshold: control the final payload; defaults keep all scores.
  • group_entities: merge adjacent spans of the same label after formatting.
  • formatter_kwargs: forwarded to openmed.processing.format_predictions.
  • assert_context: opt in to deterministic clinical assertion labels. Each entity receives negation, uncertainty, experiencer, and temporality under entity.metadata["clinical_context"].
  • Sentence options (sentence_detection, sentence_language, sentence_clean, sentence_segmenter, sentence_backend) wrap the sentence engine so each prediction carries the sentence span; disable them if latency matters more than helper metadata.

The context stage is disabled by default. Enable it for clinical entities that will flow into grounding, FHIR export, or problem-list review:

result = analyze_text(
    "No evidence of pneumonia.",
    assert_context=True,
)
print(result.entities[0].metadata["clinical_context"])

Optional YASBD sentence backend

The default sentence_backend="auto" path is unchanged: OpenMed uses its built-in Indic and Chinese segmenters where appropriate and pySBD elsewhere. YASBD is neither installed nor imported by a core OpenMed installation.

Install the experimental backend explicitly when you want to benchmark it on your own workload:

pip install "openmed[yasbd]"

Then select it for either the low-level sentence API or analyze_text:

from openmed import analyze_text
from openmed.processing.sentences import segment_text

spans = segment_text(
    "Patient is stable. Follow up tomorrow.",
    language="en",
    backend="yasbd",
)

result = analyze_text(
    "Patient is stable. Follow up tomorrow.",
    sentence_backend="yasbd",
)

The adapter preserves OpenMed's exact, contiguous source offsets and assigns inter-sentence whitespace to the preceding span, matching the existing span contract. YASBD remains an explicit opt-in because sentence boundaries can differ between engines; validate representative clinical and multilingual inputs before adopting it in production. If the extra is missing, explicitly selecting "yasbd" raises an installation error instead of silently changing behavior.

Chunking & truncation

result = analyze_text(
    text=long_report,
    model_name="pharma_detection_superclinical",
    aggregation_strategy=None,  # work with raw tokens
    max_length=512,             # forwarded to HF pipeline
    truncation=True,            # enforce length (default)
    sentence_detection=False,   # skip sentence detection to save ~2ms per note
    sentence_backend="auto",    # "auto" (default) or "yasbd" (experimental, faster)
)

When you need full-control over tokenizer behaviour:

  • Pass max_length/truncation via pipeline_kwargs. If you skip truncation, the helper sets the tokenizer max length to unlimited (0) so HF pipelines accept longer inputs.
  • Provide batch_size or num_workers in pipeline_kwargs and they will be forwarded to the pipeline call but not to the constructor.
  • Enable medical token remapping with OpenMedConfig(use_medical_tokenizer=True) to group outputs onto clinical-friendly tokens without changing the model tokenizer.

Loading from a local path

Pass an existing model directory to model_name or model_id when the model files are already present on disk:

import os
from openmed import OpenMedConfig, analyze_text

local_path = os.path.abspath("./models/OpenMed-NER-DiseaseDetect-SuperClinical-434M")
config = OpenMedConfig(device="cpu")

result = analyze_text(
    "Patient presents with chronic myeloid leukemia and Type 2 diabetes.",
    model_id=local_path,
    config=config,
)

for entity in result.entities:
    print(entity.text, entity.label)

legacy_payload = result.to_dict()
print(legacy_payload["model_name"])

When the identifier points to an existing local path, OpenMed asks Transformers to load with local_files_only=True by default. That keeps air-gapped deployments from validating or downloading the model from the Hugging Face Hub. If any required tokenizer, config, or weight file is missing, loading fails locally with the underlying Transformers error.

Streaming multiple texts

analyze_text is optimized for single inputs. For batch jobs, keep a ModelLoader instance around and reuse its pipelines:

from openmed import ModelLoader, format_predictions

loader = ModelLoader()
pipeline = loader.create_pipeline("disease_detection_superclinical")

for note in notes:
    raw = pipeline(note, batch_size=16)
    formatted = format_predictions(raw, note, model_name="Disease Detection")
    print(formatted.entities[:3])

See ModelLoader & Pipelines for details on caching, GPU selection, and tokenizer reuse.

HTML/CSV rendering

html = analyze_text(
    text,
    model_name="oncology_detection_superclinical",
    output_format="html",
    formatter_kwargs={
        "html_class": "openmed-highlights",
        "tag_colors": {"CANCER": "#d97706"},
    },
)

csv_rows = analyze_text(
    text,
    model_name="pharma_detection_superclinical",
    output_format="csv",
)

The HTML formatter emits a ready-to-embed snippet for dashboards; CSV mode writes row strings (header + body). Both respect confidence_threshold and group_entities.

Validation behaviours

Behind the scenes analyze_text calls:

  • validate_input — trims whitespace and enforces max lengths.
  • validate_model_name — normalizes registry aliases.
  • Sentence detection (openmed.processing.sentences) — optional segmentation with language hints and a selectable backend.
  • OutputFormatter — see Advanced NER & Output Formatting for available kwargs.

If you need custom validation or logging, inject your own OpenMedConfig or reuse a configured ModelLoader.