You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
HALAJ-SCHEMA is a streamlined data model designed for extracting structured disease information from Persian/English clinical reports using Large Language Models. It extends core OMOP CDM principles while optimizing for NLP-based extraction workflows.
Key Features
Feature
Description
Multi-Document Support
Handles radiology, pathology, lab, endoscopy, operative, discharge, and clinical notes
Type-Specific Metadata
Each document type has tailored metadata extraction fields
Bilingual Extraction
Native handling of Persian source with English standardized output
Dynamic Schemas
Disease-specific schemas evolve through LLM-driven discovery
ICD Mapping Cache
Learned cache with vector similarity fallback
Minimal Storage
Reference to source system, text is optional/temporary
Design Decisions
Decision
Rationale
JSONB for modalities
Flexible type-specific metadata, Pydantic validates before insert
flowchart TD
subgraph Stage1["Stage 1: Document Ingestion & Extraction"]
A[/"📄 Receive Document"/] --> B["Determine document_type"]
B --> C["Create PERSON if new"]
C --> D["Create CLINICAL_DOCUMENT"]
D --> E["Create PROCESSING_LOG"]
E --> F["🤖 LLM Extraction"]
F --> G["Store modalities with type-specific details"]
G --> H["Store abnormalities"]
H --> I["Create DISEASE_EXTRACTION records"]
end
subgraph Stage2["Stage 2: ICD Mapping"]
I --> J["Normalize disease_name"]
J --> K{"Check ICD_MAPPING cache"}
K -->|"Found"| L["Use existing mapping"]
K -->|"Not Found"| M["Vector search ICD_EMBEDDING"]
M --> N["🤖 LLM selects best match"]
N --> O["Create ICD_MAPPING entry"]
O --> L
L --> P["Update DISEASE_EXTRACTION"]
end
subgraph Stage3["Stage 3: Metadata Extraction"]
P --> Q{"Schema exists?"}
Q -->|"Yes"| R["🤖 Extract metadata using schema"]
Q -->|"No"| S["Add to SCHEMA_GENERATION_QUEUE"]
S --> T["Skip metadata for now"]
R --> U["Update extraction.metadata"]
T --> U
end
subgraph Stage4["Stage 4: Schema Generation"]
V["⏰ Background Job"] --> W{"sample_count >= threshold?"}
W -->|"Yes"| X["Fetch sample documents"]
X --> Y["Cluster + representative sampling"]
Y --> Z["🤖 LLM generates field_definitions"]
Z --> AA["Create DISEASE_SCHEMA"]
AA --> AB["Re-process pending extractions"]
end
U -.-> V
SELECTde.disease_name_english,
de.icd_code,
cd.document_type,
COUNT(*) as occurrence_count
FROM disease_extraction de
JOIN clinical_document cd ONde.document_id=cd.document_idWHEREcd.document_type='RADIOLOGY'GROUP BYde.disease_name_english, de.icd_code, cd.document_typeORDER BY occurrence_count DESC;
Find Diseases with Specific Modality Details
-- Find pancreatic findings in contrast-enhanced CTSELECTde.disease_name_english,
de.temporality,
cd.modalities->de.modality_index->'radiology_details'->>'contrast_phases'as phases,
cd.document_dateFROM disease_extraction de
JOIN clinical_document cd ONde.document_id=cd.document_idWHEREcd.document_type='RADIOLOGY'ANDde.disease_name_english ILIKE '%pancrea%'AND (cd.modalities->de.modality_index->'radiology_details'->>'contrast_iv')::boolean= true;
Pathology Findings with IHC Results
SELECTde.disease_name_english,
de.icd_code,
cd.modalities->de.modality_index->'pathology_details'->>'tumor_grade'as grade,
cd.modalities->de.modality_index->'pathology_details'->'ihc_results'as ihc
FROM disease_extraction de
JOIN clinical_document cd ONde.document_id=cd.document_idWHEREcd.document_type='PATHOLOGY'ANDcd.modalities->de.modality_index->'pathology_details'->'ihc_results'IS NOT NULL;
Colonoscopy Quality Metrics
SELECTds.source_nameas hospital,
AVG((cd.modalities->0->'endoscopy_details'->>'boston_prep_score')::int) as avg_prep_score,
AVG((cd.modalities->0->'endoscopy_details'->>'withdrawal_time_minutes')::int) as avg_withdrawal_time,
COUNT(*) as procedure_count
FROM clinical_document cd
JOIN data_source ds ONcd.source_id=ds.source_idWHEREcd.document_type='ENDOSCOPY'ANDcd.modalities->0->>'code'='COLON'GROUP BYds.source_name;
Patient Timeline with Document Types
SELECTp.person_source_id,
cd.document_date,
cd.document_type,
de.disease_name_english,
de.temporality,
cd.modalities->de.modality_index->>'name'as modality_name
FROM person p
JOIN disease_extraction de ONp.person_id=de.person_idJOIN clinical_document cd ONde.document_id=cd.document_idWHEREp.person_source_id='PATIENT_123'ORDER BYcd.document_date;
Implementation Guide
Prerequisites
PostgreSQL 14+
pgvector extension
Python 3.10+
Pydantic 2.x
Setup Steps
Create PostgreSQL database with pgvector
psql -U postgres
CREATE DATABASE halaj;\c halaj
CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
fromhalaj_v1_modelsimportICDEmbeddingCreateforicd_code, descriptioninicd_codes:
embedding=get_embedding(description) # Your embedding functionrecord=ICDEmbeddingCreate(
icd_code=icd_code,
icd_description=description,
embedding=embedding,
embedding_model='text-embedding-3-large'
)
# Insert into database
Use Pydantic models for LLM extraction
fromhalaj_v1_modelsimportDocumentExtractionResult# Use as structured output schema for LLMresult=llm.extract(
document_text,
output_schema=DocumentExtractionResult
)
# The model includes comprehensive field descriptions for LLM prompting
LLM Prompt Structure
The Pydantic models in halaj_v1_models.py include detailed field descriptions designed for LLM consumption. Example:
# Get JSON schema for LLM promptschema=DocumentExtractionResult.model_json_schema()
prompt=f"""Extract structured information from this clinical document.Output must conform to this JSON schema:{json.dumps(schema, indent=2)}Document:{document_text}"""
Files
File
Description
README.md
This documentation
DIAGRAMS.md
Additional Mermaid visualizations
halaj_v1_ddl.sql
PostgreSQL DDL with indexes, views, functions
halaj_omop_v1_models.py
Pydantic models with LLM-ready descriptions
Version History
Version
Date
Changes
1.0
Dec 2025
Added document_type, type-specific metadata for all document types
License
Please include a short citation to this project when using it (commercial or non‑commercial). Suggested short citation:
