Enhance MetaTraits transform with chemical mapping and expanded METPO predicates

[realmarcin]

Summary

This PR enhances the MetaTraits transform to resolve more chemical-related traits by integrating unified chemical mapping infrastructure and expanding METPO predicate coverage.

Changes

Phase 1: Chemical Mapping Infrastructure

• ✅ Merged chemical_mappings branch (164,705 ChEBI IDs)
• ✅ Added mappings/unified_chemical_mappings.tsv.gz (8.4 MB)
• ✅ Added kg_microbe/utils/chemical_mapping_utils.py (ChemicalMappingLoader class)

Phase 2-4: MetaTraits Transform Enhancement

• ✅ Integrated ChemicalMappingLoader in metatraits transform
• ✅ Implemented _resolve_chemical_trait() method with 8 pattern matchers:
  • carbon source: X → METPO:2000006 (uses as carbon source)
  • produces: X → METPO:2000202 (produces)
  • ferments: X → METPO:2000011 (ferments)
  • hydrolyzes: X → METPO:2000013 (hydrolyzes)
  • oxidizes: X → METPO:2000016 (oxidizes)
  • reduces: X → METPO:2000017 (reduces)
  • degrades: X → METPO:2000007 (degrades)
  • utilizes: X → METPO:2000001 (organism interacts with chemical)
• ✅ Expanded METPO predicate mappings from 3 to 30 predicates
• ✅ Added biolink:interacts_with → RO:0002434 mapping
• ✅ Integrated chemical resolver as Tier 1.5 in lookup hierarchy

Phase 5: ChEBI Category Fix

• ✅ Fixed ChEBI categories to use biolink:ChemicalSubstance
• ✅ Updated constants.py to normalize SmallMolecule → ChemicalSubstance
• ✅ Added new CHEBI_CATEGORY constant

Expected Impact

• 10-30% increase in mapped edges for chemical-related traits
• ~100K+ reduction in unmapped_traits.tsv
• More semantic specificity: ferments/oxidizes/degrades vs generic capable_of
• Standardized ChEBI IDs with canonical names

Baseline Metrics (Before Enhancement)

• Edges: 829,353
• Unmapped traits: 5,270,596
• Top unmapped patterns: produces: acetate (43,777), carbon source: methyl (43,770), utilizes: citrate (41,047)

Files Changed

Modified:

• kg_microbe/transform_utils/metatraits/metatraits.py (+127, -24 lines)
• kg_microbe/transform_utils/constants.py (ChEBI category updates)

Added (from chemical_mappings merge):

• mappings/unified_chemical_mappings.tsv.gz (8.4 MB, 164,705 ChEBI entries)
• kg_microbe/utils/chemical_mapping_utils.py (ChemicalMappingLoader)
• scripts/consolidate_chemical_mappings.py
• tests/test_chemical_mapping_utils.py
• Documentation in mappings/README.md and CONSOLIDATION_SUMMARY.md

Testing

• ✅ Code formatted (black + ruff)
• ✅ All commits follow project conventions
• ⏳ Full transform validation pending completion

Documentation

Comprehensive implementation summary in METATRAITS_ENHANCEMENT_SUMMARY.md

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR enhances MetaTraits trait resolution by introducing unified chemical name→ChEBI mapping, expanding METPO predicate→Biolink/RO coverage, and adding a GTDB-based MetaTraits transform with optional multiprocessing to improve throughput and mapping accuracy.

Changes:

• Integrated ChemicalMappingLoader and added pattern-based chemical/metabolic/growth/trophic resolvers to map more trait strings to ChEBI + METPO predicates.
• Expanded METPO predicate→Biolink mappings and normalized ChEBI-related Biolink categories to biolink:ChemicalSubstance.
• Added metatraits_gtdb transform and multi-process execution path (including chunking for single-file runs) plus related docs/tests.

Reviewed changes

Copilot reviewed 53 out of 56 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_metatraits.py	Updates expected chemical categories and tweaks fixture input dir usage.
tests/test_gtdb.py	Formatting-only changes.
tests/test_biolink_hierarchy.py	Formatting-only changes in assertions and calls.
tests/test_bakta.py	Formatting-only changes in test calls.
tests/test_assay_generation.py	Formatting-only changes in a parameterized test and conditions.
pyproject.toml	Adds psutil dependency (used for worker auto-sizing).
mappings/phenotype_mappings_corrected.tsv	Adds a corrected phenotype mapping reference TSV for validation workflows.
mappings/custom_mappings_not_in_metpo.tsv	Adds analysis output documenting tiers/patterns and custom mappings.
mappings/additional_metpo_mappings.tsv	Adds analysis output enumerating recommended new METPO terms/predicates.
mappings/VALIDATION_CHECKLIST.md	Adds a manual checklist for verifying phenotype mapping corrections.
mappings/METPO_TERM_REQUESTS.md	Adds a term request document (dated 2026-03-25) for METPO maintainers.
mappings/METPO_PRIORITY_CHANGE_PLAN.md	Adds plan doc for METPO-first trait resolution order and cleanup strategy.
mappings/METATRAITS_UNMAPPED_ANALYSIS.md	Adds analysis doc for unmapped trait categories and recommendations.
mappings/CUSTOM_MAPPINGS_ANALYSIS.md	Adds analysis doc contrasting custom mappings vs METPO coverage.
kg_microbe/utils/uniprot_utils.py	Formatting-only changes (line wrapping).
kg_microbe/utils/unipathways_utils.py	Formatting-only changes (line wrapping).
kg_microbe/utils/trembl_utils.py	Formatting-only changes (comprehensions and wrapping).
kg_microbe/utils/sanitize_curies.py	Formatting-only changes (line wrapping).
kg_microbe/utils/robot_utils.py	Formatting-only in list-comprehension (but reveals an existing isinstance issue).
kg_microbe/utils/pandas_utils.py	Formatting-only changes (line wrapping).
kg_microbe/utils/ontology_utils.py	Formatting-only changes (function signatures and wrapping).
kg_microbe/utils/ner_utils.py	Formatting-only changes (comprehensions).
kg_microbe/utils/microbial_trait_mappings.py	Changes CHEBI category mapping to biolink:ChemicalSubstance.
kg_microbe/utils/mediadive_bulk_download.py	Formatting-only changes (wrapping).
kg_microbe/utils/mapping_file_utils.py	Rewraps URLs and some logic lines; no functional changes intended.
kg_microbe/utils/download_utils.py	Formatting-only changes plus removal of blank lines.
kg_microbe/utils/consolidate_categories.py	Formatting-only changes (wrapping and spacing).
kg_microbe/transform_utils/wallen_etal/wallen_etal.py	Formatting-only changes (wrapping).
kg_microbe/transform_utils/uniprot_trembl/uniprot_trembl.py	Formatting-only changes (function signatures).
kg_microbe/transform_utils/rhea_mappings/rhea_mappings.py	Formatting-only changes + uses parenthesized multi-open context manager.
kg_microbe/transform_utils/ontologies/ontologies_transform.py	Formatting-only changes (wrapping).
kg_microbe/transform_utils/metatraits_gtdb/metatraits_gtdb.py	Adds new GTDB-based MetaTraits transform with local GTDB→NCBITaxon mapping and GTDB CURIE fallback.
kg_microbe/transform_utils/metatraits_gtdb/init.py	Exposes MetaTraitsGTDBTransform.
kg_microbe/transform_utils/metatraits/metatraits.py	Major update: unified chemical resolver + expanded predicates + multiprocessing/chunking + new NCBITaxon adapter logic.
kg_microbe/transform_utils/metatraits/mappings/phenotype_mappings.tsv	Corrects wrong METPO CURIEs and replaces Voges-Proskauer mapping with a custom KGM term.
kg_microbe/transform_utils/mediadive/mediadive.py	Formatting-only changes (wrapping).
kg_microbe/transform_utils/madin_etal/madin_etal.py	Formatting-only changes (wrapping).
kg_microbe/transform_utils/example_transform/example_transform.py	Formatting-only change (wrapping).
kg_microbe/transform_utils/disbiome/disbiome.py	Formatting-only changes (wrapping).
kg_microbe/transform_utils/constants.py	Adds METATRAITS_GTDB and normalizes CHEBI category constants toward biolink:ChemicalSubstance.
kg_microbe/transform_utils/bakta/utils.py	Formatting-only changes (function signature).
kg_microbe/transform_utils/bakta/create_samn_mapping.py	Formatting-only changes (wrapping).
kg_microbe/transform_utils/bakta/bakta.py	Formatting-only changes (wrapping).
kg_microbe/transform_utils/bactotraits/bactotraits.py	Formatting-only changes (wrapping).
kg_microbe/transform_utils/bacdive/bacdive.py	Minor messaging formatting; continues using unified chemical mappings with fallback.
kg_microbe/transform.py	Registers metatraits_gtdb transform.
kg_microbe/run.py	Suppresses a transitive deprecation warning; formatting-only changes in click options.
kg_microbe/query.py	Formatting-only changes (logging call).
kg_microbe/download.py	Formatting-only changes (function signature and comprehension wrapping).
kg_microbe/bactotraits_to_mongo.py	Formatting-only changes (wrapping).
download.yaml	Adds GTDB MetaTraits files + GTDB↔NCBI mapping TSV URLs.
MULTIPROCESSING_IMPLEMENTATION_SUMMARY.md	Adds a narrative summary of multiprocessing design/behavior and configuration.
CLAUDE.md	Documents multiprocessing behavior/configuration for MetaTraits transforms.

Comments suppressed due to low confidence (2)

kg_microbe/utils/microbial_trait_mappings.py:1

• The updated phenotype_mappings.tsv introduces object_source = KGM, but _OBJECT_SOURCE_TO_CATEGORY doesn’t include a KGM entry. If load_microbial_trait_mappings() uses this map directly, loading the mapping file will raise a KeyError (or otherwise fail to assign object_category) when it encounters the new KGM row. Add an explicit KGM mapping (and ensure the loader supports custom prefixes) so the transform can ingest the updated phenotype mappings reliably.
  kg_microbe/utils/robot_utils.py:1
• isinstance(terms, List) uses typing.List, which is not valid for runtime isinstance checks and will raise a TypeError in modern Python. Replace this with a runtime type (e.g., list) or a collections.abc type (e.g., Sequence) while excluding str if needed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Unconditionally calling local_db.unlink() on any adapter initialization failure can delete a valid on-disk database in the face of transient errors (e.g., file locks, partial read, permission issues). To avoid destructive behavior, only unlink when it’s a symlink (or otherwise clearly a generated cache artifact), or rename to a quarantine path and retry; additionally, consider narrowing the exception handling to errors that definitively indicate corruption.

Suggested change

	# Remove invalid file/symlink
	local_db.unlink()
	# Remove invalid symlink, or quarantine regular file instead of deleting
	try:
	if local_db.is_symlink():
	local_db.unlink()
	else:
	quarantine_path = local_db.with_suffix(local_db.suffix + ".quarantine")
	local_db.rename(quarantine_path)
	except OSError:
	# If cleanup fails, continue to fallback mechanisms
	pass

[Copilot]

_merge_worker_outputs() loads all node/edge TSVs into pandas DataFrames and concatenates them, which can easily exceed memory for large MetaTraits outputs (especially edges) and undermines the “streaming writer” intent. Consider a streaming merge strategy (append + external sort/uniq by key columns, chunked pandas reads, or a lightweight SQLite/duckdb staging table) to keep peak memory bounded while still deduplicating.

[Copilot]

Mutating the module-level METATRAITS_INPUT_FILES global to drive GTDB behavior creates implicit side effects and makes the transform order-dependent within a long-lived process (and harder to reason about under multiprocessing/re-entrancy). Prefer making the input file list an instance attribute or overriding a method in the base class that returns the expected file set, so MetaTraitsGTDBTransform doesn’t need to rewrite globals.

Suggested change

	# Call parent class run() method with GTDB-specific file list
	# The parent class handles the actual processing logic
	# We've overridden _search_ncbitaxon_by_label() to use GTDB mapping
	# Temporarily replace parent's input file list with GTDB files
	from kg_microbe.transform_utils.metatraits.metatraits import METATRAITS_INPUT_FILES
	original_files = METATRAITS_INPUT_FILES.copy()
	METATRAITS_INPUT_FILES.clear()
	METATRAITS_INPUT_FILES.extend(METATRAITS_GTDB_INPUT_FILES)
	try:
	# Call parent run() method
	super().run(data_file=data_file, show_status=show_status)
	finally:
	# Restore original file list
	METATRAITS_INPUT_FILES.clear()
	METATRAITS_INPUT_FILES.extend(original_files)
	# Delegate processing to the parent class without mutating
	# any module-level configuration. GTDB-specific behavior is
	# handled by this subclass (e.g., NCBITaxon label resolution).
	super().run(data_file=data_file, show_status=show_status)

[Copilot]

The documentation states multiprocessing is auto-enabled only when “2+ input files exist”, but MetaTraitsTransform.run() now also enables multiprocessing for a single input file via chunk-splitting (_run_parallel_chunked). Update the docs to match the implemented behavior (or adjust the code to match the documented behavior) to prevent confusion and misconfiguration.

Suggested change

	- **Auto-enabled**: Multiprocessing is ON by default when 2+ input files exist
	- **Auto-enabled**: Multiprocessing is ON by default when either (a) 2+ input files exist, or (b) a single large input file is internally chunk-split