fix: handle annotation availability and external labels, reporting only available cell annotations

config/README.md

@@ -325,7 +325,16 @@ SpaceBlocks allows you to input:
 
 - **Externally assembled h5ad AnnData objects** — run `mode: decoupled` and point `contract_dir` at the directory of pre-built contract h5ads. The heads are skipped; the core validates and analyses them directly.
 - **Pre-computed clusters / annotations** — set `use_precomputed_clusters: true` and `precomputed_metadata_dir` to reuse Leiden clusters and metadata; for niches, set `spatial_niches.use_precomputed: true` with `spatial_niches.niche_dir`.
-- **Externally annotated data** — set `external_annotation.enabled: true` with `external_annotation.column`, and choose whether to keep or discard unannotated barcodes downstream via `external_annotation.keep_unannotated` (see [section 5](#5-key-sections)).
+- **Externally annotated data** — set `external_annotation.enabled: true` and use
+  `external_annotation.column` for the source column. Provide it in each
+  `metadata_{sample}.tsv`, or, in `mode: decoupled`, directly in each contract
+  h5ad's `obs`. The source column may have any name; SpaceBlocks copies it to the
+  canonical `cell_type_external` column used by downstream rules. Choose
+  whether to keep or discard unannotated barcodes via
+  `external_annotation.keep_unannotated` (see [section 5](#5-key-sections)). With
+  `keep_unannotated: true`, a mixture of real labels and `Unannotated` remains an
+  active annotation and both are shown in overview and composition plots. Only a
+  column containing no real labels is omitted from annotation plots and reports.
 
 These files are generated during the run, and can be shared with minimum effort to reproduce downstream results from raw data.
 

config/config.yaml

@@ -190,10 +190,10 @@ analysis:
 snakemake_cell_markers: config/snakemake_cell_markers.tsv
 # (default) cluster → cell-type TSV (fill template from generate_annotation_template)
 cluster_annotations: config/cluster_annotations.tsv
-# (optional) use external tsv with annotations
+# (optional) external labels from metadata TSVs, or from contract h5ad obs in decoupled mode
 external_annotation:
   enabled: false
-  column: manual_celltype
+  column: manual_celltype  # arbitrary source name; normalized to cell_type_external
   keep_unannotated: true
 ingest_ref: "/path/to/ingest_ref_data/bcc_and_normal_Garnier_PNAS_2024.h5ad"  # optional annotated scRNA-seq reference (ingest_ref)
 ingest_ref_label_key: 04_celltypes

docs/rules.md

@@ -155,7 +155,7 @@ This second CoreBlock includes rules from sample annotation to integration and d
 
 #### `annotate_cells`
 Adds cell-type annotations from a manually curated TSV (cluster → cell type), or from an external annotation TSV (barcode → cell type).
-Writes `adata_{sample}_annotated.h5ad` plus annotation plots and composition barplots.
+Writes `adata_{sample}_annotated.h5ad` plus annotation plots and composition barplots. The output records which annotation columns contain real labels; placeholder-only columns such as an all-`Unannotated` `cell_type_tsv` are retained for compatibility but are not treated as available annotations. The external source column configured by `external_annotation.column` may have any name and is copied to the canonical `cell_type_external` column. It can come from `metadata_{sample}.tsv` or, in decoupled mode without external metadata, directly from the contract h5ad's `obs`. When `keep_unannotated: true`, mixed real and `Unannotated` labels remain active and the unannotated cells are retained in overview and composition plots.
 
 #### `integrate_samples`
 Concatenates all annotated samples, runs Harmony batch correction on the specified variable (`integration.integrate_key`), geosketch subsampling, and composition barplots.
@@ -189,7 +189,7 @@ Generates integrated (Harmony) and unintegrated results, with UMAPs coloured by
 #### `sample_report`
 Assembles a compact, multi-page PDF with graphical outputs per sample, ideal to share with internal and external collaborators.
 
-Each page contains the most important data-descriptive information: cluster / annotation / region / niche UMAPs and matching spatial maps, bar plots for absolute and relative composition, and a dot plot with the top-10 expressed markers per cell type.
+Each page contains the most important data-descriptive information: cluster / annotation / region / niche UMAPs and matching spatial maps, bar plots for absolute and relative composition, and a dot plot with the top-10 expressed markers per cell type. Only annotation sources containing real labels are shown; in runs without a manual TSV annotation, the external or ingest annotation also drives cell-type composition.
 
 ---
 

workflow/Snakefile

@@ -239,7 +239,8 @@ RUN_DE = bool(ANALYSIS.get("run_pseudobulk_de", False))
 
 RUN_LEIDEN_ANALYSIS = bool(ANALYSIS.get("run_leiden_analysis", True))
 
-# External annotation: fail fast if enabled but the column is not present in every sample's metadata
+# External annotation: validate metadata headers now, or decoupled h5ad obs per sample
+# in validate_input when no external metadata directory is configured.
 check_external_annotation()
 
 # Consistency: analysing the external annotation downstream requires it to be produced.

workflow/rules/common.smk

@@ -58,9 +58,9 @@ def _expand_base_dir(node, root="config"):
 # ── Small derivations called from the Snakefile globals block ────────────────
 def check_external_annotation():
     """Fail fast (at parse time, before any job) when external_annotation is enabled
-    but not fully in place: the annotation column must be present in the metadata for
-    EVERY sample. External labels can only come from precomputed_metadata_dir (the
-    pipeline's own metadata does not carry them), so that dir is required."""
+    but not fully in place. Metadata TSV columns are checked here; in decoupled mode,
+    an absent metadata directory means the configured column will instead be checked
+    in each contract h5ad by validate_input."""
     cfg = config.get("external_annotation", {}) or {}
     if not cfg.get("enabled", False):
         return
@@ -71,10 +71,13 @@ def check_external_annotation():
         )
     meta_dir = config.get("precomputed_metadata_dir", "") or ""
     if not meta_dir:
+        if IS_DECOUPLED:
+            return
         sys.exit(
             "[config error] external_annotation.enabled requires 'precomputed_metadata_dir' "
             f"to point at a directory of metadata_{{sample}}.tsv files carrying the '{col}' "
-            "column (the pipeline's own metadata does not contain external labels)."
+            "column. In decoupled mode only, the column may instead be stored directly "
+            "in each contract h5ad's obs."
         )
     missing_file, missing_col = [], []
     for s in SAMPLE_IDS:
@@ -470,10 +473,11 @@ def _geojson_validation_report(sample):
 def _external_metadata_input(wildcards):
     """Tracked per-sample external metadata, or no input when the feature is off."""
     cfg = config.get("external_annotation", {}) or {}
-    if not cfg.get("enabled", False):
+    metadata_dir = config.get("precomputed_metadata_dir", "") or ""
+    if not cfg.get("enabled", False) or not metadata_dir:
         return []
     return os.path.join(
-        config.get("precomputed_metadata_dir", "") or "",
+        metadata_dir,
         f"metadata_{wildcards.sample}.tsv",
     )
 

workflow/schemas/config.schema.yaml

@@ -501,13 +501,19 @@ properties:
         default: false
       column:
         type: string
-        description: obs/metadata column holding the external label.
+        description: >-
+          Source column holding the external label, either in
+          metadata_{sample}.tsv or, in decoupled mode without external metadata,
+          directly in the contract h5ad obs. The name is arbitrary; SpaceBlocks
+          copies it to the canonical cell_type_external obs column downstream.
       keep_unannotated:
         type: boolean
         default: true
         description: >-
           false → keep only externally-annotated cells and skip pipeline QC (external
-          labels drive QC); true → normal QC, labels overlaid at annotation.
+          labels drive QC); true → normal QC, labels overlaid at annotation and
+          cells labelled Unannotated retained. A mixed labelled/Unannotated column
+          remains available for plots; only placeholder-only columns are omitted.
 
   contract:
     type: ["object", "null"]

workflow/scripts/annotate_cells.py

@@ -32,7 +32,11 @@
     _here = os.getcwd()
 sys.path.insert(0, _here)
 from composition_barplots import composition_pair, find_niche_column
-from external_metadata import normalized_labels, optional_input_path, read_cell_metadata
+from annotation_utils import (
+    normalize_annotation_labels,
+    record_active_annotation_columns,
+)
+from external_metadata import optional_input_path, read_cell_metadata
 
 
 # ── Logging ──────────────────────────────────────────────────────────────────
@@ -256,77 +260,98 @@ def generate_annotation_plots(adata, annot_key, label, plots_dir, sample_id,
         annot_col, resolution = find_sample_column(annot_df, sample_id)
 
         if annot_col is None:
-            log.warning("Sample '%s' not found in TSV. All cells → 'Unannotated'.", sample_id)
-            adata.obs["cell_type_tsv"] = "Unannotated"
-            Path(out_adata_path).parent.mkdir(parents=True, exist_ok=True)
-            adata.write(out_adata_path)
-            sys.exit(0)
-
-        log.info("Using column '%s' (resolution: %s)", annot_col,
-                 resolution if resolution else "not specified")
-
-        if resolution is not None:
-            leiden_key = f"leiden_{resolution.replace('.', '_')}"
+            log.warning(
+                "Sample '%s' not found in cluster_annotations TSV. "
+                "cell_type_tsv set to 'Unannotated'; continuing with any "
+                "external/ingest annotation.",
+                sample_id,
+            )
+            adata.obs["cell_type_tsv"] = pd.Categorical(
+                ["Unannotated"] * adata.n_obs
+            )
         else:
-            leiden_cols = [c for c in adata.obs.columns if c.startswith("leiden_")]
-            if leiden_cols:
-                leiden_key = leiden_cols[0]
-            else:
-                raise ValueError("No leiden columns found in adata.")
+            log.info("Using column '%s' (resolution: %s)", annot_col,
+                     resolution if resolution else "not specified")
 
-        if leiden_key not in adata.obs.columns:
-            available = [c for c in adata.obs.columns if c.startswith("leiden_")]
-            raise ValueError(
-                f"Leiden column '{leiden_key}' not found. Available: {available}."
+            if resolution is not None:
+                leiden_key = f"leiden_{resolution.replace('.', '_')}"
+            else:
+                leiden_cols = [c for c in adata.obs.columns if c.startswith("leiden_")]
+                if leiden_cols:
+                    leiden_key = leiden_cols[0]
+                else:
+                    raise ValueError("No leiden columns found in adata.")
+
+            if leiden_key not in adata.obs.columns:
+                available = [c for c in adata.obs.columns if c.startswith("leiden_")]
+                raise ValueError(
+                    f"Leiden column '{leiden_key}' not found. Available: {available}."
+                )
+
+            cluster_to_celltype = annot_df[annot_col].dropna().to_dict()
+            cluster_to_celltype = {str(k): str(v) for k, v in cluster_to_celltype.items()}
+            log.info("Cluster → cell type mapping (%d entries): %s",
+                     len(cluster_to_celltype), cluster_to_celltype)
+
+            adata.obs["cell_type_tsv"] = (
+                adata.obs[leiden_key].astype(str)
+                .map(cluster_to_celltype)
+                .fillna("Unannotated")
+                .astype("category")
             )
 
-        cluster_to_celltype = annot_df[annot_col].dropna().to_dict()
-        cluster_to_celltype = {str(k): str(v) for k, v in cluster_to_celltype.items()}
-        log.info("Cluster → cell type mapping (%d entries): %s",
-                 len(cluster_to_celltype), cluster_to_celltype)
-
-        adata.obs["cell_type_tsv"] = (
-            adata.obs[leiden_key].astype(str)
-            .map(cluster_to_celltype)
-            .fillna("Unannotated")
-            .astype("category")
-        )
-
-        tsv_counts = adata.obs["cell_type_tsv"].value_counts()
-        log.info("TSV annotation distribution:\n%s", tsv_counts.to_string())
+            tsv_counts = adata.obs["cell_type_tsv"].value_counts()
+            log.info("TSV annotation distribution:\n%s", tsv_counts.to_string())
 
     # ── 2. External annotation (from metadata TSV column) ────────────────
     # When enabled the pipeline MUST find the column for this sample — a missing
     # source or column is a hard error (fail loudly rather than silently skipping),
     # so a run only proceeds when the external labels are in place for every sample.
-    ext_enabled = False
     if isinstance(EXT_ANNOT_CFG, dict) and EXT_ANNOT_CFG.get("enabled", False):
         ext_col = EXT_ANNOT_CFG.get("column", "")
         if not ext_col:
             raise ValueError("external_annotation.enabled is true but no 'column' is set in config.")
-        if not external_metadata_path:
-            raise FileNotFoundError(
-                f"external_annotation enabled but no metadata found for sample "
-                f"'{sample_id}'.")
+        # Prefer the explicitly supplied metadata TSV. In decoupled mode, a custom
+        # source column may instead already be present in the contract h5ad and is
+        # preserved by preprocessing.
+        if external_metadata_path:
+            _ext_source = external_metadata_path
+            log.info(
+                "Loading external annotation from '%s' in %s …",
+                ext_col,
+                _ext_source,
+            )
+            saved = read_cell_metadata(
+                _ext_source, required_columns=(ext_col,)
+            )
+            external_labels = saved[ext_col].reindex(
+                adata.obs_names.astype(str)
+            )
+        elif ext_col in adata.obs.columns:
+            _ext_source = f"processed contract h5ad obs['{ext_col}']"
+            log.info("Loading external annotation from %s …", _ext_source)
+            external_labels = adata.obs[ext_col]
+        else:
+            raise ValueError(
+                f"external_annotation column '{ext_col}' not found for sample "
+                f"'{sample_id}': no matching precomputed metadata column or h5ad "
+                "obs column is available"
+            )
 
+        adata.obs["cell_type_external"] = normalize_annotation_labels(
+            external_labels
+        ).astype("category")
+        adata.uns["external_annotation_source_column"] = ext_col
+        n_external_unannotated = int(
+            (adata.obs["cell_type_external"] == "Unannotated").sum()
+        )
         log.info(
-            "Loading external annotation from '%s' in %s …",
+            "  External annotation '%s' copied to 'cell_type_external': %d types "
+            "(%d cells unmatched → Unannotated)",
             ext_col,
-            external_metadata_path,
-        )
-        saved = read_cell_metadata(
-            external_metadata_path, required_columns=(ext_col,)
-        )
-        labels = normalized_labels(saved, ext_col)
-        adata.obs["cell_type_external"] = (
-            labels.reindex(adata.obs_names.astype(str))
-            .fillna("Unannotated")
-            .astype("category")
+            adata.obs["cell_type_external"].nunique(),
+            n_external_unannotated,
         )
-        ext_enabled = True
-        log.info("  External annotation: %d types (%d cells unmatched → Unannotated)",
-                 adata.obs["cell_type_external"].nunique(),
-                 int((adata.obs["cell_type_external"] == "Unannotated").sum()))
 
     # ── 2b. Spatial niche labels (from spatial_niches rule / external) ───
     _ni = getattr(snakemake.input, "spatial_niche", "")
@@ -363,11 +388,25 @@ def generate_annotation_plots(adata, annot_key, label, plots_dir, sample_id,
     # ── 3. Plots ─────────────────────────────────────────────────────────
     log.info("Generating annotation plots …")
 
+    active_annotations = record_active_annotation_columns(adata)
+    log.info("Active annotation columns: %s", active_annotations or "none")
+    if (
+        isinstance(EXT_ANNOT_CFG, dict)
+        and EXT_ANNOT_CFG.get("enabled", False)
+        and "cell_type_external" not in active_annotations
+    ):
+        log.warning(
+            "External annotation is enabled, but '%s' contains no non-placeholder "
+            "labels matching this sample. No external-annotation plots will be made.",
+            EXT_ANNOT_CFG.get("column", ""),
+        )
+
     # Apply custom palettes if configured
     if isinstance(ANNOTATION_COLORS, dict):
-        for obs_key in ["cell_type_tsv", "cell_type_ingest", "cell_type_external"]:
+        for obs_key in active_annotations:
             cd = ANNOTATION_COLORS.get(obs_key, {})
             if cd and obs_key in adata.obs.columns:
+                adata.obs[obs_key] = adata.obs[obs_key].astype("category")
                 cats = adata.obs[obs_key].cat.categories
                 adata.uns[f"{obs_key}_colors"] = [cd.get(str(c), "#cccccc") for c in cats]
 
@@ -378,22 +417,20 @@ def generate_annotation_plots(adata, annot_key, label, plots_dir, sample_id,
             REGION_COLORS.get(str(c), "#cccccc") for c in cats
         ]
 
-    generate_annotation_plots(adata, "cell_type_tsv", "tsv", plots_dir,
-                              sample_id, DE_N_GENES, library_id)
-
-    # If ingest labels exist, plot those too
-    if has_ingest:
-        generate_annotation_plots(adata, "cell_type_ingest", "ingest", plots_dir,
-                                  sample_id, DE_N_GENES, library_id)
-
-    if ext_enabled:
-        generate_annotation_plots(adata, "cell_type_external", "external", plots_dir,
-                                  sample_id, DE_N_GENES, library_id)
+    annotation_labels = {
+        "cell_type_tsv": "tsv",
+        "cell_type_ingest": "ingest",
+        "cell_type_external": "external",
+        "cell_type_refined": "refined",
+    }
+    for annot_col in active_annotations:
+        generate_annotation_plots(
+            adata, annot_col, annotation_labels.get(annot_col, annot_col),
+            plots_dir, sample_id, DE_N_GENES, library_id,
+        )
 
     # Side-by-side comparison (dynamic number of panels)
-    annot_cols_present = [c for c in ["cell_type_tsv", "cell_type_ingest",
-                                       "cell_type_external"]
-                          if c in adata.obs.columns]
+    annot_cols_present = active_annotations
     n_panels = len(annot_cols_present)
     if n_panels >= 2:
         fig, axes = plt.subplots(1, n_panels, figsize=(8 * n_panels, 6),
@@ -423,9 +460,7 @@ def generate_annotation_plots(adata, annot_key, label, plots_dir, sample_id,
             log.warning("Spatial comparison plot failed: %s", e)
 
     # ── Per-sample composition barplots (sample / region / niche) ────────
-    annot_cols_present = [c for c in ["cell_type_tsv", "cell_type_ingest",
-                                      "cell_type_external"]
-                          if c in adata.obs.columns]
+    annot_cols_present = active_annotations
 
     # sample column (one bar per sample; here a single sample). Per-sample adatas
     # carry "sample"; "sample_batch" is an integration-time concat label and does