Begin v3 with IGVF uniform pipeline processed data

[adamklie]

Context

For igvf-data/igvf_sc-islet_10X-Multiome, start v3 using IGVF uniform pipeline.

Tasks

• Sort out the R2 barcode issue for element samples
• Download new data and run sample QC (all 54 samples, RNA + ATAC QC complete)
• RNA integration + cell-type annotation (298,085 cells, 4 populations; Harmony round 2)
• v2 vs v3 comparison audit (scratch/2026_04_09/v3_uniform_pipeline/4_comparison/ — barcodes, cell-type composition, MT-gene reference audit)
• v3 ATAC merge + integration
• v3 peak calling (tracked under Finalize peak calling #5)
• v3 differential analysis (tracked under Finalize DESeq2 pseudobulk methodology for RNA and ATAC #4, once methodology locked)

[adamklie]

Dependencies

• Blocked by Finalize DESeq2 pseudobulk methodology for RNA and ATAC #4 (needs finalized DESeq2 methodology to apply to v3)

[adamklie]

Started to handle the barcode issue here igvf-data/igvf_sc-islet_10X-Multiome/scratch/2026_03_18

The 20 element samples R2 files have been reset. We now need to reupload the rev comped barcodes using igvf_utils

We also need to create some reproducible code that does the whole fix here: igvf-data/igvf_sc-islet_10X-Multiome/bin/10_submission. It's a similar situation to the padded Ns thing (more info here: igvf-data/igvf_sc-islet_10X-Multiome/scratch/2026_02_23)

[adamklie]

Implementation: R2 Barcode Fix + Submission Documentation

Problem

10 samples sequenced on Element AVITI have ATAC R2 barcodes in forward orientation. The IGVF uniform pipeline expects reverse complement (Illumina default). These 10 samples fail during barcode matching.

What was done

1. Submission documentation (igvf-data/SUBMISSION.md)

• Full guide to the 10X Multiome IGVF portal submission pipeline
• Covers all 5 steps in bin/10_submission/, plus barcode fixes
• Documents common issues (N-padding, R2 orientation) and how to diagnose/fix
• Portal API reference and metadata hierarchy

2. Revcomp fix added to 3_portal_submission_tables.ipynb

• Added a new section "Handling R2 barcode orientation for AVITI-sequenced samples" following the same pattern as the existing N-padding fix
• Creates sequence_files_revcomp.txt with both N-padding AND revcomp fixes applied
• Uses the already-generated revcomped FASTQs at fastq/patched_revcomp/ (20 files)
• Calculates new MD5 checksums and updates aliases with _revcomp suffix

Files modified/created

• igvf-data/SUBMISSION.md — new submission documentation
• bin/10_submission/3_portal_submission_tables.ipynb — added 6 cells for revcomp fix (cells 69-74)

Next steps

• Run the updated notebook to generate sequence_files_revcomp.txt
• Upload patched files to IGVF portal (coordinate with DACC if needed)
• Request pipeline re-run on the 10 affected analysis sets

Reference

• Exploratory work: scratch/2026_03_18/confirm_and_fix_barcodes.ipynb
• N-padding precedent: scratch/2026_02_23/
• Patched files: fastq/patched_revcomp/ (20 files, 10 samples × 2 lanes)

[adamklie]

Status Update (2026-03-23)

R2 Barcode Fix — COMPLETED

• Confirmed barcode orientation issue across all 54 samples (10 AVITI = Forward, 44 Illumina = RevComp Last 16)
• Reverse-complemented FASTQs generated (20 files at fastq/patched_revcomp/)
• Barcode verification: 20/20 PASS against seqspec onlist (IGVFFI7587TJLC, 736k barcodes)
• Sample swap check: every revcomp file's first read = RC(original first read)
• Portal upload: 20/20 files re-uploaded to existing accessions via conn.upload_file()
• All files were upload_status: invalidated, status: in progress — eligible for re-upload

Submission Documentation

• Created igvf-data/SUBMISSION.md — full 10X Multiome submission pipeline guide
• Added revcomp section to bin/10_submission/3_portal_submission_tables.ipynb

Files created

• scratch/2026_03_23/barcode_verification/verify_revcomp_barcodes.py — verification script
• scratch/2026_03_23/barcode_verification/barcode_verification_results.csv — results
• scratch/2026_03_23/barcode_verification/reupload_revcomp_files.py — upload script
• scratch/2026_03_23/barcode_verification/2026_03_23_portal_submission.sh — submission wrapper
• igvf-data/SUBMISSION.md — documentation

Remaining for v3

• Wait for portal validation of re-uploaded files
• Request uniform pipeline re-run on 10 affected analysis sets
• Download processed data and run sample QC
• Integration, peak calling, differential analysis on v3 data

[adamklie]

Status Update (2026-04-09)

Uniform pipeline — COMPLETE for 10 re-run samples

Portal validation passed and uniform pipeline has finished for the 10 AVITI samples with corrected R2 barcodes.

Next steps

1. Download uniform pipeline outputs for the 10 samples
2. Sample QC — run through pipelines/multiome-sample-qc/
3. Cell annotation — run through pipelines/multiome-cell-annotation/
4. Formal comparison to CellRanger v2:
   • Compare cell annotations (old CellRanger vs new uniform pipeline) for the 10 overlapping samples
   • Compare RNA embeddings/UMAPs
   • Compare ATAC embeddings/UMAPs
   • Quantify concordance: cell type proportions, marker gene expression, peak accessibility
5. Decision point: do we proceed with uniform pipeline data for all samples, or stick with CellRanger?

Remaining for full v3

• Once comparison validates uniform pipeline: download + process remaining 44 samples
• Repeat full downstream pipeline (integration, peak calling, DE) on v3 data

[adamklie]

Status Update (2026-04-09) — Detailed Plan

Prior work (scratch/2026_02_28/)

Significant work already done on 38 samples that had uniform pipeline outputs at the time:

• Preprocessing (Ensembl → gene symbol mapping) — 38 samples
• RNA QC — 38 samples
• RNA integration (merge + normalize + reduce_dimensions) — 38 samples
• QC comparison notebook (cell counts, barcode Jaccard — median 0.85)
• ATAC QC — never started
• Cell annotation — never run
• ATAC integration — never run

Important: The 10 AVITI samples in that run had wrong R2 barcodes. Those have since been fixed and re-run through the uniform pipeline. The 10 AVITI results from 2026_02_28 are stale; the 28 Illumina sample results should still be valid.

Execution plan (scratch/2026_04_09/v3_uniform_pipeline/)

Step 1: Set up workspace
Move reusable code/metadata from scratch/2026_02_28/. Don't copy stale AVITI results.

Step 2: Download updated AVITI samples (10 samples)
Download re-run uniform pipeline outputs. Preprocess h5ads.
QC: verify barcode orientation is correct.

Step 3: RNA QC (10 new samples only)
Run RNA QC on re-downloaded samples. Keep 28 Illumina results from prior run.
QC: cell counts, barcode Jaccard, doublet rates, gene count distributions.

Step 4: ATAC QC (all 38 samples)
Run AMULET + ATAC QC — never done before.
QC: TSS enrichment, fragment length distributions, ATAC-RNA barcode overlap.

Step 5: Integration (RNA + ATAC)
Re-run RNA integration with corrected AVITI samples. Run ATAC integration.
QC: UMAPs by batch/condition/timepoint, batch mixing metrics, cluster stability.

Step 6: Cell annotation
Run cell annotation pipeline.
QC: marker gene expression, cell type proportions, annotation confidence.

Step 7: Formal comparison to v2 (decision point)

• Cell type label concordance (per-barcode, 16-mer matching)
• Embedding comparison (RNA + ATAC UMAPs side by side)
• Pseudobulk correlation (v2 vs v3 per cell type)
• Cell type proportions comparison
• Decision: proceed with v3 for all samples, or stay with v2?

[adamklie]

Status Update (2026-04-09) — Execution Progress

Completed today

• Downloaded 16 missing uniform pipeline samples from portal (all 54 now available)
• Preprocessed h5ads (Ensembl → gene symbols) for 16 new samples
• Consolidated all 54 preprocessed + 38 old RNA QC results into unified workspace at scratch/2026_04_09/v3_uniform_pipeline/
• Fixed cellcommander editable install (broke during project reorg)
• RNA QC completed — job 9061665, 16/16 tasks successful, 5-10 min each

CellCommander fix: AMULET without h5ad

The uniform pipeline provides RNA h5ad and ATAC fragment files separately (no combined multiome h5). The old AMULET workflow loaded the CellRanger raw_feature_bc_matrix.h5 to get barcodes, but AMULET only needs the fragment file — the h5ad was just used as a barcode container.

Changes to tools/CellCommander/cellcommander/detect_doublets/:

• argparser.py: --input_h5ad_path now optional (was required=True)
• cli.py: validate_args handles input_file=None
• run.py: Added amulet_only code path — when method=amulet and no h5ad, runs directly on fragments and returns (no UMAP, no h5ad output)
• amulet.py: run_amulet() accepts fragments_file param directly; when adata=None, saves results as standalone CSVs and txt files (same format downstream code expects: doublet_barcodes.txt, amulet_scores.csv, etc.)

Backward compatible — old invocations with --input_h5ad_path still work unchanged.

Running now

• AMULET — job 9091057, 54 tasks, 10 concurrent, 8G mem (fragments-only, no h5ad loading)

Next steps

• After AMULET: consolidate results, run ATAC QC
• Integration (RNA + ATAC) with all 54 samples
• Cell annotation
• Formal comparison to v2