Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
9968576
Make the DNASeq injector dormant while dnaseq processing is redone
jbrestel Jul 30, 2026
60a3e36
Stop consuming communitycount, which this branch stopped producing
jbrestel Jul 30, 2026
28b06e4
Inject the DeRisi time series questions instead of hardcoding them
jbrestel Jul 30, 2026
e785390
Drop a dataset-specific attribute from a generic compound search
jbrestel Jul 30, 2026
9ff6094
Add VariationAttributes tuning table
jbrestel Jul 30, 2026
9bcf147
Add a minimal buildable variation record
jbrestel Jul 30, 2026
48f13bc
Add variation classification attributes from VariationFeature
jbrestel Jul 30, 2026
d35e557
Add SNP and Indel allele attributes as separate sections
jbrestel Jul 30, 2026
af9b2aa
Add variation strain and call statistic attributes
jbrestel Jul 30, 2026
d8ccd76
Add variation gene aggregate, per-caller effect rollups, collapsed co…
jbrestel Jul 30, 2026
e3571e2
Fix variation impact sorting and the collapsed-MAF help text
jbrestel Jul 30, 2026
f9264c0
Add variation record overview and default summary columns
jbrestel Jul 30, 2026
dd4efa6
Add variation TranscriptProducts record table
jbrestel Jul 30, 2026
9cc505b
Correct the TranscriptProducts strain_count help text
jbrestel Jul 30, 2026
b19c82d
Add variation PredictedEffects record table
jbrestel Jul 30, 2026
cf0dd13
Categorize variation record attributes and tables
jbrestel Jul 30, 2026
d636b44
Label the overview allele rows by class instead of header rows
jbrestel Jul 30, 2026
51cb41b
Assemble the "allele (frequency)" strings in SQL, not as textAttributes
jbrestel Jul 30, 2026
19b6527
Fix ontology parenting and four review findings
jbrestel Jul 30, 2026
e0bb106
Rename distinct_strain_count to "Called Strain Count (including refer…
jbrestel Jul 30, 2026
2723cda
Point the variation record at the real tuning table
jbrestel Jul 30, 2026
d3616c6
Add the variation record design spec and implementation plan
jbrestel Aug 5, 2026
6be5539
Reconcile the variation plan's progress with what actually shipped
jbrestel Aug 5, 2026
5fd66ca
Add the variation searches design spec (scaffolding + ID search)
jbrestel Aug 5, 2026
f77d97a
Add the variation searches implementation plan
jbrestel Aug 5, 2026
6130ef9
Add variation search scaffolding and the VariationBySourceId search
jbrestel Aug 5, 2026
ff563a5
Disambiguate the spec reference in variationParams.xml
jbrestel Aug 5, 2026
cf87f9d
Import the variation question, query, and param files
jbrestel Aug 5, 2026
eaa4757
Categorize the VariationBySourceId search under Genetic Variation
jbrestel Aug 5, 2026
6961842
Mark the variation searches spec implemented
jbrestel Aug 5, 2026
3dc299a
Add the VariationsByIsolateGroup design spec
jbrestel Aug 5, 2026
c1829d6
Add VariationsByIsolateGroup implementation plan
jbrestel Aug 5, 2026
46e06f4
Point buildNumber at build-71 webservices files
jbrestel Aug 5, 2026
a7a54fa
Add organismVQ.withVariationsTree for the variation searches
jbrestel Aug 5, 2026
1487112
Add hidden eda_sample_table_suffix param and VariationVQ querySet
jbrestel Aug 5, 2026
71bfa81
Add EDA-driven variation_sample_meta filterParam and its two queries
jbrestel Aug 5, 2026
e5946d4
Fix illegal -- inside XML comments in the plan's code blocks
jbrestel Aug 5, 2026
dee13ba
Copy the four HSSS path and threshold params into variationParams
jbrestel Aug 5, 2026
594c502
Fix wdkXml grep patterns in the plan: single quotes, not double
jbrestel Aug 5, 2026
d870ea6
Add VariationsByIsolateGroup process query
jbrestel Aug 5, 2026
13c85bf
Fix the plugin source path in the plan (src/main/java, not lib/java)
jbrestel Aug 5, 2026
7ac703c
Add the VariationsByIsolateGroup question
jbrestel Aug 5, 2026
06cf403
Add the required dynamicAttributes block to the plan's Task 7
jbrestel Aug 5, 2026
cdd33e3
Categorize VariationsByIsolateGroup under Genetic Variation
jbrestel Aug 5, 2026
f5345b4
Design VariationsByLocation and VariationsByGeneIds
jbrestel Aug 5, 2026
7b8501d
Add implementation plan for VariationsByLocation and VariationsByGeneIds
jbrestel Aug 5, 2026
87b5db5
Fix the plan's installed-jar check: read the class, do not grep the l…
jbrestel Aug 5, 2026
6b38902
Add chromosomeOptionalForVariations param and its vocabulary
jbrestel Aug 5, 2026
ef12c77
Add the VariationsByLocation search
jbrestel Aug 5, 2026
45a2e2a
Add the VariationsByGeneIds search
jbrestel Aug 5, 2026
fb976c0
Categorize VariationsByLocation and VariationsByGeneIds
jbrestel Aug 5, 2026
a74be0f
Design VariationsByTwoIsolateGroups, the last of the five snp ports
jbrestel Aug 5, 2026
4656225
Add implementation plan for VariationsByTwoIsolateGroups
jbrestel Aug 5, 2026
9f6aae1
Add the two-group sample filters and four thresholds
jbrestel Aug 5, 2026
f087d78
Fix the plan's expected param count: fourteen, not thirteen
jbrestel Aug 5, 2026
27ba528
Add the VariationsByTwoIsolateGroups search
jbrestel Aug 5, 2026
e131375
Categorize VariationsByTwoIsolateGroups
jbrestel Aug 5, 2026
f41a71b
Design the port of the four remaining Genetic Variation searches
jbrestel Aug 6, 2026
9a31137
Add implementation plan for the Genetic Variation searches port
jbrestel Aug 6, 2026
adb9ebe
Restore the GenesByNgsSnps process query on variationParams
jbrestel Aug 6, 2026
80189c9
Switch the plan's query verification to -showParams
jbrestel Aug 6, 2026
ebfb517
Restore the GenesByNgsSnps question
jbrestel Aug 6, 2026
382386a
Fix the plan's service checks for the webapp context path
jbrestel Aug 6, 2026
0745c44
Use -- not ## for SQL comments in GenesByNgsSnps postCacheUpdateSql
jbrestel Aug 6, 2026
ea74772
Add transitional apidbtuning CNV tables
jbrestel Aug 6, 2026
8aa395e
Correct the webready CNV tables for the next workflow run
jbrestel Aug 6, 2026
2ae20ca
Merge master into dnaseq-merge-experiments
jbrestel Aug 6, 2026
7473351
Order needsUpdateProgram before program in the Pubmed tuning table
jbrestel Aug 6, 2026
2d0a87b
Add CNV organism vocabularies and the cnv_sample_meta filter
jbrestel Aug 6, 2026
8eca4e9
Port GenesByCopyNumber onto the corrected CNV tables
jbrestel Aug 6, 2026
9123b6a
Use percentile_cont, not Oracle median(), in the CNV medians CTEs
jbrestel Aug 6, 2026
6d32f8c
Port GenesByCopyNumberComparison onto the corrected CNV tables
jbrestel Aug 6, 2026
f95b152
Restore the CNV summary columns on both gene CNV searches
jbrestel Aug 6, 2026
340d236
Port SequencesByPloidy onto the corrected chromosome CNV table
jbrestel Aug 6, 2026
afc84f3
Categorize the three CNV searches under Genetic Variation
jbrestel Aug 6, 2026
52a1b82
Fix Task 14's unpassable gate in the port plan
jbrestel Aug 6, 2026
e0bbc54
Retire the dead CNV organism and strain params
jbrestel Aug 6, 2026
63d6e80
Compute ref_cn from the annotation, per chromosome
jbrestel Aug 6, 2026
b13b371
Apply final code review findings
jbrestel Aug 6, 2026
d48ac7d
Add GeneVariationSummary tuning + webready tables
jbrestel Aug 7, 2026
9a38645
Add ungated piN/piS columns for display alongside their denominator
jbrestel Aug 7, 2026
61e49a3
Fix: reference TranscriptAttributes unversioned, not with &1
jbrestel Aug 7, 2026
ed68d81
Wire the gene-record Genetic Variation attributes
jbrestel Aug 7, 2026
c6eddbb
Group the gene variation attributes into six record subsections
jbrestel Aug 7, 2026
1c3bd3b
Rename the genotype-ploidy attribute so it cannot be read as copy number
jbrestel Aug 7, 2026
f3f46c6
Offer the reference strain in the HSSS sample filters
jbrestel Aug 7, 2026
4266421
Show dataset display names in the sample filters, not DS_ IDs
jbrestel Aug 7, 2026
c8a99bd
Rename the Variation record to Variant; display as "Short Variant"
jbrestel Aug 8, 2026
24935d7
Mark the four HSSS searches as SNV-only in their display names
jbrestel Aug 8, 2026
bfb8418
Name the region and gene SNV searches for what they compute
jbrestel Aug 8, 2026
652c1c2
Merge branch 'feature/variant-rename' into dnaseq-merge-experiments
jbrestel Aug 8, 2026
7312822
Rename the GenesByNgsSnps display name to SNV Characteristics
jbrestel Aug 8, 2026
0cf8251
Add GenesByVariantCharacteristics, a precomputed SNV characteristics …
jbrestel Aug 8, 2026
0520b48
Use the transcript record's own variation attributes, not new dynamic…
jbrestel Aug 8, 2026
c319350
Declare span_snp_density on the GenesByNgsSnps process query
jbrestel Aug 8, 2026
1827678
Label the GenesByNgsSnps result columns for what they now compute
jbrestel Aug 8, 2026
052be33
Label the GenesByNgsSnps params for what they now filter
jbrestel Aug 8, 2026
06cca80
Point the GenesByNgsSnps description at the normalized ratio
jbrestel Aug 8, 2026
186d484
Drop the synonymous-site-fraction figures from the GenesByNgsSnps des…
jbrestel Aug 8, 2026
2dea527
Mark the GenesByNgsSnps result columns as per-sample-group
jbrestel Aug 8, 2026
83974cb
Merge remote-tracking branch 'origin/master' into dnaseq-merge-experi…
jbrestel Aug 8, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 23 additions & 17 deletions Model/lib/psql/webready/orgSpecific/ChrCopyNumbers_p.psql
Original file line number Diff line number Diff line change
@@ -1,22 +1,28 @@
:CREATE_AND_POPULATE
-- Mirrors the ChrCopyNumbers tuningTable in apiTuningManager.xml; any edit here
-- MUST be mirrored there. See the port design doc s7.
--
-- Two changes: the dead PANIO_p join is gone, and the source is
-- GenomicSeqAttributes_p rather than TranscriptAttributes_p. Chromosome ploidy
-- is sequence-level, and SequencesByPloidy already reads the sequence table.
SELECT DISTINCT
ta.project_id
, ta.org_abbrev
, current_timestamp as modification_date
, ta.na_sequence_id
, ta.chromosome
sa.project_id
, sa.org_abbrev
, sa.organism
, sa.taxon_id
, current_timestamp as modification_date
, sa.source_id
, sa.na_sequence_id
, sa.chromosome
, ccn.chr_copy_number AS ploidy
, io.input_pan_id
, io.output_pan_id
, regexp_replace(pan.name, '_Ploidy$', '') AS eda_sample_stable_id
FROM apidb.ChrCopyNumber ccn
, :SCHEMA.TranscriptAttributes_p ta
, :SCHEMA.PANIO_p io
WHERE ta.na_sequence_id = ccn.na_sequence_id
AND ta.chromosome IS NOT NULL
AND ccn.protocol_app_node_id = io.output_pan_id
and ta.org_abbrev = ':ORG_ABBREV'
and io.org_abbrev = ':ORG_ABBREV';


:DECLARE_PARTITION;
, study.protocolappnode pan
, :SCHEMA.GenomicSeqAttributes_p sa
WHERE ccn.protocol_app_node_id = pan.protocol_app_node_id
AND sa.na_sequence_id = ccn.na_sequence_id
AND sa.chromosome IS NOT NULL
AND sa.org_abbrev = ':ORG_ABBREV';


:DECLARE_PARTITION;
13 changes: 5 additions & 8 deletions Model/lib/psql/webready/orgSpecific/ChrCopyNumbers_p_ix.psql
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
CREATE index ChrCN_ix
ON :SCHEMA.ChrCopyNumbers_p (org_abbrev, input_pan_id, na_sequence_id)
-- Was two indexes, on (org_abbrev, input_pan_id, na_sequence_id) and
-- (org_abbrev, output_pan_id). Both pan columns are now eda_sample_stable_id,
-- which would make the second a redundant prefix of the first.
CREATE index ChrCN_ix
ON :SCHEMA.ChrCopyNumbers_p (org_abbrev, eda_sample_stable_id, na_sequence_id)
;


CREATE index ChrCN_output
ON :SCHEMA.ChrCopyNumbers_p (org_abbrev, output_pan_id)
;

61 changes: 45 additions & 16 deletions Model/lib/psql/webready/orgSpecific/GeneCopyNumbers_p.psql
Original file line number Diff line number Diff line change
@@ -1,30 +1,59 @@
:CREATE_AND_POPULATE
-- Mirrors the GeneCopyNumbers tuningTable in apiTuningManager.xml, which
-- carries the searches until this file's next workflow run. Any edit here
-- MUST be mirrored there. See
-- docs/superpowers/specs/2026-08-06-genetic-variation-searches-port-design.md s7.
--
-- The PANIO_p join was removed: study.Input has no rows, so PANIO_p is empty
-- and this table came out empty. Organism identity now comes from
-- TranscriptAttributes_p and sample identity from the protocolappnode name.
--
-- ref_cn is computed from the annotation (same ortholog group, same chromosome)
-- and deliberately IGNORES apidb.genecopynumber.ref_copy_number, which the
-- loader computes per sample and which therefore undercounts for partially
-- covered samples. That made ref_cn vary per gene, and hit_medians GROUPs BY it,
-- so WDK rejected the answer with a row-count mismatch.
WITH grp AS (
SELECT DISTINCT ta.gene_source_id, ta.chromosome, oga.group_id
FROM :SCHEMA.TranscriptAttributes_p ta
JOIN apidb.orthologgroupaasequence oga ON oga.aa_sequence_id = ta.aa_sequence_id
WHERE ta.org_abbrev = ':ORG_ABBREV'
), refcn AS (
SELECT a.gene_source_id, count(DISTINCT b.gene_source_id) AS ref_cn
FROM grp a
JOIN grp b ON b.group_id = a.group_id AND b.chromosome = a.chromosome
GROUP BY 1
)
SELECT DISTINCT ':PROJECT_ID' as project_id
, ':ORG_ABBREV' as org_abbrev
, ta.organism
, ta.taxon_id
, current_timestamp as modification_date
, ta.source_id
, ta.gene_source_id
, REGEXP_REPLACE(pan.name, '_[A-Za-z0-9]+ (.+)$', '') AS strain
, regexp_replace(pan.name, '_GeneCNV$', '') AS eda_sample_stable_id
, gcn.haploid_number AS raw_estimate
, gcn.ref_copy_number AS ref_cn
-- 1 = the gene itself, for a gene in no ortholog group (67 such in pfal).
-- CAUTION: ref_cn=1 is also the NORMAL result - 89% of pfal genes have no
-- same-chromosome paralog - so if this table is built before the orthomcl
-- load has run, it is silently indistinguishable from a correct one for most
-- rows. The tuningManager copy guards this with an externalDependency on
-- apidb.OrthologGroupAaSequence; here it is a workflow ORDERING requirement.
, COALESCE(r.ref_cn, 1) AS ref_cn
, CASE WHEN (gcn.haploid_number < 0.01) THEN 0
WHEN (0.01 < gcn.haploid_number AND gcn.haploid_number < 1.85) THEN 1
ELSE round(gcn.haploid_number) END AS haploid_number
, ta.chromosome
, ta.na_sequence_id
, io.input_pan_id
, io.output_pan_id
FROM apidb.genecopynumber gcn
, study.protocolappnode pan
, :SCHEMA.TranscriptAttributes_p ta
, :SCHEMA.PANIO_p io
WHERE gcn.protocol_app_node_id = pan.protocol_app_node_id
AND gcn.na_feature_id = ta.gene_na_feature_id
AND gcn.protocol_app_node_id = io.output_pan_id
AND (ta.gene_type = 'protein coding' or ta.gene_type = 'protein coding gene')
AND ta.org_abbrev = ':ORG_ABBREV'
AND io.org_abbrev = ':ORG_ABBREV';


:DECLARE_PARTITION;
JOIN study.protocolappnode pan
ON pan.protocol_app_node_id = gcn.protocol_app_node_id
JOIN :SCHEMA.TranscriptAttributes_p ta
ON ta.gene_na_feature_id = gcn.na_feature_id
LEFT JOIN refcn r
ON r.gene_source_id = ta.gene_source_id
WHERE ta.gene_type IN ('protein coding', 'protein coding gene')
AND ta.org_abbrev = ':ORG_ABBREV';


:DECLARE_PARTITION;
5 changes: 2 additions & 3 deletions Model/lib/psql/webready/orgSpecific/GeneCopyNumbers_p_ix.psql
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
CREATE INDEX GeneCN_ix
ON :SCHEMA.GeneCopyNumbers_p (org_abbrev, input_pan_id, na_sequence_id)
CREATE INDEX GeneCN_ix
ON :SCHEMA.GeneCopyNumbers_p (org_abbrev, eda_sample_stable_id, na_sequence_id)
;

Loading