Rename the Variation record to Variant; add a precomputed SNV characteristics search - #212
Open
jbrestel wants to merge 101 commits into
Open
Rename the Variation record to Variant; add a precomputed SNV characteristics search#212jbrestel wants to merge 101 commits into
jbrestel wants to merge 101 commits into
Conversation
IsolatesHTS no longer describes how DNASeq datasets are processed on this branch,
and both of its methods fail on the new shape:
- injectTemplates() derived per-sample gbrowse/jbrowse databases from getSampleList(),
which keys samples on organismAbbrev + datasetClassCategory + experimentName. The
new dnaseqExperiment class carries an empty category while its samples span two
("Genetic variation" for SNPs, "Structural variation" for CNVs), so the key cannot
match and getSampleList() throws "No sample names found for dataset ...".
- addModelReferences() registered SnpQuestions / SnpRecordClasses references. That XML
is not in the compiled model and is to be superseded by the variation record.
Both bodies are commented out rather than deleted, and the class stays in place as a
no-op so the presenters naming it keep resolving. getPropertiesDeclaration() is left
alone: presenters still supply hasCNVData, and DatasetInjector.addPropValues validates
supplied props against the declaration.
Note the deeper issue for whoever writes the replacement: keying the experiment ->
sample lookup on a single datasetClassCategory cannot work for an experiment that
aggregates SNP and CNV samples, even once dnaseqExperiment declares a category.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Commit 643ddb4 commented out the CommunityCount intermediate table and the communityCount column of OrganismAttributes in apiTuningManager.xml (it read from a hardcoded usercomments_userdb_betan.mappedComment), but left every consumer in place, so model load failed with: Database error while attempting to parse sqlQuery OrganismAttributes.organismAttrs: ERROR: column oa.communitycount does not exist Consumers brought into line with the producer, following the snpcount/popsetcount precedent already in these files: - organismAttributeQueries.xml: the communitycount column declaration and its select expression are commented out. - organismRecords.xml: the communitycount columnAttribute is commented out. - organismQuestions.xml: communitycount removed from five summary attribute lists, which is mandatory since WDK resolves every attribute a summary names. community_flag is kept, as a literal 'none' rather than derived from the missing column. That is exactly what "CASE oa.communitycount WHEN 0 THEN 'none'" produced for a zero count, so the organism page renders as it would with no comments, rather than losing an attribute that organismRecords.xml:305 and its red-dot <img> still use. Deriving it from NULL would have been worse than removing it: CASE NULL WHEN 0 falls through to ELSE, so every organism would have advertised comments it does not have. The individuals.txt ontology entry is left alone, matching how snpcount and popsetcount were handled: the category tree tolerates entries for absent attributes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These two questions were written by hand in geneQuestions.xml while the attribute
their summaries name is generated by the presenter:
summary="...,exprGraphAttrpfal3D7_microarrayExpression_Derisi_TimeSeries_RSRC_expr_graph"
Expression.java:136 builds that name and injects the textAttribute that defines it. So
a hand-written question depended on generated output, coupled only by a name string
with nothing enforcing that both exist. On an instance where this dataset is not
loaded, the presenter is skipped and the question survives, referencing an attribute
nobody generated:
Summary attribute field [exprGraphAttrpfal3D7_..._expr_graph] defined in question
[GeneQuestions.GenesByMicroarraypfal3D7_..._RSRC] is invalid
The questions are curated rather than generic -- they use the PFTimeSeries vocabulary
queries, an extra samples_fc_comp_generic param, and fold_change_chosen_display in
place of fold_change_chosen -- which is why the presenter sets hasMultipleSamples,
hasPercentileData and hasPageData to false and why they were hardcoded in the first
place. So they move to their own microarrayDeRisiTimeSeries.dst rather than being
folded into the generic expression templates, and MicroarrayDeRisiTimeSeries injects
them. Question and attribute now appear and disappear together.
Only three values are parameterised -- ${datasetName} for the question and graph
attribute names, ${includeProjects} and ${includeProjectsExcludeEuPathDB} for scoping,
both already set by Expression.java to precisely the literals that were hardcoded.
Display strings stay literal because they describe this experiment. addModelReferences
now derives the question names from getDatasetName() rather than spelling out the
dataset, so the references cannot drift from what is injected.
Not yet verified: that the generated question is equivalent to the deleted static XML.
Nothing is generated on an instance where this dataset is absent, so this branch can
only show that its absence is handled. Before this reaches a full-data site, run a
build with the dataset declared and diff the generated question against the static
version in this commit's parent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CompoundQuestions.CompoundsByFoldChange named one dataset's graph column in its summary:
summary="...,metaboliteProfiles_LlinasMetabolites_RSRC_metabolite_graph"
That attribute is injected per metabolomics dataset by metabolomics.dst, so naming a
particular dataset's copy in a generic search has two consequences: model load fails
wherever that dataset is not loaded ("Summary attribute field [...] is invalid"), and
where it is loaded the search shows that one dataset's graph regardless of what the
user searched for.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One row per variant locus (4,390,908 in unidb_shu_a), backing a new "variation" WDK record that replaces the deprecated snp record. Holds only columns that are derived, aggregated, or require a join. Intrinsic per-locus facts -- variant_type, is_coding, all 22 snp_*/indel_* allele columns, the strain counts -- are read directly from apidb.VariationFeature, which is already one row per locus with a unique source_id. That split keeps ~570MB of derived strings out of the tuning table and gives every column one known source. Both internalDependency elements are load-bearing: built before TranscriptAttributes the gene aggregate is silently empty rather than an error, and built before GenomicSeqAttributes the project_id/organism join drops every row. Notable derivations: - gene_ids is a string_agg, not a lookup: 25,545 loci overlap more than one gene, so the multi-gene case is correct by construction. - most_severe_impact and effect_summary are split per caller, because snpeff and product_call disagree on 19% of paired calls and that disagreement is content rather than noise. - collapsed_allele renders both classes for a MIXED locus (e.g. "A>C; A>AC"), which a single major/minor pair would have destroyed. The SELECT was verified against unidb_shu_a before being committed: 4,390,908 rows, 100% join coverage against GenomicSeqAttributes, and the expected values at three loci covering SNV, INDEL, and MIXED. Design: docs/superpowers/specs/2026-07-30-variation-record-design.md and plans/2026-07-30-variation-record.md in agentic-veupath-dev. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Record class, alias query, and the tuning-backed identity/location attributes. Builds green and loads by ID; 13 attributes registered. sequence_source_id and location are exposed as first-class non-internal attributes rather than being parsed back out of source_id, because they are the coordinate a future VCF/tabix lookup and an EDA sample join both key on. Two things differ from the written plan, both discovered by building it: - The snp imports in apiCommonModel.xml sit inside an "UNCOMMENT WHEN SNPS are AVAILABLE" comment block, so inserting after them would have silently disabled the variation record. Imports are placed after that block instead. - variationTableQueries.xml is not created yet. WDK's RELAX NG schema requires a querySet to contain at least one query, so the intended empty-but-valid placeholder fails validation. The file arrives with its first real query. Temporarily reads jbrestel.VariationAttributes; see Task 14 of the plan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First attribute query reading apidb.VariationFeature directly, per the sourcing rule: intrinsic per-locus facts come from the base table, only derived/joined columns come from the tuning table.
Both allele classes are preserved rather than collapsed: 129,850 MIXED loci populate snp_* and indel_* simultaneously and describe the same strains two ways, so a single major/minor pair would silently lose half the data. HGVS attributes carry help text explaining that the major-allele HGVS is empty whenever the major allele equals the reference.
Grouped together deliberately: these aggregates are what the future per-strain VCF-backed table will detail, so it lands beside them.
…lumns Gene linkage is an aggregate rather than a lookup, so the 25,545 multi-gene loci are correct by construction instead of by a later fix. Effect rollups stay split by caller: snpeff and product_call disagree on 19% of paired calls, 62% of which are product_call's strain-aware downstream_frameshift. Collapsed allele and MAF columns exist only because WDK tables render on record pages, so results pages need pre-aggregated values.
Review of the Task 8 commit found two issues. most_severe_impact_snpeff and most_severe_impact_product_call hold text, so WDK sorted them alphabetically (HIGH, LOW, MODERATE, MODIFIER) -- contradicting the severity ranking their own help text advertises. Adds computed rank columns and points sortingColumn at them, entirely within the WDK query so no tuning table change or rebuild is needed. Also completes collapsed_minor_allele_frequency's help text, which named itself derived but omitted the pointer to the canonical SNP Alleles and Indel Alleles sections that the spec requires and collapsed_allele has.
Overview shows both allele sections so a MIXED locus reads honestly. Summary columns use the collapsed allele and MAF, since results pages cannot render tables.
One row per transcript and observed codon. This is where gene strand lives, since strand is only unambiguous per transcript once a locus can overlap two genes.
The help said "Number of strains observed with this codon". It is not per-codon: it is per amino-acid PRODUCT, repeated across every codon row sharing that product. At Variant_Pf3D7_01_v3_29514 four synonymous V codons (GTA/GTC/GTG/GTT) all read 162 while the single A codon reads 1. Summing per-codon gives 649 strains at a locus whose distinct_strain_count is 160. Verified at scale rather than inferred from one locus: across all 1,192,971 product groups having more than one codon row, strain_count is constant within the group -- zero exceptions. A scientist summing this column would have over-counted badly, so the displayName now says "(per product)" and the help states the repetition explicitly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One table with a visible Source column, not two tables, so caller provenance travels with every row and disagreements stay legible. LEFT joins TranscriptAttributes because na_feature_id is null for intergenic calls; an inner join would drop them silently.
Adds SNP Alleles, Indel Alleles, and Strain Statistics category nodes so the two allele classes read as distinct sections on the record page. Requires wb ontology; wb model alone does not regenerate the OWL and would leave every attribute uncategorized with no error.
The overview used '<dt><b>SNP Alleles</b></dt><dd></dd>' as a section header, but the client DROPS any dt/dd pair whose dd is empty. Both headers therefore vanished, and a MIXED locus rendered two identical Reference/Major/Minor triplets with nothing indicating which was the SNP view and which the indel view -- precisely the ambiguity the two-section design exists to prevent. Verified on Variant_Pf3D7_01_v3_12, which reads: SNP Reference A | SNP Major A (0.9316) | SNP Minor C (0.0598) Indel Reference A | Indel Major A (0.9316) | Indel Minor AC (0.0085) The same empty-dd rule is why Gene(s) and Most Severe Impact (Product Call) are correctly absent on that locus: both are genuinely null there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A textAttribute template of "$$allele$$ ($$frequency$$)" renders a bare " ()" when both operands are null, so every pure-SNV record page showed "Indel Major ()" and "Indel Minor ()" -- about 4M of the 4.39M records, reading as a rendering bug rather than as "no indel at this locus". Moving the concatenation into SQL with a CASE yields NULL instead, and the client already drops any dt/dd pair whose dd is empty (the same rule that correctly hides Gene(s) on an intergenic locus). Verified on all three variant types via the service: SNV -> snp_* populated, indel_* null INDEL -> indel_* populated, snp_* null MIXED -> both populated and on the rendered SNV page: zero occurrences of "()" in the overview. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Final review found a critical layout defect plus four cheaper ones. CRITICAL -- the three new category nodes were attached to the ontology ROOT, not to DNA polymorphism. individuals.txt column 2 is the parent IRI and column 3 its label; the hand-typed rows left both empty and put the IRI in column 4 (recordClassName) with 'category' in column 5, so each row asserted a nonexistent record class and declared no parent. SNP Alleles, Indel Alleles and Strain Statistics rendered as top-level peers of Genomic Location, ahead of it. Now verified as "Genetic variation > DNA polymorphism > SNP Alleles" in the assembled tree, with H2/H3/H4 nesting on the page. record_overview was parented to topic_2885 with record+download scope, unlike all 15 other record classes; it rendered as an ordinary attribute row labeled "Overview" inside the DNA polymorphism subsection and became a selectable report column carrying ~4KB of HTML per record. Now topic_0219 / record-internal. The tuning table joins DatasetPresenter but never declared the dependency, the exact failure mode its own comment warns about -- an undeclared dependency yields a silently stale or absent join rather than an error. linkedGeneIds rendered text byte-identical to gene_ids, so the page and column chooser showed two columns both named "Gene ID(s)" with the same content. Dropped; gene_ids is used directly. The four *_and_freq columns also duplicated their base column's displayName and are now "... Allele + Frequency". effect_summary_* had no help, so a blank "Effects (Product Call)" next to a populated SnpEff column read as "this caller found no effect" rather than "did not call here" -- true for 2,700,000 of 4,390,908 loci. Also set displayOrder within the allele and statistics sections: they sorted alphabetically, which put "Reference Allele" LAST. Now reference -> major -> minor -> HGVS. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ence)" The old label "Strain Count" sat beside "Called Strain Count" (159) and "No-Call Strain Count" (57) and read as though those two should sum to it. They do not, and are not meant to. called + no_call is the genotyped panel -- 216 at Pf3D7_01_v3:29514, and call_rate = 159/216 = 0.7361 confirms that denominator on all 4,390,908 loci. distinct_strain_count is called + 1, verified as exactly +1 on every row, the extra strain being the reference, which carries the reference allele by definition. It is the denominator behind the allele frequencies. Renamed in both places it appears -- the attribute displayName and the record overview -- so the same number is not shown under two names. 160 now sits directly beside 159, making the +1 self-evident without help text. Reference-strain explanation confirmed by John rather than inferred. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the temporary jbrestel.VariationAttributes stub with ApidbTuning.VariationAttributes now that the tuning job has built it, and removes the TEMPORARY STUB comment from both query files. 6 SQL references flipped: 4 in variationAttributeQueries.xml, 2 in variationTableQueries.xml. The real table was compared against the stub before flipping rather than assumed equivalent: identical column sets and ordering, and EXCEPT in both directions returned 0 rows over all 4,390,908 rows. Aggregates populated as expected (gene_ids on 2,879,337 loci -- zero would have meant the TranscriptAttributes dependency failed to resolve). Record pages verified unchanged across all five test loci: SNV, INDEL, MIXED (both allele classes), multi-gene, and a caller-disagreement locus. Assembled SQL confirmed to read ApidbTuning via wdkQuery -showQuery. grep -rn jbrestel over Model/lib/wdk and Model/lib/xml returns nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These were authored in the agentic-veupath-dev control-plane checkout because that is where the build commands run from, but they describe work that lands entirely here. Moved to sit beside the code they specify, matching the convention already used for the EDA data tables and the fungidb strain-segment record. Content is carried over verbatim; the plan's progress block is stale relative to the commits on this branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The progress block had been stale since Task 4, claiming Task 5 was next while tasks 5-14 were all committed on this branch, and all 91 step checkboxes were unchecked. Replaced with a task-to-commit map covering 0-14, and checked the boxes. Also retires the stub warning. Both halves are settled: Task 14 (2723cda) flipped all six query references to ApidbTuning.VariationAttributes, and a tuning run built the real table on unidb_shu_a (variationattributes1121, 1688 MB, 4,390,908 rows, plus the view; all 17 va.* columns the model reads resolve). jbrestel.VariationAttributes no longer exists in the database. The plan's original assumption that the flip would cost dev-instance buildability no longer holds, and said so misleadingly. Task bodies keep their jbrestel snippets — that is what was executed — with a banner not to copy them forward. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the question-side scaffolding for the variation record -- a variationParams paramSet, a VariationsBy querySet, a VariationQuestions questionSet, and their apiCommonModel.xml imports -- plus the first search, VariationBySourceId, ported from the deprecated NgsSnpBySourceId. Settled decisions worth the reader's time: the nine-project includeProjects list is deliberate and must not be narrowed to the subset appDb's loaded three; ID matching is exact source_id only, with the alias query named as the home for any future tolerance; the question inherits the record's default summary columns rather than overriding them. The four remaining snp searches are out of scope and explained: they are HSSS processQuery searches over per-strain data, which is the seam the record spec deferred. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five tasks: create the three question-side XML files, import them and confirm wb model loads, add the individuals.txt row and run wb ontology, verify the search in the browser, close out the spec. Uses a cheapest-rung-first verification ladder (xmllint, wb, the WDK service, then the browser) in place of unit tests, since the change is declarative model XML and the sanity model is dead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three new files -- a variationParams paramSet holding the ID datasetParam, a VariationsBy querySet, and a VariationQuestions questionSet -- giving the variation record its first search, ported from the deprecated snp record's NgsSnpBySourceId. Not yet imported into apiCommonModel.xml, so this commit changes nothing that builds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"See spec section 5.1" did not say which of the two specs in docs/superpowers/specs/ it meant. The claim is accurate against the searches spec, but the record spec's section 5 has no subsections, so a maintainer who opens the record spec -- the likelier bookmark when editing this record -- finds nothing and concludes the comment is stale. Names the spec, matching the disambiguation style already used in variationQueries.xml. Comment-only; no element, attribute, value or SQL change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wires the new variation search files into the model. wb model succeeds and the assembled PlasmoDB model now contains VariationQuestions.VariationBySourceId. The search is not yet categorized, so it will not appear in the searches menu until the individuals.txt row lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Places the search under edamontology topic_0199 with menu and webservice scopes, matching the Popset ID search and the parent already used by the variation record's gene-linkage and effect-rollup attributes. Verified by running wb ontology and grepping the freshly built gus_home categories_merged.owl: the class appears exactly once, with rdfs:subClassOf topic_0199, targetType search, and scopes menu and webservice. The /service/ontologies/Categories check could not be run -- the browser is blocked at the veupathdb.org pre-release autologin gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scaffolding and VariationBySourceId are built and verified on the plasmodb dev instance: wb model and wb ontology both pass, the search appears in the searches menu, a single ID goes straight to the record page, and a multi-ID run shows the record's default summary columns. Records in the plan what was substituted and what remains unrun -- the WDK service endpoints were unreachable behind the autologin gate so server-side equivalents were used, and the empty-project case still needs a site whose project lacks variation data. The four HSSS-backed searches remain out of scope pending the per-strain seam. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same treatment as GenesByCopyNumber: the input_pan_id/output_pan_id joins had no data behind them, so sample identity moves to the EDA sample stable id and scoping to organism rather than the retired organismSinglePickCnv.
Both attributesList blocks were commented out, so ten declared CNV columns never reached the results table and the searches returned bare gene rows. Restored with the CNV columns only.
Also adds an explicit organism predicate. The query previously had none, relying on CNV_strain's organism-scoped vocabulary; with samples now coming from an EDA study, a name colliding across organisms would leak rows. Retires the strains regex too - it required a space these sample names never contain, so it stripped nothing and displayed 427_Ploidy.
SequencesByPloidy moves from Curation and Annotation, so all eight dnaseq searches sit in one menu section.
The gate asked for zero assembled-model references to the params being retired, but wdkXml -model dumps a registry of every declared param and query, so the deletion targets necessarily appear in their own inventory entries. It could never return zero. The real condition is that every reference comes from within the deletion set itself, and that no Question references any of them. Both now stated explicitly, with the verified line-by-line breakdown. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
organismVQ.CNV returns zero rows; CnvSamplesMetadataByOrganism and CnvMetadataSpecByOrganism read apidbTuning.Metadata and .Ontology, which do not exist in this build. Their last consumers were ported off in the preceding commits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
apidb.genecopynumber.ref_copy_number is computed per sample: the loader counts only same-ortholog-group genes present in that sample's own result set, so a partially covered sample undercounts it. Verified: of 72 pfal genes in group OG7_0000041, M283 has 46 present and stores 46, G213 51, D003 63, while the 202 fully covered samples all store 72. hit_medians GROUPs BY ref_cn, so those genes emitted several rows and WDK rejected the answer - 'Joined attribute query returned a different number of rows (376) than the ID query alone (100)' - which 500'd both gene CNV searches in the UI. Computing it from the annotation instead makes it constant per gene and matches the column's documented definition, which asks for genes on the same chromosome in the same ortholog group. Verified: fan-out at the 216-sample selection drops from 4 rows per id to 1. The loader remains worth fixing separately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four issues, all confirmed against the live instance before fixing: 1. GenesByNgsSnps had no organism vocabulary override, and the comment claiming the working variation searches use none was false - all four override queryRef to organismVQ.withVariationsTree (variationQueries.xml:44,87,127,170). Without it the dropdown offered every annotated organism, and picking one with no dnaseq study makes eda_sample_table_suffix return zero rows, so the samples filter has nothing to render and the search breaks rather than returning empty. 2. organismVQ.CNVGene and CNVChr had no project predicate, unlike 29 of the file's other vocabularies. The CNV tuning tables span FungiDB, PlasmoDB and TriTrypDB, so a PlasmoDB deployment offered A. fumigatus and would have returned FungiDB transcripts into a PlasmoDB result. Verified after the fix: the vocabulary is now P. falciparum alone, and UniDB still sees all three. 3. The ref_cn comment framed a definitional change as mere decontamination. It is both: the stored column is per-sample AND counts the ortholog group genome-wide, while this counts it per chromosome per the attribute's documented definition. Comments and design s3.5 now say so, including that result sets move. 4. apidb.OrthologGroupAaSequence was an undeclared dependency of the GeneCopyNumbers tuning table, so an ortholog reload would not have triggered a rebuild. Declared. Also documented that ref_cn=1 is the normal case for 89% of pfal genes, so a table built before the orthomcl load would look correct for most rows. Design doc updated for the two-vocabulary split, the ref_cn definition, and a sunset list that was short by two. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the six SNP attributes retired from the gene record (total_hts_snps, hts_nonsynonymous_snps, hts_synonymous_snps, hts_noncoding_snps, hts_stop_codon_snps, hts_nonsyn_syn_ratio), which are already commented out on master along with the TranscriptAttributes_p psql that fed them. The old set had two defects not carried forward: "synonymous" was a residual (total - nonsyn - stop - noncoding), so anything unclassified inflated it; and the ratio reported 0 when there were no synonymous sites, displaying maximal nonsynonymous signal as the minimum value. One row per gene per project. Effect classes partition total_variants (verified 0 violations), as does the impact summary. Adds what the SNP-only model could not express: indel and MIXED loci (14% of pfal), impact summary, segregating loss-of-function with the common subset separated, site-normalized nucleotide diversity, and sample-size context. Three deliberate design points, detailed in the file comments and the spec: - Two grains. Display counts are per gene, unioned across transcripts, most-severe-wins; pi statistics are confined to the representative longest-CDS transcript, whose id is stored. piN/piS is defined for one CDS. - Suppress, never degrade. Frequency-derived statistics accumulate only over loci clearing an allele floor (pi >= 4, common >= 20, rare >= 100) and are NULL below it, never 0. Sample size is per-locus: min(called_strain_count) is 1 in every loaded organism. Counts remain valid at any sample size, so a single-sample organism keeps everything the old section had. - Ploidy is derived, never hardcoded. Measured pfal 1.01, tbru 2.27, afum 2.01 - afum being a haploid fungus called as diploid, which a hardcoded lookup would encode as truth. Site counts are Nei-Gojobori, derived inline from the genetic code. The pooled synonymous-site fraction in pfal is 17.49%, not the textbook ~25%; without this normalization the median piN/piS is 2.0 (implying genome-wide positive selection), with it 0.512 (the expected purifying-selection signature). pfal median pi of 1.5e-3 per site matches published estimates, and known genes rank as expected - AMA1, MSP1, MSP2 high and common-missense dominated; Kelch13 low and rare dominated. Verified against unidb_shu_a: the tuning SQL builds 27,297 rows matching the validated prototype on all counts, bins, ratios and ploidy; the webready copy builds 5,579 pfal rows matching the tuning copy at zero differences across all compared columns. webready is partitioned LIST (org_abbrev) like its peers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The >= 5 synonymous-site guard on pi_n_pi_s is statistically right but blanks
57.5% of pfal genes (3,209 of 5,579), including precisely the most-searched ones.
The cause is intrinsic: strongly selected genes accumulate few SYNONYMOUS variants,
so AMA1 (4 sites), PfCRT (2) and Kelch13 (3) all suppress while MSP1 (28) survives.
A blank reads as "no data" to a biologist.
So each caller now has a pair: the guarded column for sorting, searching and
filtering, where a 2-site gene would otherwise dominate a descending sort; and an
ungated twin for display next to tx_n_synonymous_*, letting the reader discount a
thin denominator themselves ("10.32, from 4 synonymous sites").
The ungated column is NOT redundant with the model dividing pi_nonsyn by pi_syn.
Those are stored rounded to 6 decimals against a pi of ~1e-3, so a reconstructed
ratio drifts up to 0.048 from the correctly computed one and 9% of genes differ
(1,484 of 16,453). Two code paths must not produce two different numbers for the
same statistic.
Verified against unidb_shu_a: 27,297 tuning rows and 5,579 webready pfal rows, with
zero differences from the deployed apidbtuning.genevariationsummary on every
pre-existing column, and zero differences between the two copies on the new ones.
The ungated column is populated for 4,421 pfal genes vs 2,370 guarded, recovering
2,051; where both are present they agree exactly. The 1,158 still blank have no
synonymous diversity to divide by, so they are correctly undefined rather than
suppressed.
Requires a tuning manager rerun to materialize the new columns.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tuning manager substitutes &1 with the version of the table BEING BUILT, so TranscriptAttributes&1 resolved to transcriptattributes1125 - a version that does not exist, since TranscriptAttributes is at 1118. Internal dependencies are referenced by their unversioned name and resolve through the apidbtuning view; &1 belongs only on the table being created and its own indexes. VariationAttributes already had this right and was the precedent I should have followed. Broke the tuning run with: ERROR: relation "transcriptattributes1125" does not exist LINE 51: FROM TranscriptAttributes1125 My verification missed it because the test harness rewrote TranscriptAttributes&1 to apidbtuning.TranscriptAttributes as part of emulating the substitution - so it silently tested the correct form while the shipped XML held the broken one. The harness now asserts that no dependency carries &1 after the built table's own name is substituted, and fails rather than repairing it. No value changes: 27,297 rows, zero differences from the previous build on every column including syn_sites, both piN/piS variants, and rep_transcript_source_id. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Surfaces apidbtuning.GeneVariationSummary on the gene record page and in the tabular results view, replacing the six retired hts_* attributes. Genes and transcripts are a special case: the tabular results view is TranscriptRecordClass while the record page is GeneRecordClass. So each side gets its own attribute query over the same tuning table, and the ontology scope column is what actually routes them - GeneRecordClass rows carry 'record', the TranscriptRecordClass rows carry 'results'. This is not duplication for its own sake; it is the mechanism, and it is why the retired attributes existed as both hts_* and gene_hts_*. The split is also used deliberately for piN/piS. The results view sorts on the GUARDED column (>= 5 synonymous sites), because a gene resting on 2 synonymous sites would otherwise top every descending sort. The record page shows the UNGATED value next to pi_syn_sites_used, so a reader can discount a thin denominator instead of seeing a blank - the guard alone hides 57.5% of pfal genes, including AMA1, PfCRT and Kelch13, because strongly selected genes accumulate few synonymous variants. Record page carries the full set (32 columns: basis, counts, impact summary, effect classes, loss-of-function, selection, second caller, and the raw count ratio for continuity). Results view carries the sortable subset (13 columns) so the table stays usable. Sample basis leads the section on purpose - strains sampled, ploidy, mean call rate. Mean call rate across pfal is 0.552, so "few variants" frequently means poor coverage rather than conservation, and that is the most common misreading of variation data. Help text spells out that a blank means "not enough data", never zero; the raw count ratio is explicitly labelled as not a measure of selection, since its pfal median is 2.22 against a site-normalized piN/piS median of 0.512. Both queries are kept separate from the Bfmv flat view deliberately, so they can change without a Bfmv rebuild - being welded into it is why the old attributes were commented out rather than fixed. Verified against unidb_shu_a with %%PARTITION_KEYS%% substituted: both queries execute (5,720 gene rows, 5,791 transcript rows for pfal), every referenced tuning column exists, and declared <column> elements match actual output columns exactly in both directions. The LEFT JOIN populates 5,579 of 5,720 pfal genes, leaving 141 null for genes with no variants. All four XML files parse; the 40 new ontology rows all have 14 fields. Needs 'wb ontology' rather than 'wb model' - individuals.txt changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All 30 record-page attributes were flat under DNA polymorphism, which is unreadable at that count. They now sit in six collapsible subsections following the display order in the design spec: Sample Basis, Variant Counts, Predicted Impact, Predicted Consequences, Loss of Function, Diversity and Selection. Three groupings are deliberate rather than cosmetic: - Sample Basis is its own group and comes first. Mean call rate across pfal is 0.552, so "few variants" frequently reflects coverage rather than conservation. That has to be read before anything below it. - "Synonymous Sites Behind piN/piS" sits immediately after piN/piS inside Diversity and Selection, so the denominator cannot be visually separated from the value it qualifies. That adjacency is the entire reason we chose display-with-denominator over suppression. - Predicted Impact is kept separate from Predicted Consequences. Impact is the four-bucket triage summary, consequences the SO-term detail; merging them buries the four numbers most readers actually want. Category ids are prefixed GeneVariation* rather than reusing the branch's Variation*Category names: category nodes carry no recordClassName, so identical ids would have merged these groups with the variation record's SNP Alleles / Indel Alleles / Strain Statistics. Display orders 10-15 place them after those three under the shared DNA polymorphism node. The results-view attributes are filed into the same six groups so the attribute picker is organized too, not just the record page. Verified after wb ontology on jbrestel.plasmodb.org: all six categories present in /service/ontologies/Categories with the expected children, all six render as wdk-RecordSubsection blocks on the gene page, and no transcript-scope gene_* attribute leaks onto the record page. On AMA1 all three count partitions independently sum to the 104 total (95+9+0 SNV/indel/mixed; 1+82+4+17 impact; 82+4+1+17 consequences), so the invariant is visible to the reader. Build clean, both error logs silent, no field-count or duplicate-id regression in individuals.txt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Ploidy (as called)" collided with the established meaning of ploidy in this model. apidb.ChrCopyNumber -> ChrCopyNumbers.ploidy is MEASURED chromosome copy number, surfaced by SequencesByPloidy under the display name "Copy Number/Ploidy", and GenesByCopyNumber's own description defines gene dose as ploidy * haploid_number. A reader seeing "Ploidy (as called) 1.01" on the gene page would reasonably take it as a copy-number statement about the gene. It is not. The attribute is total_ploidy_count / called_strain_count on apidb.VariationFeature - how many allele copies per sample the variant caller reported. Data provenance, not biology. Now "Genotype Ploidy (variant caller)", with help text that disclaims the copy-number reading outright and points at the CNV searches for the real thing. The help also notes that a haploid organism processed with diploid settings reads close to 2, so the afum case (2.01 for a haploid fungus) presents as a finding about the upstream calling rather than looking like a bug in this column. An XML comment records why the name is what it is, so it does not get "simplified" back to Ploidy. Only the record XML changed - the ontology rows carry a blank displayName column on purpose, so wb model was sufficient and individuals.txt is untouched. Verified on jbrestel.plasmodb.org after wb model: /service/record-types/gene reports the new displayName, the help text carries the disclaimer, the Sample Basis subsection renders "Genotype Ploidy (variant caller) 1.01", and the old label is gone from the page. Build clean, all logs silent including both error logs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The EDA dnaseq study holds only the resequenced isolates, so the reference strain never appeared in variation_sample_meta - even though HSSS has always carried it as strain id 1 (3D7 on PlasmoDB, TREU927 on TriTrypDB, Af293 on FungiDB). The searches could always include it; only the UI could not offer it. SamplesMetadataByStudyWithRef unions three synthesized attribute rows onto the attributevalue select: strain, organism, and dataset. Those are the identity a reference strain genuinely has - it has no collection site, host, or alignment statistics, and an unset attribute correctly excludes it from those facets. Keyed on provider_label rather than the VAR_<hash> stable_id, because the label is site-specific (parasite_strain/parasite_organism vs fungal_strain/ fungus_organism - note fungus, not fungal). Values reuse the isolates' shape, so the reference joins the existing facet buckets instead of creating singletons: the organism facet stays one bucket at 217, not two. variation_sample_meta, _a and _b move to the new query; cnv_sample_meta keeps the original. The reference is real for HSSS but not for copy number, whose searches filter a table with no reference rows - offering it there would be an option that silently returns nothing. Verified on the running instance: all five HSSS filters report 217 samples and include the reference, CNV reports 216 and excludes it, and a search over 3D7 + B082 returns variants. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
EDA stores the dataset attribute as a dataset stable ID (DS_1d17c1883c) - the right thing to store, an opaque thing to show. The Dataset facet listed four DS_ hashes. Both metadata queries now LEFT JOIN apidbtuning.datasetpresenter and select COALESCE(dp.display_name, av.string_value). Applied to SamplesMetadataByStudy as well as SamplesMetadataByStudyWithRef, so cnv_sample_meta gets it too. The fork between those two queries is about which samples exist, not about presentation, and a facet reading DS_302d3e3bc0 in the CNV searches but a sentence in the variation searches - for the same samples - would be a worse bug than the one being fixed. COALESCE rather than the join value alone, so a sample survives when no presenter row exists; on a gated dev instance datasetpresenter holds only loaded datasets. The join is narrowed by provider_label rather than left to match the DS_ shape, which works today only by coincidence. dataset_presenter_id is unique, so no row multiplication - counts are identical before and after (3771/216, 3774/217). Verified on the running instance: the HSSS filters show four display names including "Genome Sequence and Annotation" for the reference, CNV shows the same three isolate datasets and no reference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The record class was named Variation while every source_id it holds reads
Variant_<sequence>_<location>, and its own attributes already said Variant
Type / Variant Call Set. "Variation" is the phenomenon; a record is one
concrete allele at a locus, so it is a variant.
Renames the WDK layer only - record class, question set and its five
searches, query sets, param set, category ontology nodes, and the six model
files - and sets displayName to Short Variant / Short Variants, naming what
the record actually holds (SNVs and small indels) rather than variation in
general.
Deliberately unchanged:
- urlName stays "variation" so existing record urls keep resolving; the
feature is not public yet, so this can move in a separate change.
- variation_sample_meta{,_a,_b} and chromosomeOptionalForVariations are
contracts with the HSSS plugins in ApiCommonWebService; renaming them
breaks the searches at run time, not build time.
- apidb.Variation* and apidbtuning.VariationAttributes /
GeneVariationSummary are database objects, out of the model's reach.
Verified on a dev instance after wb ontology: all five searches resolve
with expandParams, and the category tree keeps all 59 nodes in their
sections with no stale VariationRecordClass references.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
HSSS computes over single-nucleotide variants only, so these four cannot return an indel no matter what the record class is called. VariantBySourceId keeps its neutral name because an ID lookup goes to the tuning table and does return both. Prefixes shortDisplayName too: a strategy box showing bare "Location" or "Two Groups" is exactly where the SNV-only scope would otherwise be lost. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"SNV Gene ID(s)" and "SNV Genomic Location" named their input; all four HSSS searches compute differences within a sample group, so they now say so: "SNV Differences for Gene(s)" and "SNV Differences for Genomic Location". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…search The sibling of GenesByNgsSnps. That one runs HSSS over a sample set the user picks and takes minutes; this one reads apidbtuning.GeneVariationSummary, returns in milliseconds, and reports the same numbers the gene record page shows. Neither replaces the other: dropping the HSSS search would delete the sample-set comparison, and not having this one leaves the record page and the search disagreeing about the same gene with no explanation. GenesByNgsSnps is renamed to "SNV Characteristics Within a Group of Samples" to say so. Filtering is ONE filterParam over sixteen statistics, not sixteen range params. The range-param version was written first and abandoned: a numberRangeParam cannot express "untouched", so every range applies on every run and any gene NULL in that column is excluded - and GeneVariationSummary NULLs its frequency statistics below their allele floors on purpose, which for tbruTREU927 is 100% of rows for three columns. Working around that needed a full-span escape clause with bounds hardcoded in two files, which then broke when NumberRangeParamHandler rounded a value onto the declared ceiling. The filterParam has none of it: a gene with no value contributes no metadata row, distributions come from the data, and adding a statistic is one ontology row plus one VALUES tuple. Also adds organismVQ.withVariantSummary, scoping the organism tree to organisms that actually have a summary row rather than to a datasource type. Verified against unidb_shu_a on a dev instance: the impact distribution sums to all 5,579 pfal genes (HIGH 2,797, matching SQL), piN/piS reports 2,370 - exactly the genes with a non-null value - and filtered runs match direct SQL counts (piN/piS >= 1.5 -> 271 genes/272 transcripts; n_lof >= 5 -> 317/319). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ones TranscriptRecordClass already publishes this tuning table's sortable subset, and gene_pi_n_pi_s there is gvs.pi_n_pi_s_snpeff - the same column the search's filter reads. The dynamic columns were showing identical numbers under parallel names, which is a divergence waiting to happen. Drops the dynamicAttributes block and the gvs_* columns from the id query, leaving it returning only the four columns WDK needs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Not just the wsColumn: both postCacheUpdateSql blocks enumerate the result columns explicitly in their INSERT and SELECT lists, so without adding it there the sibling transcripts they backfill would carry NULL span density while the matched transcripts carried a value. Required by the plugin change in ApiCommonWebService 7a61e99 - getColumns now advertises span_snp_density, and PluginExecutor.validateColumns rejects a plugin advertising a column the wsColumn list lacks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds SNVs per kb (gene span) alongside the now-actually-CDS density, renames the ratio to say it is site-normalized, and corrects three labels that never matched the classifier: nonsynonymous excludes stop-gained, and non-coding is really unclassified - class 0 means no product byte was available, which covers unclassifiable positions as well as genuinely non-coding ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Density is CDS density, the ratio is site-normalized, and the class enum terms follow the corrected column names. Internal enum values are unchanged - they are a contract with legalParams and with the filter's branches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The PlasmoDB description promised normalized Dn/Ds in a future release; it is here. Also states in both descriptions that these statistics are sample-set scoped and so not comparable to the precomputed ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…cription The bullet keeps saying the ratio is site-normalized; it no longer quotes the 17.49% figure or the 1.43x factor at the user. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These are computed over the samples the user selected, but a results table can carry them alongside the whole-cohort columns from the gene record (Total Variants, Variants per kb, piN/piS) with nothing to tell them apart. Each now carries a (sample group) qualifier. The two density labels swap their existing parentheses for commas so no header ends up with two bracketed clauses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Model-side of the SNV work. Must merge together with
ApiCommonWebServicePR #20 — that PR's plugin advertises aspan_snp_densitycolumn andvalidateColumnsrejects it unless this PR'swsColumndeclaration is present.Renames the Variation record to Variant
The record class was
Variationwhile everysource_idit holds readsVariant_<sequence>_<location>, and its own attributes already said "Variant Type" and "Variant Call Set". "Variation" is the phenomenon; a record is one concrete allele at a locus.Renames the WDK layer only — record class, question set and its five searches, query sets, param set, category ontology nodes, and six model files — and sets
displayNameto Short Variant / Short Variants, naming what the record actually holds (SNVs and small indels).Deliberately unchanged:
urlNamestaysvariationso existing record URLs resolve (the feature is not public yet, so this can move separately);variation_sample_meta{,_a,_b}andchromosomeOptionalForVariationsare contracts with the HSSS plugins and renaming them breaks the searches at run time, not build time.The four HSSS searches now lead with SNV — HSSS computes over single-nucleotide variants only and cannot return an indel however the record class is named.
VariantBySourceIdis not prefixed, because an ID lookup goes to the tuning table and does return both.Adds GenesByVariantCharacteristics
A precomputed sibling to
GenesByNgsSnps. That one runs HSSS over a user-selected sample set and takes minutes; this readsapidbtuning.GeneVariationSummary, returns in milliseconds, and reports the same numbers as the gene record page. Neither replaces the other — dropping the HSSS search would delete the sample-set comparison, and without this one the record page and the search disagree about the same gene with no explanation.Filtering is one
filterParamover sixteen statistics, not sixteen range params. The range-param version was built first and abandoned: anumberRangeParamcannot express "untouched", so every range applies on every run and any gene NULL in that column is excluded — andGeneVariationSummaryNULLs its frequency statistics below their allele floors by design, which fortbruTREU927is 100% of rows for three columns. Working around that needed a full-span escape clause with bounds hardcoded in two files, which then broke whenNumberRangeParamHandlerrounded a value onto the declared ceiling. ThefilterParamhas none of it, and distributions come from the data rather than from hand-declared bounds.Result columns reuse the transcript record's own attributes rather than new dynamic ones —
gene_pi_n_pi_sthere isgvs.pi_n_pi_s_snpeff, the same column the filter reads, so the filtered value and the displayed value agree by construction.Corrects GenesByNgsSnps labels
Its statistics changed in PR #20, so the strings describing them changed with them: density is now genuinely CDS with the old gene-span value beside it, the ratio says it is site-normalized, "Nonsynonymous SNPs" → Missense SNVs (it never included stop-gained), "Non-coding SNPs" → Unclassified SNVs (class 0 means no product byte was available). All eight result columns carry a (sample group) qualifier, since a results table can hold them next to the whole-cohort columns from the gene record.
Also adds
span_snp_densityto bothpostCacheUpdateSqlblocks, not just thewsColumnlist — those blocks enumerate result columns explicitly and backfill sibling transcripts, so omitting it would leave siblings NULL while matched transcripts carried a value.Verification
Built and exercised on a dev instance throughout. Category tree keeps all 59 nodes after the rename with no stale
VariationRecordClassreferences; all five renamed searches resolve withexpandParams; the new search's filter distributions match SQL (impact classes sum to all 5,579 pfal genes; piN/piS reports exactly the 2,370 genes that have a value).Note on scope
This branch carries ~90 prior commits of dnaseq work authored before this session — CNV search ports, the
GeneVariationSummarytuning table, gene-record variation wiring. They are on the branch but are not part of this change.🤖 Generated with Claude Code