Per-gene and per-histology summary views

[pj-sullivan]

Purpose/implementation Section

What scientific question is your analysis addressing?

Summarising the exisiting tables to create a summary view for TAPESTRY front end, instead of large, slow queries from the API.

What was your approach?

Add two PostgreSQL materialized views to TAPESTRY-data-preprocessing:

• tej_gene_summary — aggregates TEJ data per gene: sample count, junction count, plot group count, domains affected, consensus specificity (oncofetal / tumor-specific), junction status (annotated / novel junction / novel splice site), and preference code breakdown (A3±, A5±, EI, ES, RI). Indexed on gene.
• tej_histology_summary — aggregates the same junction-level breakdowns per histology (plot_group), plus gene and sample counts. Indexed on plot_group.

Add a refresh_views.py pipeline script that issues REFRESH MATERIALIZED VIEW for both views and can be run after any pipeline that updates the underlying tej, sample_tej, tej_recurrence, or tej_domain tables.

What GitHub issue does your pull request address?

Closes #17

Directions for reviewers. Tell potential reviewers what kind of feedback you are soliciting.

This worked for me when rebuilding after an edit, but not sure if it'll work from scratch. If the container is already running, this may allow you to not have to rebuild all the tables. Just check to see if the results say Refreshed materialized views.

docker compose run --rm pipeline python -m db.migrate && docker compose run --rm pipeline python -m pipelines.refresh_views

If above doesn't work

You may have to get rid of the previous version for this to work, so I'll include both commands. May take ~10 mins (thanks CPM data):

docker compose down -v
docker compose up pipeline

Results (just check the final rows have pipeline-1 | Refreshed materialized views.):

Which areas should receive a particularly close look?

View heads for review
tej_gene_summary:

 gene  | num_samples | num_junctions | num_plot_groups | num_domains_affected | num_oncofetal | num_tumor_specific | num_annotated_junction | num_novel_junction | num_novel_splice_site | num_pref_a3_short | num_pref_a3_long | num_pref_a5_short | num_pref_a5_long | num_pref_ei | num_pref_es | num_pref_ri 
-------+-------------+---------------+-----------------+----------------------+---------------+--------------------+------------------------+--------------------+-----------------------+-------------------+------------------+-------------------+------------------+-------------+-------------+-------------
 AAK1  |          15 |             1 |               3 |                    0 |             0 |                  1 |                      1 |                  0 |                     0 |                 0 |                0 |                 0 |                0 |           0 |           1 |           0
 AANAT |          41 |             1 |               2 |                    0 |             0 |                  1 |                      1 |                  0 |                     0 |                 1 |                0 |                 0 |                0 |           0 |           0 |           0
 AAR2  |         193 |             2 |              15 |                    0 |             1 |                  1 |                      2 |                  0 |                     0 |                 0 |                0 |                 0 |                0 |           1 |           1 |           0
 AARS1 |          87 |             1 |               9 |                    0 |             0 |                  1 |                      1 |                  0 |                     0 |                 0 |                0 |                 1 |                0 |           0 |           0 |           0
 AATK  |           7 |             2 |               1 |                    1 |             0 |                  2 |                      2 |                  0 |                     0 |                 0 |                0 |                 1 |                0 |           1 |           0 |           0

tej_histology_summary

             plot_group            | num_samples | num_junctions | num_genes | num_oncofetal | num_tumor_specific | num_annotated_junction | num_novel_junction | num_novel_splice_site | num_pref_a3_short | num_pref_a3_long | num_pref_a5_short | num_pref_a5_long | num_pref_ei | num_pref_es | num_pref_ri 
----------------------------------+-------------+---------------+-----------+---------------+--------------------+------------------------+--------------------+-----------------------+-------------------+------------------+-------------------+------------------+-------------+-------------+-------------
 Atypical Teratoid Rhabdoid Tumor |          83 |          2941 |      1538 |          1133 |               1808 |                   2826 |                103 |                    12 |               132 |              133 |               207 |               91 |        1217 |        1082 |          79
 Choroid plexus tumor             |          36 |          2486 |      1086 |           483 |               2003 |                   2411 |                 65 |                    10 |               123 |              148 |               145 |               68 |        1277 |         704 |          21
 Craniopharyngioma                |          71 |          1993 |      1017 |           411 |               1582 |                   1918 |                 71 |                     4 |               110 |              115 |               144 |               63 |         842 |         693 |          26
 DIPG or DMG                      |         186 |          1034 |       603 |           390 |                644 |                   1002 |                 25 |                     7 |                50 |               41 |                67 |               24 |         440 |         396 |          16
 Ependymoma                       |         176 |          2680 |      1233 |           534 |               2146 |                   2596 |                 73 |                    11 |               119 |              145 |               176 |               67 |        1341 |         822 |          10
 Germ cell tumor                  |          24 |           792 |       506 |           285 |                507 |                    761 |                 30 |                     1 |                48 |               37 |                57 |               22 |         363 |         241 |          24
 Low-grade glioma                 |         427 |          1277 |       678 |           308 |                969 |                   1230 |                 42 |                     5 |                50 |               56 |                92 |               39 |         579 |         444 |          17
 Medulloblastoma                  |         242 |          2196 |      1118 |          1021 |               1175 |                   2121 |                 66 |                     9 |                92 |               97 |               156 |               58 |         994 |         767 |          32
 Meningioma                       |          46 |          1218 |       615 |           253 |                965 |                   1181 |                 34 |                     3 |                44 |               64 |                85 |               45 |         536 |         432 |          12
 Mesenchymal tumor                |          51 |          1073 |       607 |           329 |                744 |                   1040 |                 32 |                     1 |                45 |               54 |                77 |               34 |         454 |         389 |          20
 Mixed neuronal-glial tumor       |         152 |           948 |       513 |           285 |                663 |                    917 |                 25 |                     6 |                37 |               40 |                65 |               27 |         430 |         330 |          19
 Neurofibroma plexiform           |          29 |           579 |       286 |           109 |                470 |                    565 |                 13 |                     1 |                32 |               31 |                28 |               19 |         255 |         208 |           6
 Oligodendroglioma                |          32 |           524 |       250 |           200 |                324 |                    516 |                  5 |                     3 |                23 |               27 |                22 |               15 |         258 |         167 |          12
 Other CNS embryonal tumor        |          44 |           756 |       446 |           394 |                362 |                    734 |                 21 |                     1 |                47 |               39 |                38 |               19 |         345 |         251 |          17
 Other high-grade glioma          |         263 |          1565 |       918 |           652 |                913 |                   1504 |                 54 |                     7 |                69 |               63 |               109 |               44 |         679 |         573 |          28
 Other tumor                      |          24 |           771 |       450 |           141 |                630 |                    747 |                 24 |                     0 |                43 |               42 |                54 |               31 |         315 |         254 |          32
 Schwannoma                       |          30 |          1377 |       695 |           254 |               1123 |                   1341 |                 34 |                     2 |                56 |               70 |                96 |               49 |         623 |         472 |          11

Is there anything that you want to discuss further?

Is the analysis in a mature enough form that the resulting figure(s) and/or table(s) are ready for review?

Results

What types of results are included (e.g., table, figure)?

What is your summary of the results?

Reproducibility Checklist

• The dependencies required to run the code in this pull request have been added to the project Dockerfile.

Documentation Checklist

• This analysis module has a README and it is up to date.
• The analytical code is documented and contains comments.