Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/changelog/2026-W47.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,19 @@
**Files touched**: `engine/entities/samples.ts`, `engine/entities/aquifers.ts`, `engine/entities/streams.ts`

**Prevention**: `engine/tests/blockWording.test.ts` checks the aquifer and streams wording; `engine/tests/stepLabels.test.ts` checks the sample step labels, and its raw-type-name check now allows labels that contain their type name; `engine/tests/__snapshots__/question-text.txt` re-recorded.

3. **Added: Counts Next to Every Filter Dropdown Option** (2026-11-17)

**Component**: `buildOptionCountQuery` in `engine/templates/fusedQueries.ts`, `options` on the entity modules in `engine/entities/`, `useOptionCounts` in `ui/hooks/useDiscoveryQueries.ts`

**Change**: Issue #165, a follow-up from #70. Industry, Feature Type, Aquifer Type and Well classification now show how many entities each option would find: in the selected state or counties, or nationwide with none. Industry used to show counts only once a state was picked, Well classification only for the whole of Illinois or Maine, and Feature Type and Aquifer Type none. Feature Type parents show the sum of their children; Aquifer Type sums the raw values each option covers.

**How**: each count is the run's own count step with the filter left open and grouped by option, so it checks the region exactly as the run does. Hancock County wells: 7,296, where a count by subdivision link alone gave 11,644. Live queries on `federation`, cached for the session, nothing stored. A failed count leaves the dropdown working, with no numbers.

**Cost**: measured live on 2026-11-17, 1.8s or less for every type, nationwide and for Maine, Hancock and Illinois, except Illinois aquifers cold (30.1s, 0.46s on a rerun; the run's own count step for that block took 25.8s), where the first open may show no numbers.

**Unchanged**: the SPARQL every question sends (`query-shapes.txt` did not move). Substance and Material counts.

**Files touched**: `engine/templates/fusedQueries.ts`, `engine/templates/regions.ts` (`buildIndustryCountsQuery` removed), `engine/templates/wells.ts`, `engine/entities/*.ts`, `engine/optionCounts.ts` (new), `ui/hooks/useDiscoveryQueries.ts`, `ui/components/QueryEditor/` (Facility, SurfaceWater, Aquifer and Well filters, `EntityBlock.tsx`, `FlatSelect/withCount.ts` new), `docs/wiki/Dropdowns Industry Counts.md` (now "Option counts"), `docs/wiki/Dropdowns.md`

**Prevention**: `engine/tests/optionCounts.test.ts` checks that every counted type's count query carries the same region lines as its count step, for no region, a state, a county and two counties, and how rows become counts; `ui/tests/optionCounts.test.tsx` checks each dropdown shows its counts for the right region, a failed count, and that a region already seen is not fetched again.
187 changes: 90 additions & 97 deletions docs/wiki/Dropdowns Industry Counts.md
Original file line number Diff line number Diff line change
@@ -1,132 +1,125 @@
# Industry counts
# Option counts

Not a dropdown of its own. These are the facility counts shown next to each entry in the
[Industry dropdown](Dropdowns%20Industry), and they come from a separate query against a
different endpoint, so they get their own page.
Not a dropdown of its own. These are the numbers next to each option in the
[Industry](Dropdowns%20Industry), Feature Type, Aquifer Type and Well classification
dropdowns, and they all come from one query shape, so they share this page. (Substance and
Material count observations with their own queries, on their own pages.)

![Industry counts](images/dropdown_industry_counts.png)

## What it does

Once a state or some counties are selected, each node in the industry tree picks up a number:
how many facilities of that kind actually exist in that region. It turns an abstract list of
5596 NAICS codes into something you can steer by, because you can see at a glance which
industries are worth filtering on and which will return nothing.
Each option shows how many entities picking it would find. With a state or counties
selected, the count is for that region. With none, it is nationwide. An option the region
has none of shows no number and is dimmed in the tree dropdowns.

With no region selected there are no counts at all. The query is gated behind having a state,
because counting every facility in the country is both slow and not useful for picking a
filter.
A count previews the answer, so it has to find exactly what the run finds. It is the run's
own count step (`buildEntityCountQuery`, the step that opens every run) with the filter left
open and grouped by option. That matters most for types whose region check looks at the
entity as well as its S2 cell. Wells in Hancock County, Maine (2026-11-17):

Counts roll up the tree. A leaf shows what the endpoint returned for that exact code. A
parent shows the sum of itself and everything beneath it, computed on the client by
`rollupCounts()`. So a sector node showing 400 means four hundred facilities spread across
its descendants, not four hundred facilities tagged with the bare sector code.
| Region check | Wells |
|---|---|
| Subdivision link only | 11,644 |
| S2 cell only | 7,428 |
| Both, as the run checks wells | 7,296 |

There is **no fallback**. If the query fails you get an empty object and the tree simply
shows no numbers. Nothing breaks, you just lose the guidance.
A count built any other way would promise one number and the map would show another.

Tree dropdowns roll counts up: a parent shows the sum of its children, computed on the
client by `rollupCounts()`. That is sound because the children partition the parent (an
FTYPE value or a NAICS code belongs to one parent only).

There is **no fallback**. A failed count leaves the dropdown working, with no numbers.

## The SPARQL query

Endpoint is `federation`, not `fiokg`. This is the important detail: the industry list comes
from `fiokg` but the counts cannot, because counting facilities per region needs the industry
graph and the spatial graph joined together, and only `federation` has both.
Endpoint `federation` for every type. Built by `buildOptionCountQuery(type, regionCodes)`.
Industry in Hancock County:

```sparql
SELECT ?industryCode (COUNT(DISTINCT ?facility) AS ?num) WHERE {
VALUES ?_regionRoot { kwgr:administrativeRegion.USA.23 }
?s2 rdf:type kwg-ont:S2Cell_Level13 ;
spatial:connectedTo ?_regionRoot ;
kwg-ont:sfContains ?facility .
?facility fio:ofIndustry ?industryCode .
?industryCode a naics:NAICS-IndustryCode .
} GROUP BY ?industryCode
SELECT ?option (COUNT(DISTINCT ?facilityP) AS ?count) WHERE {
?s2probe rdf:type kwg-ont:S2Cell_Level13 .
?s2probe spatial:connectedTo kwgr:administrativeRegion.USA.23009 .
?facilityP spatial:connectedTo ?_regionLinkP .
?_regionLinkP kwg-ont:administrativePartOf* kwgr:administrativeRegion.USA.23009 .
?s2probe kwg-ont:sfContains ?facilityP .
?facilityP fio:ofIndustry ?option .
?option a naics:NAICS-IndustryCode .
} GROUP BY ?option
```

Line by line:
* The three lines after the S2 cell line are the run's region check for this type, taken from
the same code (`regionClause` and the type's `regionCheck`). With no region they are
absent and the count is nationwide.
* The last lines are what the type binds when one option is picked, with the option left
open in `?option`. Each type's module in `engine/entities/` supplies it as
`options.bind`:

* `VALUES ?_regionRoot { ... }` is the selected region. One entry for a whole state, or one
per county when counties are picked. `23` is Maine.
* `?s2 rdf:type kwg-ont:S2Cell_Level13` brings in S2 cells, the grid the whole app uses for
spatial joins. Each cell covers roughly 1.2 square kilometres.
* `spatial:connectedTo ?_regionRoot` keeps only the cells inside the chosen region.
* `kwg-ont:sfContains ?facility` gets the facilities inside those cells.
* `?facility fio:ofIndustry ?industryCode` is what kind of facility it is, and the
`naics:NAICS-IndustryCode` line keeps only real NAICS codes rather than any other
classification hanging off the same predicate.
* `COUNT(DISTINCT ?facility)` per code. `DISTINCT` matters, because a facility sitting on a
cell boundary can be reached more than once.
| Dropdown | Type | `?option` is |
|---|---|---|
| Industry | facilities | `fio:ofIndustry`, a NAICS code IRI. No geometry check, as when an industry is picked |
| Feature Type | waterBodies, streams | `nhdplusv2:hasFTYPE`, e.g. `"LakePond"` |
| Aquifer Type | aquifers | `saw_water:aquiferType`, a raw value such as `"sand_gravel"` |
| Well classification | wells | Illinois `wellPurpose`, Maine `ofWellType` or `hasUse`, an IRI |

Note the route through S2 cells rather than asking directly which facilities are in Maine.
That is the standard pattern across this app: everything spatial goes through S2 Level 13
cells. See [`docs/ARCHITECTURE.md`](https://github.com/SAWGraph/explorer-app/blob/main/docs/ARCHITECTURE.md)
for why.
### What comes back, and how it is keyed

Verified against the live endpoint. Maine returns 664 distinct industry codes with counts,
from single facility codes like `111110` Soybean Farming through to the busier ones.
One row per option, `?option` and `?count`. `optionCountsFrom()` keys them the way each
dropdown keys its options, using the type's `options.key`:

### What comes back
* Industry: the IRI is cut to the bare code (`NAICS-111110` to `111110`).
* Aquifer Type: raw values are summed into the two options through `AQUIFER_TYPE_VALUES`.
Sand & gravel covers `sand and gravel`, `coarse-grain_materials` and `sand_gravel`.
* Well classification: keyed by IRI. An Illinois entry stands for an active and a plugged
purpose, so `wellTokenCount()` sums the IRIs in the entry's token.
* Feature Type: the FTYPE value as is.

Full URIs, not bare codes:
### Measured

```
http://w3id.org/fio/v1/naics#NAICS-111110 1
http://w3id.org/fio/v1/naics#NAICS-11121 4
```
Live, 2026-11-17, first run unless noted:

Unlike the industry list query, this one does not strip the prefix in SPARQL. The hook does
it, splitting on `#` or `/` and then removing the leading `NAICS-`. The result has to match
the bare codes the tree is keyed on, so if either side changes shape the numbers silently
stop appearing.
| Region | Feature Type (water bodies / flowlines) | Aquifer Type | Wells | Industry |
|---|---|---|---|---|
| Nationwide | 0.33s / 0.14s | 0.22s | 0.22s | 0.15s |
| Maine | 0.31s / 0.17s | 0.40s | 0.43s | 1.42s |
| Hancock | 0.18s / 0.24s | 0.36s | 0.43s | 0.35s |
| Illinois | 0.17s / 0.54s | 30.1s cold, 0.46s rerun | 0.66s | 1.74s |

Illinois aquifers are slow cold because the subdivision check touches every Illinois
subdivision an aquifer overlaps. The run's own count step for the same block took 25.8s
cold.

Sample counts from the same run: nationwide Lake / Pond 60,083, Stream / River 353,003;
Hancock Lake / Pond 348, sand & gravel aquifers 181, Maine bedrock wells 7,236.

## How it is wired up

* There is no control of its own. The numbers are passed into `HierarchicalSelect` as a
`counts` prop at
[`FacilityFilters.tsx:24`](https://github.com/SAWGraph/explorer-app/blob/main/src/components/QueryEditor/FacilityFilters.tsx#L24).
* The data comes from
[`useIndustryCounts(region)`](https://github.com/SAWGraph/explorer-app/blob/main/src/hooks/useDiscoveryQueries.ts#L76),
called at
[`FacilityFilters.tsx:13`](https://github.com/SAWGraph/explorer-app/blob/main/src/components/QueryEditor/FacilityFilters.tsx#L13).
* The query text is built by
[`buildIndustryCountsQuery()`](https://github.com/SAWGraph/explorer-app/blob/main/src/engine/templates/regions.ts#L14).
* It runs through
[`executeSparql()`](https://github.com/SAWGraph/explorer-app/blob/main/engine/endpoint.ts)
against the `federation` URL in
[`endpoints.ts`](https://github.com/SAWGraph/explorer-app/blob/main/src/constants/endpoints.ts).
* The rollup is
[`rollupCounts()`](https://github.com/SAWGraph/explorer-app/blob/main/src/components/QueryEditor/HierarchicalSelect/useNaicsTree.ts#L148),
now shared with the [Material](Dropdowns%20Material) dropdown. Summing children into a
parent is only sound when the children partition the parent. That holds for NAICS by
construction, and for material types because `MIN(?bucketPrio)` forces each one into
exactly one group. It would not hold for a set of overlapping categories.

Cached under `['industryCounts', <region key>]` with `staleTime: Infinity` and gated by
`enabled: !!region?.stateCode`. The region key sorts the county codes before joining them, so
picking the same counties in a different order reuses the cache instead of refetching.

The hook returns a plain `Record<string, number>` keyed by bare code, which is what
`rollupCounts()` expects as its leaf counts.
* `useOptionCounts(type, region)` in `ui/hooks/useDiscoveryQueries.ts` runs the query and
returns `Record<string, number>`. Cached under `['optionCounts', type, <region key>]` with
`staleTime: Infinity`, so each type and region is fetched once a session. The region key
sorts the county codes, so the same counties in another order reuse the cache.
* It runs when the dropdown mounts: Feature Type as soon as the Type is Surface Water (both
water bodies and flowlines are counted, since the tree holds both), the others once
"+ Add Filters" is open.
* `FacilityFilters` and `SurfaceWaterFilters` pass the counts to `HierarchicalSelect` as its
`counts` prop. `AquiferFilters` and `WellFilters` put them in the label with `withCount()`.
* Well classification lists its options from `hydrologykg` with counts for the whole of
Illinois or Maine. Those are not shown; the region counts replace them.

## If it looks wrong

**No numbers anywhere.** Either no state is selected, which is expected, or the `federation`
query failed. There is no fallback, so failure looks exactly like "not selected yet". Check
the network tab to tell them apart.

**Numbers appear on some nodes but not others.** Those codes have no facilities in the
selected region. Zero is not returned, so nothing shows.
**No numbers anywhere.** The count query failed, or has not come back yet. Check the
network tab for a `federation` request with `GROUP BY ?option`.

**A parent shows a bigger number than the sum of its visible children.** Expected when a
child code exists in the counts result but not in the industry list, or when the tree is
flatter than NAICS because of a missing intermediate code. The rollup only sums nodes that
are in the tree.
**Numbers on some options but not others.** The region has none of those. Zero is not
returned, so nothing shows.

**Numbers vanish after switching from a state to counties.** New cache key, new fetch. If
they do not come back, one of the county codes is not resolving to a region root.
**A count does not match the map.** Compare the region lines of the count with the run's
count step for the same block. `engine/tests/optionCounts.test.ts` asserts they are the same
for every type and region form.

**Counts look too high for a small county.** Facilities on S2 cell boundaries can appear in
cells belonging to more than one region. `COUNT(DISTINCT ?facility)` dedupes within a single
result, but the cell to region join is inclusive by design.
**Numbers vanish after switching from a state to counties.** New cache key, new fetch.

## See also

Expand Down
12 changes: 6 additions & 6 deletions docs/wiki/Dropdowns Industry.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ selected. Tick some of a branch and the parent shows an indeterminate state.
The full list is large, 5596 rows from the endpoint, which is why it is a searchable tree
rather than a list you scroll. Typing filters the tree and auto expands whatever matched.

Each node also shows how many facilities of that kind exist in the currently selected region.
Those numbers come from a completely separate query, documented in
[Dropdowns Industry Counts](Dropdowns%20Industry%20Counts).
Each node also shows how many facilities of that kind exist in the selected region, or
nationwide with none. Those numbers come from a separate query, documented in
[Option counts](Dropdowns%20Industry%20Counts).

### Nothing ticked is not the same as everything ticked

Expand Down Expand Up @@ -152,8 +152,8 @@ what the query actually returned before assuming the tree builder is broken.

**Labels with odd trailing spaces.** Source data. They come through as is.

**No counts next to anything.** No region is selected, or the counts query failed. See
[Dropdowns Industry Counts](Dropdowns%20Industry%20Counts).
**No counts next to anything.** The counts query failed or has not come back. See
[Option counts](Dropdowns%20Industry%20Counts).

**Selecting a parent does not match the facilities you expected.** Selections are expanded to
descendant codes, so a parent only covers children that exist in this list. If the facility
Expand All @@ -162,4 +162,4 @@ you want carries a code absent from `fiokg`, no amount of parent ticking will re
## See also

* [Dropdowns](Dropdowns), the index of all filter controls
* [Dropdowns Industry Counts](Dropdowns%20Industry%20Counts), the numbers shown on each node
* [Option counts](Dropdowns%20Industry%20Counts), the numbers shown on each node
Loading
Loading