Paging is applied before the anonymization floor, so pages come back short and the remainder is unreachable

[blarghmatey]

Expected Behavior

A caller paging a list endpoint should be able to retrieve every row it is entitled to see. Requesting limit=N should return min(N, remaining_visible_rows) rows, and increasing limit up to total_count should eventually yield the whole visible result set.

Current Behavior

LIMIT/OFFSET are applied in SQL, but the k-anonymity floor drops rows in Python after the query returns. The two are applied in the wrong order, so a page is filled with raw rows and then thinned:

• build_select puts the bound LIMIT %s OFFSET %s on the query:

  ol-analytics-api/src/ol_analytics_api/core/db/query.py

  Line 64 in a0d2e86

  return f"SELECT {columns} FROM {schema_table}{where} ORDER BY {order} LIMIT %s OFFSET %s" # noqa: S608

• suppress_small_cohorts then drops every row whose primary cohort is below the floor:

  ol-analytics-api/src/ol_analytics_api/core/anonymization.py

  Line 99 in a0d2e86

  if (row.get(policy.primary) or 0) < floor:

• The org endpoints wire them together in that order:

  ol-analytics-api/src/ol_analytics_api/tenants/b2b_dashboard/routers/organizations.py

  Lines 123 to 128 in a0d2e86

  	rows = await fetch_and_suppress(
  	query,
  	(org_slug, page.limit, page.offset),
  	spec.model,
  	settings.anonymization_floor,
  	)

Three consequences:

1. Short pages. len(data) is limit minus however many rows in that window were sub-floor, so a full page looks like a partial one.
2. The tail is unreachable. With total_count (added in feat(b2b): carry a total row count in the analytics envelopes #15) reporting the floor-gated total, a client that asks for limit=total_count still gets fewer than total_count rows whenever any row inside that window is suppressed — and asking again with the same limit returns the identical short page. There is no limit value that reliably yields the last rows.
3. OFFSET walks the wrong sequence. Offsets index into raw rows, not visible ones, so a client stepping offset by len(data) silently skips visible rows; stepping by limit is correct but then page boundaries don't line up with anything the client can observe.

The floor is 5 for the b2b_dashboard tenant, so this bites any org with sub-floor course runs, contracts, or programs — most acutely small orgs, where suppressed rows are a large fraction of the grain.

Steps to Reproduce

Against GET /api/v1/analytics/organizations/{org_slug}/enrollment-funnel for an org with, say, 352 course-run rows of which 12 have enrolled_learners < 5:

1. GET ...?limit=200&offset=0 — returns fewer than 200 rows (however many of the first 200 raw rows were sub-floor).
2. Note total_count in the envelope reports 340 (the floor-gated total, correctly).
3. GET ...?limit=340&offset=0 — returns 328 rows, not 340.
4. Repeat step 3. Same 328 rows. The remaining 12 visible rows cannot be retrieved at any limit below max_page_size, and above 352 the endpoint 422s.

Possible Solution

Push the primary-cohort gate into the SQL WHERE clause so LIMIT/OFFSET operate on the visible result set:

SELECT ... FROM <schema>.<mv>
WHERE organization_key = %s AND <primary_cohort> >= %s
ORDER BY ... LIMIT %s OFFSET %s

build_count already does exactly this gating for the count (added in #15 — see the build_count docstring for why the count could not be a plain COUNT(*)), so the predicate and its justification already exist; this would reuse it in build_select. That makes len(data) == min(limit, total_count - offset) hold, and makes OFFSET index the same sequence the client sees.

suppress_small_cohorts should stay as-is regardless — it still has to null secondary counts and their derived values, and keeping the row-drop there as well is a harmless belt-and-braces on the chokepoint.

Additional Details

Found while consuming total_count in the MIT Learn B2B analytics dashboard: mitodl/mit-learn#3679

The dashboard currently works around it by only offering its "Show all" control when the click would actually raise the limit, so a section that has already asked for total_count shows an honest "Showing 328 of 340." with no further action offered rather than a button that does nothing (mitodl/mit-learn@cbbbaa110). That is a UI mitigation, not a fix — the last rows are still unreachable.

Not urgent at current org sizes (the dashboard asks for 200 and no org is near that yet), but it should be fixed before any org's grain approaches the page size, and the fix is small.