feat(b2b_analytics): add learner-grain MVs for the learner-records API

[blarghmatey]

What are the relevant tickets?

N/A. Upstream of the learner-records tenant specced in mitodl/ol-analytics-api#55 (docs/b2b-learner-records-design.md §1, gaps 1, 3, 4, 5). Depends on mitodl/ol-infrastructure#5835.

Description (What does it do?)

Every mv_b2b_* view is pre-aggregated, so nothing can back a per-learner API. This adds two StarRocks MVs in a new b2b_learner_records database (models/b2b_learner_records/). They're kept out of b2b_analytics so these PII-bearing views and the k-anonymized aggregate views can be granted separately.

• mv_b2b_learner_enrollment, at org × contract × course run × learner. It uses the funnel view's joins without the group by, plus dim_user, grade and certificate. is_passing and certificate state are emitted separately, and the API derives completion_status from both.
• mv_b2b_learner, at org × learner. It unions the membership roster (bridge_user_organization) with active enrollments, so assigned-but-unstarted seats appear. It emits membership_source (roster / enrollment / both). Rollups count active enrollments only, which matches the API default include_inactive=false.

Choices a reviewer should check:

• The organization is attributed only through the contract that owns the course run, never through organization_administration_report's free-text organization_key fallback.
• learner_id is user_global_id, with a not_null test in both views. user_pk is emitted only as an internal join key, because it can still re-key.
• record_updated_on is the updated_since cursor. It covers enrollment, grade and certificate changes only, plus program certificates in mv_b2b_learner. Roster changes, dim_user edits and course-run edits carry no change timestamp upstream, so they don't move it. Full change tracking is a follow-up.
• mv_b2b_learner reads the dimensional sources directly instead of selecting from mv_b2b_learner_enrollment. materialized_view_relations refreshes MVs in sorted name order, so the rollup would refresh first and serve the previous refresh's data.
• program_certificates_earned counts a program certificate only if the program contains a course the learner is actively enrolled in under the org's contracts. Otherwise a certificate earned outside the licence would be reported to the partner.

There are no consent or activity columns yet. The API returns outcomes_shared: false and null activity fields until those land upstream.

No Dagster change is needed. The refresh asset and the drift check both take schema-qualified relations from the manifest, and the drift query already filters on a list of schemas.

How can this be tested?

• ol-dbt validate: 0 errors. Pre-commit (sqlfluff, yamlfmt, yamllint) passes.
• I rendered both models against production dimensional Iceberg tables in DuckDB (ol-dbt local register --database ol_warehouse_production_dimensional). These checks ran on DuckDB. Neither model has been built on StarRocks yet. Results:
  • mv_b2b_learner_enrollment: 37,188 rows, all unique on (contract_pk, courserun_pk, user_pk). 0 null user_global_id, 0 null record_updated_on. 21 distinct sso_organization_ids, 66 contracts.
  • Reconciled against mv_b2b_enrollment_completion_funnel: 1,184 / 1,184 (contract, run) rows match on enrolled, active, passing and certified learner counts.
  • mv_b2b_learner: 16,976 rows, unique on (org, user), 0 null user_global_id. Membership: 10,622 both, 45 enrollment only, 6,309 roster only.
  • courses_enrolled agrees with the enrollment view's active-run count for 10,667 / 10,667 enrolled (org, learner) pairs. 0 rows have passed or certified above enrolled, and 0 roster-only rows have courses.
  • mv_b2b_learner.record_updated_on is null for exactly the 6,309 roster-only learners and for none of the others. 663 learners hold a counted program certificate.
• After #5835 deploys and this merges, the next starrocks_dbt_assets run creates both MVs. Confirm with DESC b2b_learner_records.mv_b2b_learner_enrollment and DESC b2b_learner_records.mv_b2b_learner.

Additional Context

Findings from those checks that the API tenant has to handle:

• 13,649 enrollment rows have a null sso_organization_id: their org has no Keycloak UUID, so an API filtering on it can't reach them. The existing MVs have the same gap.
• 101 enrollment rows carry a non-revoked certificate with is_passing false. completion_status has to treat an unrevoked certificate as certified without requiring is_passing.

Checklist:

• Deploy feat(starrocks): provision a b2b_learner_records database ol-infrastructure#5835 (creates the b2b_learner_records database and its app grants) before merging

🤖 Generated with Claude Code

https://claude.ai/code/session_017FmjwANjfgtuKAtsRPf6Ly

[github-actions]

🔎 ol-dbt impact — column-level blast radius

0 breaking, 0 surrogate-key regeneration, 0 warning, 2 info across 2 changed model(s).

Details

• ℹ️ mv_b2b_learner — courses_certified, courses_enrolled, courses_passed, email, first_enrolled_on, full_name, is_organization_manager, last_enrolled_on, membership_source, organization_key, organization_name, program_certificates_earned, record_updated_on, sso_organization_id, user_global_id, user_pk → 0 downstream model(s)
• ℹ️ mv_b2b_learner_enrollment — b2b_contract_name, certificate_is_revoked, certificate_issued_on, contract_id, contract_pk, courserun_end_on, courserun_pk, courserun_readable_id, courserun_start_on, courserun_title, email, enrollment_created_on, enrollment_is_active, enrollment_mode, enrollment_status, full_name, grade_value, is_passing, letter_grade, organization_key, organization_name, record_updated_on, sso_organization_id, user_global_id, user_pk → 0 downstream model(s)

_{Posted by ol-dbt impact (annotate-only — does not block merge).}

[Copilot]

🟡 Changes recommended

Cursor completeness, nullable learner identifiers, and PII schema isolation must be addressed.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds learner-grain StarRocks materialized views for the proposed B2B learner-records API.

Changes:

• Adds enrollment-detail and organization-level learner views.
• Adds learner identity, roster, completion, and certificate fields.
• Documents the new models and dimensional sources.

File summaries

File	Description
mv_b2b_learner.sql	Adds organization-level learner rollups.
mv_b2b_learner_enrollment.sql	Adds per-enrollment learner records.
_b2b_analytics__sources.yml	Declares learner and roster sources.
_b2b_analytics__models.yml	Documents both materialized views.

Review details

Suppressed comments (4)

src/ol_dbt/models/b2b_analytics/mv_b2b_learner.sql:132

• record_updated_on cannot serve the promised updated_since contract for this row. A newly added roster-only learner gets NULL, roster/manager changes and removals never advance it, dim_user changes are ignored, and issuance/revocation of the program certificate counted above is not included. Incremental clients will therefore miss changed learner records. Please propagate timestamps for every source that can alter this row (and define deletion/tombstone behavior) before using this field as the cursor.

    er.record_updated_on

src/ol_dbt/models/b2b_analytics/mv_b2b_learner_enrollment.sql:46

• This cursor advances only for enrollment, grade, and certificate facts, although the API contract defines updated_since as records changed since the cursor. Changes to projected learner identity (email/full_name) or course-run metadata alter the API record without changing this value, so incremental clients retain stale data. Include change timestamps for all projected mutable sources or narrow the API's documented sync semantics.

    greatest(
        coalesce(e.enrollment_updated_on, e.enrollment_created_on),
        coalesce(g.grade_updated_on, e.enrollment_created_on),
        coalesce(cert.certificate_updated_on, e.enrollment_created_on)
    )                                                                                   as record_updated_on

src/ol_dbt/models/b2b_analytics/mv_b2b_learner.sql:117

• The linked API schema requires learner_id to be a non-null UUID and orders /learners by that unique field, but this model explicitly permits user_global_id to be null (as documented in the accompanying YAML). Such rows cannot be serialized under that contract or paged by the promised key. Either exclude/backfill users without a global ID or make the API contract and identifier strategy nullable-safe.

    u.user_global_id,

src/ol_dbt/models/b2b_analytics/mv_b2b_learner_enrollment.sql:27

• The linked API schema requires learner_id to be a non-null UUID, while this view does not enforce a non-null user_global_id and its YAML states null is valid. Any such row cannot satisfy the enrollment response contract or its unique paging order. Add an explicit non-null policy here, backfill the identifier, or revise the API contract.

    u.user_global_id,

• Files reviewed: 4/4 changed files
• Comments generated: 2
• Review effort level: Balanced (auto)

Note

Copilot is running an experiment and ran this review at Balanced.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[blarghmatey]

Addressed Copilot's review. The two schema threads are fixed and resolved (f62117d, plus mitodl/ol-infrastructure#5835). Copilot's four suppressed comments:

• learner_id nullable (both views): both now carry a not_null test on user_global_id, so a null fails the build instead of reaching a partner. Production has 0 nulls today.
• record_updated_on incomplete (both views): mv_b2b_learner now also includes program-certificate update times. For the rest, the documented scope is narrowed: the cursor covers enrollment, grade and certificate changes. Roster, dim_user and course-run edits carry no change timestamp upstream, so there is nothing to propagate yet. A partner picks those up from a full reload of the bulk export. Real change tracking, including a removal/tombstone for learners dropped from a roster, is filed as a follow-up (a dbt snapshot over the projected fields).