feat(dimensional): add a user_fk-keyed learner x course run x day activity fact

[blarghmatey]

What are the relevant tickets?

N/A. Part of the learner-records API work (spec: mitodl/ol-analytics-api#55, docs/b2b-learner-records-design.md §1 gap 2).

Description (What does it do?)

Adds afact_learner_courserun_daily_activity: one row per (platform, user_fk, course run, day) with any tracked course activity, rolled up from the five event facts.

Per-learner activity exists today only in organization_administration_report, which groups by dim_user.email. That email is a coalesce across a person's accounts and moves when they edit it, so joining the report back to a learner drops or mis-attributes some. The learner-records API needs last_active_on, days_active and per-run activity counts keyed on the same user_fk as the enrollment, grade and certificate facts.

Columns: videos_played, problems_attempted, navigation_events, discussion_events, chatbot_interactions. last_active_on is max(activity_date_key) and days_active is count(distinct activity_date_key) over this table.

The day is each event's date_fk (int YYYYMMDD, FK to dim_date), emitted as activity_date_key. It started as cast(event_timestamp as date), which Trino evaluates in each value's own offset but DuckDB evaluates in the session zone, so the same input produced different rows per engine. date_fk is derived from event_timestamp_iso8601 in the source's own timezone, which is the day Trino's cast already produced.

Metric definitions follow the report, except:

• problems_attempted counts problem_check only. The report's problems_count also counts showanswer.
• Chatbot submits with no block_id still count. The report's session_id || block_id makes them null and drops them.
• Canvas chatbot events are excluded, since their course ids aren't Open edX runs.
• Certificate and enrollment days aren't activity here. The report's active_count counts certificates.

It's a full-rebuild table (the dimensional default), not incremental, because user_pk can still re-key and an incremental table would keep old days under a stale key.

This also changes tfact_problem_events (from review). It is incremental, and its watermarks only re-select events newer than the last run, so a dim_user re-key left historical rows under the obsolete user_fk — the hazard tfact_grade and tfact_certificate already guard against with a stale_user_fk_lookup. It now reads back rows at or below the watermark whose stored user_fk no longer matches what dim_user resolves, re-resolves the key per platform (one equi-join CTE per platform, each pruning to one partition), and unions them in; delete+insert on event_id replaces them. Rows above the watermark are excluded so an event the source CTEs re-select cannot also arrive from there and double-insert. It is meant to take effect on ordinary incremental runs, without a full refresh.

How can this be tested?

• ol-dbt validate: 0 errors. Pre-commit (sqlfluff, yamlfmt, yamllint) passes. CI is green on this branch.
• Not exercised locally or in CI: the new tfact_problem_events CTEs sit inside {% if is_incremental() %}, and both dbt parse and sqlfluff render is_incremental() as false. That branch has not been run against a real target.
• I reconciled the model against the production organization_administration_report per (email, day, run) in DuckDB, over production dimensional Iceberg tables scoped to MITx Online course runs under a B2B contract. It has not been built on Trino.
  • 80,469 rows, unique on the grain and on activity_key. 0 null courserun_fk. 8,612 learners, 866 runs.
  • 80,028 (email, day, run) rows appeared in both. videos_played equalled the report's videos_watched on 79,989 of them (99.95%). Chatbot interactions were at least the report's on all 80,028.
  • 420 days appeared only in the fact, and 1,930 active days only in the report. None of the 1,930 had video or chatbot activity in the report, which fits the definition differences above (certificate days, showanswer-only days) or email attribution.
  • Those numbers were measured on the earlier revision, which bucketed with cast(event_timestamp as date) and DuckDB pinned to UTC. Rather than rebuild the fact after rebucketing onto date_fk, I checked the two are the same day: across all five event facts over the same B2B-scoped rows (5,420,066 rows), date_fk equals the calendar date at the front of event_timestamp_iso8601 on every row, with none null. That is the date Trino's cast returns, so the rebucketing leaves production output for this population unchanged.
• Run it yourself: dbt build --select afact_learner_courserun_daily_activity, which runs the uniqueness and not_null tests with it.

Additional Context

• Day boundaries are not UTC. Measured on production: 15.2M of tfact_discussion_events' 17.5M rows carry a local offset (-04:00/-05:00) rather than Z, and 2.9M of those fall on a different day under UTC. Bucketing on date_fk keeps the source-local day the existing report already uses; converting to UTC would have changed production days for those rows.
• Reading activity_date_key as a date: join dim_date rather than casting the integer.
• Wiring this into the learner-records MVs (feat(b2b_analytics): add learner-grain MVs for the learner-records API #2669) is a follow-up. Those MVs aren't merged yet, and the fact has to exist in Glue first.

🤖 Generated with Claude Code

https://claude.ai/code/session_017FmjwANjfgtuKAtsRPf6Ly

[github-actions]

🔎 ol-dbt impact — column-level blast radius

0 breaking, 0 surrogate-key regeneration, 0 warning, 1 info across 1 changed model(s).

Details

• ℹ️ afact_learner_courserun_daily_activity — activity_date_key, activity_key, chatbot_interactions, courserun_fk, courserun_readable_id, discussion_events, navigation_events, platform, problems_attempted, user_fk, videos_played → 0 downstream model(s)

_{Posted by ol-dbt impact (annotate-only — does not block merge).}

[Copilot]

🟡 Changes recommended

Upstream stale user keys and session-dependent date bucketing can produce incorrect learner activity records.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a full-rebuild learner/course-run/day activity fact for the learner-records API.

Changes:

• Aggregates five event facts by learner, course run, platform, and day.
• Documents grain, metrics, foreign keys, and dbt tests.

File summaries

File	Description
afact_learner_courserun_daily_activity.sql	Builds the daily activity fact.
_dim__models.yml	Documents and tests the new model.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Balanced (auto)

Note

Copilot is running an experiment and ran this review at Balanced.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.