From 8b224fbc6a572f9638410ee3fe05fda4e72560b0 Mon Sep 17 00:00:00 2001 From: Tobias Macey Date: Thu, 10 Sep 2026 14:38:16 -0400 Subject: [PATCH 1/7] docs: spec the B2B learner records tenant and provider authorization The learner records design had only ever existed as untracked files, so nobody could review it or link to it. This commits the one-pager, the data-backing/tenancy doc and the draft OpenAPI contract as they stood, plus a proposal for the blocking open question: who authorizes a training provider to read an organization's learner data. The proposal separates the provider's credential (a Keycloak client, Pulumi-owned per the B2B onboarding substrate/tenant split) from the per-organization grant (a mitxonline record). The organization authorizes, MIT records and enforces it, and phase 1 lets the organization see and revoke grants before self-service authorization exists. It is marked pending sign-off. Nothing is implemented. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EBTqmr3ki2pAAP1EeWH7wg --- docs/b2b-learner-records-design.md | 333 +++++++ docs/b2b-learner-records-onepager.md | 225 +++++ ...-learner-records-provider-authorization.md | 197 +++++ docs/openapi/b2b-learner-records-v1.yaml | 816 ++++++++++++++++++ 4 files changed, 1571 insertions(+) create mode 100644 docs/b2b-learner-records-design.md create mode 100644 docs/b2b-learner-records-onepager.md create mode 100644 docs/b2b-learner-records-provider-authorization.md create mode 100644 docs/openapi/b2b-learner-records-v1.yaml diff --git a/docs/b2b-learner-records-design.md b/docs/b2b-learner-records-design.md new file mode 100644 index 0000000..563acbc --- /dev/null +++ b/docs/b2b-learner-records-design.md @@ -0,0 +1,333 @@ +# B2B Learner Records — data backing, scope, and tenancy + +Companion to [`openapi/b2b-learner-records-v1.yaml`](openapi/b2b-learner-records-v1.yaml). +Status: proposal. Nothing here is implemented. + +## The ask + +An M2M-authenticated API giving a B2B partner (or a training provider they +contract) a per-learner view of progress against their site licence: who is on +the licence, when they enrolled, when they were last active, whether they +completed each course. + +## 1. Does the data exist? + +Mostly. Every identity, enrollment and completion field is available in the +dimensional layer today. **Activity — including "last active", the field the ask +names explicitly — is the gap.** + +### Available today + +| Need | Source | Note | +| --- | --- | --- | +| Stable learner id | `dim_user.user_global_id` | Keycloak `sub`; already the cross-system identifier. Survives email changes. | +| Email, full name | `dim_user.email`, `.full_name` | `email` is a *coalesce* across platform accounts — the most recently active address, not necessarily the one the org enrolled them under. Not a safe join key. | +| Org membership roster | `bridge_user_organization` | `(user_fk, organization_fk, is_manager)`. Includes members with zero enrollments. | +| Org identity | `dim_organization` | `sso_organization_id` is the Keycloak org UUID the existing tenant already filters on. | +| Contract | `dim_contract` | Name, term, `b2b_contract_max_learners` (seat limit), membership type. | +| Contract → course run | `bridge_organization_courserun` | Grain `(org, contract, courserun)`. | +| Enrolled when | `tfact_enrollment.enrollment_created_on` | Plus `enrollment_is_active`, `enrollment_mode`, `enrollment_status`. | +| Completed | `tfact_grade` (`is_passing`, `grade_value`, `letter_grade`) + `tfact_certificate` (`certificate_issued_on`, `certificate_is_revoked`) | Two distinct signals — see below. | +| Course run metadata | `dim_course_run` | Title, start/end, `is_current`. | + +### The gaps + +**1. No learner-grain view exists.** All six `b2b_analytics` MVs are +pre-aggregated by design. A learner roster needs a new dbt model — call it +`mv_b2b_learner_enrollment` at `(org × contract × courserun × learner)` — plus a +learner-grain rollup. This is the bulk of the upstream work, but it is +straightforward: the joins are exactly those in +`mv_b2b_enrollment_completion_funnel.sql` with the `group by` removed and +`dim_user` added. + +**2. Activity is keyed on email, not `user_fk`.** Per-learner activity lives +only in `reporting.organization_administration_report`, whose grain is +`(courserun_readable_id, organization_key, user_email, activity_date)`. The +dimensional facts key on `user_fk`. Joining the report back to `dim_user` on +email will silently drop or mis-attribute a tail of learners, because +`dim_user.email` is itself a coalesced choice among a person's platform +accounts. Two ways out: + +- add `user_fk` to `organization_administration_report`, or +- build the activity rollup directly from `tfact_video_events`, + `tfact_problem_events`, `tfact_discussion_events`, + `tfact_course_navigation_events` and `tfact_chatbot_events`, which already + carry `user_fk`. + +The second is more work but produces a correct `user_fk`-keyed +`(learner × courserun × day)` fact that the aggregate MVs would also benefit +from. Recommend the second. + +Until one lands, `last_active_on`, `days_active` and the per-run activity +counters ship as `null`. The spec marks them `x-data-readiness: pending-model` +so a partner sizing the integration knows which columns to expect empty. + +**3. `organization_key` is unreliable for activity attribution.** In +`organization_administration_report` it is +`COALESCE(b2b_contract_to_courseruns.organization_key, user_course_roles.organization)` +— the fallback is free text. The existing MVs drop rows where it is null. A +learner-grain view must attribute through the contract, never the free-text +fallback, or a partner will see learners who are not theirs. + +**4. Roster and enrollment disagree, and that is signal.** A learner can be on +`bridge_user_organization` with no enrollment (assigned, unstarted seat) or +enrolled under a contract without a roster row (provisioning lag). Rather than +picking one source and hiding the discrepancy, the spec surfaces +`membership_source: roster | enrollment | both`. The assigned-but-unstarted seat +is normally the most actionable row on a training dashboard, and it is invisible +in every existing MV. + +**5. Passing ≠ certified.** `tfact_grade.is_passing` and an unrevoked +`tfact_certificate` are separate facts that legitimately disagree — certificates +are issued on a schedule after grading, and audit-mode enrollments never +certify. Exposing both raw invites partners to build their own inconsistent +derivation, so the spec adds a derived `completion_status` enum +(`not_started | in_progress | passed | certified`) as the single field to read, +with the raw signals alongside. + +**6. Learner consent does not exist yet, and it gates the outcome columns.** The +only consent-shaped fields in the warehouse today are email-marketing opt-in +(`irx__*__email_opt_in`), Emeritus/Global Alumni GDPR consent dates, and +Keycloak's OAuth client-consent representations. None of them is a +learner-to-organization data-sharing consent. It needs to resolve per +`(user_fk, organization_fk)` — either a column on `bridge_user_organization` or +a fact keyed the same way — so the learner-grain models can join it directly. +See §4. + +**7. MITx Online only.** Every `b2b_analytics` MV filters +`org.platform = 'mitxonline'`. xPro B2B is order-based +(`int__mitxpro__b2becommerce_b2border`) and has no contract→courserun bridge. +Out of scope; say so to partners rather than letting them discover it. + +**8. Freshness.** The MVs are `refresh_method='manual'`, driven by a Dagster +asset. "Last active" is as stale as the last refresh. The `as_of` envelope the +existing tenant already returns carries this honestly and doubles as the +`updated_since` cursor for incremental sync. + +## 2. Existing tenant, or a new one? + +**A new tenant.** Not a close call. + +### It has the opposite privacy posture + +`b2b_dashboard` is defined by what it does not emit. Its description string is +"Aggregated-only… No individual learner PII", and `core/anonymization.py` +enforces a k-anonymity floor of 5 on every row, with `CohortPolicy.__post_init__` +raising at import time if a derived column names an unfloored cohort. That +machinery, and the tests around it, exist to make "this tenant cannot disclose an +individual" a property you can check by reading one file. + +A learner roster is individually identifying by construction. Putting it behind +the same mount means either exempting one router from `suppress_small_cohorts` — +which makes the invariant false while leaving the code that asserts it in place — +or applying the floor and shipping an endpoint that suppresses every row it +exists to return. + +### Its auth model cannot run in an M2M flow + +`require_org_manager` needs three things a client-credentials token does not +have: an `organization` claim to check membership against, a `sub` to name in +the MITx Online round-trip, and a human whose `is_manager` flag was curated in +Django admin. There is no user in this flow. The org grant has to come from the +*client's* registration, which is a different check, reached by a different +code path, with different failure modes. + +### The blast radius attaches to a different principal + +Credential issuance and revocation, rate limits, audit logging, and the DPA that +permits learner-level disclosure all attach to the partner client, not to a +logged-in org manager. Those are exactly the knobs the architecture already puts +in each tenant's own `config.py` and `auth.py`. + +### It costs one package and one registry line + +`main.py`'s `TENANTS` list is the documented extension point. A new tenant gets +its own OpenAPI document at its own mount — which is precisely the artifact you +want to hand to a partner, without the aggregate dashboard's endpoints in it. + +It also positions the pending tenant-isolation work correctly: the README already +flags that per-tenant Vault role / StarRocks user with schema-scoped grants is +outstanding. The PII-bearing tenant is the one that most needs it, and separating +now means the learner-grain schema can be granted to that role alone later. + +### Proposed shape + +``` +src/ol_analytics_api/tenants/b2b_learner_records/ + app.py # create_app(); title/description say "identifiable learner records" + config.py # own StarRocks schema, own page caps, own audit settings + auth.py # client-credentials principal, org-grant check, scope gating + models.py # Learner, Enrollment, CourseRun — no CohortPolicy + routers/ + organizations.py +``` + +Mounted at `/api/v1/learner-records`. Reuses `core/db/*`, `core/health.py`, +`core/errors.py`, `core/observability/*` unchanged. Deliberately does **not** +import `core/anonymization.py` — a reader should be able to tell the two tenants' +postures apart from the import list. + +Naming it `b2b_learner_records` rather than `b2b_dashboard_v2` or similar keeps +the distinction legible at the mount path, in log lines, and in the readiness +sub-path. + +## 3. Scope + +### In + +Three collections, all org-scoped, all read-only: + +- `GET /organizations/{organization_id}/learners` — learner grain, roster plus + progress rollup. Answers "who is on our licence and how are they doing". +- `GET /organizations/{organization_id}/enrollments` — `(learner × course run)` + grain. Answers "did this learner complete this course". The primary endpoint; + `/learners` is a convenience rollup over the same records. +- `GET /organizations/{organization_id}/courses` — the contracts and course runs + the identifiers refer to. Small, slow-changing, no personal data, cacheable. + +Plus `updated_since` incremental sync on the two record collections, because a +partner mirroring into their own LMS should not re-read the whole licence daily. + +**Two delivery channels over one schema.** The REST API above, and a +per-organization bulk export (S3/SFTP) written on each refresh, discoverable via +`/organizations/{id}/exports`. The export is a second encoding of the same +records under the same field names, produced by the same query with the same +consent enforcement — not a second data product with its own semantics. A +partner doing full reloads into an internal LMS should use the export; a partner +rendering a dashboard should use the API. + +### Out (for now) + +- **Program-grain progress.** `tfact_enrollment.enrollment_scope='program'` and + `tfact_certificate.certificate_scope='program'` exist, so this is additive + later. Left out of v1 to keep the first contract small. +- **Per-block / per-assessment detail.** `afact_problem_engagement` and friends + are per-block, and exposing them is a different product (an LRS feed, plausibly + xAPI/Caliper) with a much larger surface. +- **Write operations.** Seat assignment and enrollment stay in MITx Online. +- **xPro B2B.** No contract→courserun mapping exists. +- **Learner-level data for the interactive MIT Learn dashboard.** An org manager + arguably has the same entitlement, but that is a user-authenticated consumer + with its own governance question. If it is wanted later, `b2b_dashboard` can + add a router over the same StarRocks views under its own gate. Keeping the + concerns separate now is cheaper than un-merging them later. + +### Identity is a contractual limit, not a consent one + +`learner-records:read` returns records with `email` and `full_name` as `null`; +`learner-records:read-pii` populates them. Both scopes return the same progress +data. + +This split is **not** the consent mechanism — consent governs outcomes (§4), and +the organization already holds its learners' names and addresses because it +assigned the seats. The split exists because a contracted training provider is a +different principal from the organization: "the org may see it" does not settle +"the provider may see it." Since `learner_id` is stable, a provider that enrolled +the learners can join on an identifier it already supplied and never receive +contact details from MIT at all. + +Whether any given provider gets the PII scope is a contractual call, and one +nobody has made yet — see §4's open questions. + + +## 4. Learner consent + +**Consent gates outcomes, not identity.** The organization already holds its +learners' names and addresses — it assigned the seats. What a learner opts into +sharing is their *course status*: completion, progress and activity. So consent +is enforced by suppressing outcome fields on a record that still appears, rather +than by excluding the record. + +Concretely, every learner and enrollment record carries `outcomes_shared`. When +it is false, every progress, completion and activity field on that record is +null; identity, contract, course run and enrollment facts are unaffected. The +envelope's `outcomes_withheld_count` reports how many records in the result are +in that state. + +The field does not exist upstream yet. Enforcement fails closed, so until it +ships every record reads `outcomes_shared: false` and the outcome columns are +uniformly null. That is a degraded response rather than an empty one, which +means partner integration can proceed against real records. + +### Why suppression rather than exclusion + +Excluding non-consenting learners entirely was the obvious first design, and it +is worse on every axis that matters here: + +* **Seat accounting breaks.** A 50-seat licence would read as 12 learners, and + the organization's own roster would not reconcile against the API. +* **It protects nothing extra.** The organization holds the roster and the + identities already. Any exclusion is trivially reversible by set difference. +* **It complicates sync.** Under exclusion, a withdrawal removes a row, and + absence is not a signal an incremental consumer can act on — so withdrawals + would need a separate tombstone record type to propagate at all. Under + suppression a withdrawal is simply a changed record with `outcomes_shared: + false` and nulled outcomes; a client that upserts normally drops the data it + held, with no extra object type and no retention window to reason about. + +That third point is the one that would have been expensive to discover late. + +### What the upstream field must provide + +1. **Withdrawal must be a retained state change, not a deleted row.** A deleted + consent row is indistinguishable from one never granted, and the record's + `updated_since` timestamp would not move — so the withdrawal never reaches + the partner holding a copy. Model it as a status plus a change timestamp on a + persistent row. +2. **Resolvable per `(learner, organization)`.** A learner holding seats under + two organizations should be able to share with one and not the other. A + global flag works mechanically but makes withdrawal all-or-nothing. +3. **A change timestamp**, since `updated_since` sync is driven off it. +4. **Three states distinguishable upstream** — `never_asked`, `declined`, + `withdrawn` — even though all three render as `outcomes_shared: false`. The + organization needs the breakdown to run its own opt-in campaign. +5. **It has to reach the warehouse**, as a column on `bridge_user_organization` + or a fact keyed on `(user_fk, organization_fk)`. The API cannot consult MITx + Online per row. + +### Settled: aggregates are exempt + +The existing `b2b_dashboard` tenant's k-anonymized org-level views disclose no +individual and continue to cover the whole cohort. No consent join, no change to +`mv_b2b_*`. + +### Open: does withholding outcomes actually protect the learner? + +This is the sharpest unresolved issue in the design. + +Because the organization holds the full roster and the identities, it can see +exactly which of its learners carry `outcomes_shared: false`. The mechanism +therefore conceals a learner's *outcomes* but publishes their *decision* — and +an employee visibly declining to share progress with their employer may be more +exposed than one whose completion data was simply shared. + +No response shape fixes this while the record remains individually identifiable. +The available directions are: + +* **Contractual** — terms of use forbidding adverse action on non-participation. + Cheap, and the only option that preserves the current record shape. +* **Aggregate-only participation reporting** — expose consent coverage as a + floored count per organization and never per learner, accepting that seat + reconciliation degrades. +* **Revisit what the learner is asked** — if consent cannot be declined without + the employer knowing, the consent language should probably say so. + +Worth putting to whoever owns the consent language early, since the third option +changes their work rather than ours. + +### Open: does a contracted provider receive identity at all? + +Identity is available to the *organization*. A provider is a different +principal, and the API supports either answer today via the `read` / +`read-pii` scope split (§3). It needs a policy owner rather than a default. + +### Open: who authorizes a provider, through what workflow? + +No provisioning design exists. MIT-issued-by-ticket is simplest and leaves the +organization with no visible record of who can read its learners' data; +org self-service authorization is more work but puts the data relationship where +it belongs. This is an operational gap, not a configuration detail, and it +blocks onboarding the first provider rather than the first organization. + +Proposed answer, pending sign-off: +[`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). diff --git a/docs/b2b-learner-records-onepager.md b/docs/b2b-learner-records-onepager.md new file mode 100644 index 0000000..4869472 --- /dev/null +++ b/docs/b2b-learner-records-onepager.md @@ -0,0 +1,225 @@ +# B2B Learner Records — high-level design + +**Status:** proposal · **Owner:** Data Engineering · **Consumers:** B2B site-licence organizations and their contracted training providers + +## What it is + +A read-only, machine-authenticated interface giving an organization — or a +training provider it has contracted — per-learner progress against its site +licence: who is on the licence, when they enrolled, when they were last active, +and whether they completed each course. It exists so partners can render learner +progress inside their own systems, or mirror it into an internal LMS. + +It is deliberately not the same thing as the existing B2B analytics dashboard, +which serves aggregate, k-anonymized, org-level figures to a logged-in manager. + +## Decisions + +**A separate tenant, not a new endpoint on the existing one.** The existing B2B +analytics tenant is defined by not emitting individuals — a k-anonymity floor is +enforced on every row and asserted at import time. This service is individually +identifying by construction. Its authorization model is also incompatible: the +existing tenant resolves a logged-in user's manager status, and there is no user +in a machine-to-machine flow. Separate tenants cost one package and one registry +entry, and give the partner its own API document. + +**Machine-to-machine auth, with per-client organization grants.** Each client is +issued its own credentials and an explicit set of organizations it may read. A +provider working for several organizations holds one grant per organization, so +a contract ending revokes exactly one. Requests outside the grant are refused +identically to requests for organizations that do not exist, so the interface +cannot be used to enumerate MIT's customers. + +**Two delivery channels over one schema.** A paged REST API for dashboards and +incremental sync, and a per-organization bulk export (S3/SFTP) on the refresh +cadence for partners loading into their own systems. Same records, same field +names, same consent enforcement — the export is a second encoding of the API's +schema, not a second data product. + +**Consent gates outcomes, not identity.** Organizations already hold their +learners' names and addresses; they assigned the seats. What a learner opts into +sharing is their *course status* — completion, progress, activity. So consent is +enforced on the outcome fields, and identity is governed by the contract with +the partner rather than by the learner's consent choice. + +**Aggregates are exempt.** The existing k-anonymized org-level views disclose no +individual and continue to cover the whole cohort. No consent join, no change to +the aggregate models. + +**Consent enforcement fails closed.** No recorded opt-in means no outcome data. +Because suppression is per-field rather than per-record, the service can ship +before the consent field exists — every record simply reads `outcomes_shared: +false` with outcomes null, which is a degraded response rather than an empty +one, so partner integration can proceed against real records. One upstream +property is a hard requirement and cheap only if specified now: *withdrawal must +be a retained state change, not a deleted row*. A deleted consent row is +indistinguishable from one never granted and does not move the record's change +timestamp, so the withdrawal never reaches the partner holding a copy. + +## Shape + +Three collections, all organization-scoped and read-only: **learners** (roster +and rollup), **enrollments** (learner × course, the grain that answers "did they +complete it"), and **courses** (the contracts and course runs the identifiers +refer to), plus an **exports** manifest listing the bulk files. Incremental sync +is driven by a refresh cursor; a consent withdrawal arrives on it as an ordinary +changed record with outcomes nulled, so a client that upserts normally drops the +data it held. + +Out of scope for v1: program-level progress, per-assessment detail, any write +operation, and non-MITx-Online platforms. + +### Example records + +Every collection returns the same envelope, with `data` typed to its record: + +```json +{ + "organization_id": "8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21", + "as_of": "2026-08-13T06:15:00Z", + "total_count": 47, + "outcomes_withheld_count": 11, + "data": [] +} +``` + +`GET /organizations/{id}/learners` + +```json +{ + "learner_id": "3e1a9c74-5b2d-4f88-9a01-7c6de2b4f019", + "email": "rgarcia@contoso.example", + "full_name": "R. Garcia", + "organization_id": "8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21", + "organization_name": "Contoso Manufacturing", + "membership_source": "both", + "is_organization_manager": false, + "first_enrolled_on": "2026-02-03T14:22:11Z", + "last_enrolled_on": "2026-05-19T09:04:52Z", + "courses_enrolled": 4, + "outcomes_shared": true, + "outcomes_consent_on": "2026-02-03T14:20:04Z", + "last_active_on": "2026-08-11", + "courses_in_progress": 1, + "courses_passed": 3, + "courses_certified": 3, + "certificates_earned": 3 +} +``` + +`GET /organizations/{id}/enrollments` + +```json +{ + "learner_id": "3e1a9c74-5b2d-4f88-9a01-7c6de2b4f019", + "email": "rgarcia@contoso.example", + "full_name": "R. Garcia", + "organization_id": "8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21", + "contract_id": "6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6", + "contract_name": "Contoso 2026 Site Licence", + "courserun_id": "course-v1:MITxT+14.310x+2T2026", + "courserun_title": "Data Analysis for Social Scientists", + "courserun_start_on": "2026-02-01T00:00:00Z", + "courserun_end_on": "2026-06-30T23:59:59Z", + "enrolled_on": "2026-02-03T14:22:11Z", + "enrollment_is_active": true, + "enrollment_mode": "verified", + "enrollment_status": null, + "outcomes_shared": true, + "completion_status": "certified", + "is_passing": true, + "grade": 0.91, + "letter_grade": "A", + "certificate_issued_on": "2026-07-02T11:00:00Z", + "certificate_is_revoked": false, + "last_active_on": "2026-08-11", + "days_active": 34, + "videos_watched": 212, + "problems_attempted": 88, + "chatbot_interactions": 14 +} +``` + +`GET /organizations/{id}/courses` + +```json +{ + "organization_id": "8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21", + "organization_name": "Contoso Manufacturing", + "contract_id": "6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6", + "contract_name": "Contoso 2026 Site Licence", + "contract_is_active": true, + "contract_start_date": "2026-01-01", + "contract_end_date": "2026-12-31", + "seat_limit": 250, + "courserun_id": "course-v1:MITxT+14.310x+2T2026", + "courserun_title": "Data Analysis for Social Scientists", + "courserun_start_on": "2026-02-01T00:00:00Z", + "courserun_end_on": "2026-06-30T23:59:59Z" +} +``` + +`GET /organizations/{id}/exports` + +```json +{ + "collection": "enrollments", + "as_of": "2026-08-13T06:15:00Z", + "format": "jsonl", + "record_count": 163, + "size_bytes": 214880, + "checksum_sha256": "9f2c1b7ae4d05c8831fbb2e6a0d47c3915ee8b6042d1f7c9a3b508e2d6417f0a", + "uri": "s3://ol-b2b-exports/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz", + "expires_on": "2026-09-12T06:15:00Z" +} +``` + +Both records above show `outcomes_shared: true`. When it is `false`, every field +from `outcomes_consent_on` onward on a learner record, and from +`completion_status` onward on an enrollment record, is `null`; identity, +contract, course-run and enrollment facts are unaffected. That is also how a +consent withdrawal arrives on the sync cursor. + +## Open questions + +**1. Does withholding outcomes actually protect the learner?** Because the +organization holds the full roster, it can identify exactly who did not share by +set difference — whatever the response reports. The mechanism therefore hides a +learner's *outcomes* but not their *decision*, which exposes them to a different +pressure from their employer. Options are contractual (terms forbidding adverse +action on non-participation), or design-side (report participation only in +aggregate above a threshold, and accept degraded seat reporting). Unresolved, +and the sharpest issue in this design. + +**2. Does a contracted provider receive learner identity at all?** Identity is +available to the *organization*. A provider is a different principal, and +"the org may see it" does not settle "the provider may see it." Cheap to support +either way — credentials can be scoped to pseudonymous records — but it needs a +policy owner, not a default. + +**3. Who authorizes a provider, through what workflow?** No provisioning design +exists. MIT-issued-by-ticket is simplest and leaves the organization with no +visible record of who can read its data; org self-service authorization is more +work but puts the data relationship where it belongs. This is an operational gap, +not a configuration detail. + +*Proposed answer, pending sign-off:* +[`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). +The organization authorizes, and MIT records it. The provider's credential +(Pulumi) is separate from the organization grant (mitxonline). Phase 1 grants +are staff-recorded on written authorization, visible to the organization, and +revocable by it. Self-service authorization waits for manager designation via +Keycloak Organization Groups. + +**4. Is consent per-organization or global?** A learner holding seats under two +organizations should be able to share with one and not the other. A single +global flag works mechanically but makes withdrawal all-or-nothing. An input to +the consent design rather than a question this service can answer. + +## Dependencies + +Both are degrading rather than blocking — the service ships without either and +fills in as they land. The learner-consent field, without which every record +reads `outcomes_shared: false`; and a per-learner activity model, without which +"last active" and the engagement counters are null. A provider-authorization +workflow is the one genuine blocker, and only for onboarding the first provider. diff --git a/docs/b2b-learner-records-provider-authorization.md b/docs/b2b-learner-records-provider-authorization.md new file mode 100644 index 0000000..9453453 --- /dev/null +++ b/docs/b2b-learner-records-provider-authorization.md @@ -0,0 +1,197 @@ +# B2B Learner Records: who authorizes a training provider + +Companion to [`b2b-learner-records-design.md`](b2b-learner-records-design.md). +Status: **proposal, pending product-owner sign-off.** Answers open question 3 of +the [one-pager](b2b-learner-records-onepager.md). Questions 1, 2 and 4 stay +open; §6 notes how this changes question 2. + +## 1. The question + +A contracted training provider calls the API with its own client credentials +and reads individually identifying records for an organization's learners. +Something has to say "this provider may read this organization's learners, on +these terms, until this date". The question is who says it and where it is +recorded. + +This is what gates onboarding the first provider. The rest of the service +(tenant, dbt models, consent field) can be built without it. + +## 2. What exists today + +Nothing models the relationship. There is no provider, partner or API-client +record attached to an organization or contract in any system. + +| Piece | Where it lives today | Relevance | +| --- | --- | --- | +| Organization | mitxonline `b2b.models.OrganizationPage`, linked to Keycloak by `sso_organization_id` | The grant's subject. The API's `organization_id` is this UUID. | +| Org manager | mitxonline `b2b.models.UserOrganization.is_manager` | Set only in Django admin (`UserOrganizationAdmin`), row by row, by MIT staff. Never reaches the Keycloak token. | +| Manager designation, future | Keycloak Organization Groups, per mitodl/hq#10594. Labelled C4 in the capability glossary of the provisioning spec (mitodl/mitxonline#3922), from the onboarding RFC mitodl/hq#12784 | Gives the organization a manager role it controls, with an audit trail. Not built. | +| Service-account clients | Pulumi, `ol-infrastructure` `substructure/keycloak/olapps.py` (e.g. `mitxonline-b2b-client`), scopes via `ClientDefaultScopes` | The pattern for issuing a provider's credential. | +| Ownership split | mitxonline PR #3922, `docs/source/b2b/provisioning_api.md` | Pulumi keeps realm, flows, client scopes, **clients** and service-account grants. The mitxonline provisioning API owns everything per customer. | +| Token validation | APISIX in front of ol-analytics-api; the app decodes `X-Userinfo` (`core/auth/userinfo.py`) | No inbound client-credentials principal exists yet. | +| Service-to-service check | `b2b_dashboard` asks mitxonline whether a user is a manager with its own client-credentials token, scope `b2b:manager-check` (`tenants/b2b_dashboard/mitxonline_client.py`, mitxonline `b2b/views/v0/service.py`) | The pattern for checking a grant at request time. | +| Manager UI | mit-learn contract admin page, `app-pages/ContractAdminPage`, which already has a revoke/confirm flow for seat codes | Where an organization would see and revoke grants. | + +## 3. Separate the credential from the grant + +The two shapes in the original question ("MIT issues credentials by ticket" vs +"the org authorizes from its dashboard") conflate two different objects: + +- **The credential** identifies the provider. One Keycloak client per provider, + whatever the number of organizations it works for. It carries no + organization. Issuing it is rare, provider-level, and fits the onboarding + split: clients are substrate, so they stay in Pulumi under review. +- **The grant** is the relationship: provider X may read organization Y's + learners, with or without identity, until a date. It is per customer, it + changes whenever a contract starts or ends, and it is the thing the + organization needs to see. Under the onboarding split it belongs in + mitxonline, next to `OrganizationPage` and `ContractPage`. + +A credential without a grant reads nothing: every request returns the same 403 +as an unknown organization. So issuing credentials can stay an MIT operations +step without MIT deciding who reads whose data. + +## 4. Recommendation + +**The organization authorizes. Never the provider, and never MIT alone.** MIT +records and enforces the organization's decision. + +### Phase 1: staff-recorded grants, org-visible, org-revocable + +Enough to onboard the first provider. + +1. The provider gets a Keycloak client from a reviewed Pulumi change. It holds + no grants. +2. The organization authorizes the provider in writing. MIT confirms with the + organization's contact of record, not a contact the provider supplies. That + closes the obvious social-engineering route of a provider asking MIT for + access on the organization's behalf. +3. MIT staff record the grant in mitxonline (Django admin), with the + authorizing person and a reference to the written authorization. +4. The organization's managers see active and past grants on the mit-learn + contract admin page, and can **revoke** one there. + +Self-service revocation ships in phase 1. Self-service authorization waits for +phase 2, and the asymmetry is deliberate. Revoking can only reduce disclosure, +so the current manager flag (set by MIT staff in Django admin) is authority +enough to act on it. Authorizing increases disclosure of individually +identifying data. That should rest on a manager role the organization controls +and that leaves an audit trail, which is what C4 provides. + +The visible grant list is what separates this from "MIT-issued-by-ticket". The +objection to the ticket-only shape was that the organization has no record of +who can read its learners' data. Phase 1 fixes that without building a +self-service authorization flow. + +### Phase 2: organization self-service authorization + +Blocked by C4. The provider (or MIT staff on its behalf) creates a *pending* +grant naming the organization, and an organization manager approves it in +mit-learn. The provider can request but never activate. MIT staff keep the +ability to revoke, but no longer record authorizations. + +## 5. The grant record + +A mitxonline model in the `b2b` app. Working name `OrganizationDataGrant`. + +| Field | Note | +| --- | --- | +| `organization` | FK `OrganizationPage`. | +| `client_id` | The provider's Keycloak client ID. | +| `provider_name` | Display name for the manager UI. | +| `includes_identity` | Whether records carry `email` and `full_name`. See §6. | +| `status` | `pending`, `active`, `revoked`. | +| `expires_on` | Required. Not later than the end of the organization's latest active contract. | +| `authorized_by_name`, `authorized_by_email` | The organization-side person who authorized it. | +| `authorization_reference` | Ticket or document reference for the written authorization (phase 1). | +| `recorded_by`, `activated_on` | MIT staff user (phase 1) or the approving manager (phase 2). | +| `revoked_by`, `revoked_on`, `revocation_reason` | | + +Two rules, the same ones the design already imposes on consent: + +- **Rows are never deleted.** Revocation is a retained state change. A deleted + grant cannot answer "who could read our data in March", which is the first + question an organization asks after a dispute with a provider. +- **Expiry is computed at check time,** so a grant is active only if + `status = active` and `expires_on` is in the future. A contract ending needs + no job to revoke access. + +## 6. How the API checks a grant + +Per request, in `tenants/b2b_learner_records/auth.py`: + +1. Read the client ID from the validated token. +2. Resolve the client's active grants from mitxonline through a new service + endpoint. Use the same pattern as `b2b:manager-check`, under a new scope + (e.g. `b2b:data-grant-check`). Cache per client for a short TTL (proposed + 60 s). +3. If there is no active grant for the path's `organization_id`, return the + existing 403, identical to the one for an organization that does not exist. +4. If mitxonline is unreachable and the cache is expired, return 503. Never + serve from a stale grant list, because that would keep a revoked grant + readable. + +Revocation takes effect within one TTL. + +### Identity moves onto the grant + +The draft contract gates `email`/`full_name` on the client scope +(`learner-records:read` vs `learner-records:read-pii`). A scope belongs to the +client, and a provider working for two organizations has one client, so a +scope cannot express "identity for A, not for B". Proposed rule: identity is +returned only when **both** the client holds `learner-records:read-pii` **and** +the grant has `includes_identity`. The scope becomes MIT's ceiling for that +provider, and the grant flag is the organization's choice within it. + +This reframes open question 2. Instead of asking whether a provider's +credential gets PII, it asks whether MIT ever sets the PII ceiling for +providers, and what `includes_identity` defaults to. It still needs a policy +owner. + +### Exports need the same gate + +The draft says export files are "readable with the credentials issued +alongside the API client". Those would be standing S3/SFTP credentials outside +the grant check, so revoking a grant would leave the export readable. Proposed: +`/exports` returns short-lived presigned HTTPS URLs, minted per request after +the grant check. No standing storage credentials in v1. A push to a +partner-owned SFTP endpoint can come later, keyed off the same grant. + +## 7. Rejected alternatives + +- **Grant as Keycloak client configuration** (a hard-coded claim listing + organization IDs, or per-organization client scopes). This puts a + per-customer resource on a Pulumi-owned client, contrary to the onboarding + split. It makes every grant change an infrastructure deploy, and the + organization still sees nothing. +- **Service account as a member of the Keycloak organization,** so the + existing organization-membership mapper lists the organization in the token. + It reuses machinery, but mitxonline's org sync would pull the service account + in as a learner. It would then appear in `bridge_user_organization`, the + roster this API serves. +- **Grants landed in the warehouse and joined like consent.** Revocation would + lag the materialized-view refresh (manual, Dagster-driven), and prompt + revocation is the one property a grant must have. +- **Provider self-service.** The provider is the party whose access is in + question. + +## 8. To verify before building + +- Which claim carries the client ID in a Keycloak 26.7 client-credentials + token (`azp`, `client_id`, or both), and whether APISIX's `openid-connect` + plugin populates `X-Userinfo` with it for a bearer-only route. The + learner-records mount needs a bearer-only route; today's routes use the + redirect flow. Check on QA before writing `auth.py`. +- Whether `ContractAdminPage` is the right home for the grant list, or whether + it belongs at organization level. A grant spans contracts. + +## 9. Work this creates + +| Repo | Work | Phase | +| --- | --- | --- | +| mitxonline | `OrganizationDataGrant` model, Django admin, service endpoint + OAuth scope, manager list/revoke endpoints | 1 | +| mit-learn | Grant list and revoke action for organization managers | 1 | +| ol-infrastructure | Per-provider Keycloak client template; bearer-only APISIX route for `/api/v1/learner-records`; new service scope on the ol-analytics-api client | 1 | +| ol-analytics-api | Grant resolution with TTL cache, fail-closed; identity rule; presigned export URLs | 1 | +| mitxonline, mit-learn | Pending grants, manager approval | 2, blocked by C4 | +| ol-analytics-api | OpenAPI changes from §6 (PII rule text, export `uri` semantics) | on sign-off | diff --git a/docs/openapi/b2b-learner-records-v1.yaml b/docs/openapi/b2b-learner-records-v1.yaml new file mode 100644 index 0000000..67b78c4 --- /dev/null +++ b/docs/openapi/b2b-learner-records-v1.yaml @@ -0,0 +1,816 @@ +```yaml +openapi: 3.1.0 + +info: + title: MIT Open Learning — B2B Learner Records API + version: 1.0.0-draft.2 + summary: >- + Machine-to-machine, read-only, organization-scoped learner progress records + for B2B site-license partners and their contracted training providers. + description: | + **DRAFT — proposed contract, not yet implemented.** Published so partners can + review the record shape and size an integration. Field names and types may + change before `1.0.0` final. + + Rationale, scope and open questions: `docs/b2b-learner-records-onepager.md`. + + Three things a reader needs that the schemas do not state: + + * **Consent gates outcomes, not identity.** Progress, completion and + activity fields are populated only where the learner has opted in to + sharing their course status. `outcomes_shared` says which. Identity and + enrollment facts are not consent-gated — the organization assigned the + seat. Enforcement fails closed, so every record reads + `outcomes_shared: false` until the upstream consent field ships. + * **The same schemas are delivered two ways.** This REST API, and a + per-organization bulk export (S3/SFTP) written on each refresh. The export + carries the same records under the same field names. + * **`as_of` is the data's freshness and the sync cursor.** Records come from + materialized views refreshed on a schedule. Pass a previous `as_of` as + `updated_since` for incremental sync; a consent withdrawal arrives as a + normal changed record with `outcomes_shared: false` and outcome fields + nulled, which the client upserts to drop the data it held. + + Every schema property carries `x-data-readiness`: `available` (backed + today), `derived` (computable from existing sources), `pending-model` + (needs upstream work; ships `null` until then). + + contact: + name: MIT Open Learning — Data Engineering + url: https://github.com/mitodl/ol-analytics-api + license: + name: BSD-3-Clause + identifier: BSD-3-Clause + +servers: + - url: https://api.mitxonline.mit.edu/api/v1/learner-records + description: Production (planned) + - url: https://api.qa.mitxonline.mit.edu/api/v1/learner-records + description: QA — partner integration testing (planned) + +security: + - oauth2ClientCredentials: + - learner-records:read + +tags: + - name: learners + - name: enrollments + - name: catalog + - name: exports + +paths: + + /organizations/{organization_id}/learners: + get: + tags: [learners] + operationId: listLearners + summary: Learner roster with progress rollups + description: One row per learner associated with the organization, rolled up across all its contracts. + parameters: + - $ref: '#/components/parameters/OrganizationId' + - $ref: '#/components/parameters/ContractId' + - $ref: '#/components/parameters/LearnerId' + - $ref: '#/components/parameters/UpdatedSince' + - $ref: '#/components/parameters/IncludeInactive' + - $ref: '#/components/parameters/Limit' + - $ref: '#/components/parameters/Offset' + responses: + '200': + description: A page of learner records. + content: + application/json: + schema: + allOf: + - $ref: '#/components/schemas/OrganizationEnvelope' + - type: object + properties: + data: + type: array + items: { $ref: '#/components/schemas/Learner' } + examples: + default: + value: + organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + as_of: '2026-08-13T06:15:00Z' + total_count: 47 + outcomes_withheld_count: 11 + data: + - learner_id: 3e1a9c74-5b2d-4f88-9a01-7c6de2b4f019 + email: rgarcia@contoso.example + full_name: R. Garcia + organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + organization_name: Contoso Manufacturing + membership_source: both + is_organization_manager: false + first_enrolled_on: '2026-02-03T14:22:11Z' + last_enrolled_on: '2026-05-19T09:04:52Z' + courses_enrolled: 4 + outcomes_shared: true + outcomes_consent_on: '2026-02-03T14:20:04Z' + last_active_on: '2026-08-11' + courses_in_progress: 1 + courses_passed: 3 + courses_certified: 3 + certificates_earned: 3 + - learner_id: c04e8a17-3d62-4b95-a7e8-51fb2c8d9042 + email: jlin@contoso.example + full_name: J. Lin + organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + organization_name: Contoso Manufacturing + membership_source: both + is_organization_manager: false + first_enrolled_on: '2026-03-11T10:02:44Z' + last_enrolled_on: '2026-03-11T10:02:44Z' + courses_enrolled: 2 + outcomes_shared: false + outcomes_consent_on: null + last_active_on: null + courses_in_progress: null + courses_passed: null + courses_certified: null + certificates_earned: null + '400': { $ref: '#/components/responses/BadRequest' } + '401': { $ref: '#/components/responses/Unauthorized' } + '403': { $ref: '#/components/responses/Forbidden' } + '429': { $ref: '#/components/responses/TooManyRequests' } + '503': { $ref: '#/components/responses/ServiceUnavailable' } + + /organizations/{organization_id}/enrollments: + get: + tags: [enrollments] + operationId: listEnrollments + summary: Learner-by-course-run enrollment and completion records + description: >- + One row per (learner, course run) under the organization's contracts. + Ordered by `(learner_id, courserun_id)`. + parameters: + - $ref: '#/components/parameters/OrganizationId' + - $ref: '#/components/parameters/ContractId' + - $ref: '#/components/parameters/CourseRunId' + - $ref: '#/components/parameters/LearnerId' + - $ref: '#/components/parameters/CompletionStatusFilter' + - $ref: '#/components/parameters/UpdatedSince' + - $ref: '#/components/parameters/IncludeInactive' + - $ref: '#/components/parameters/Limit' + - $ref: '#/components/parameters/Offset' + responses: + '200': + description: A page of enrollment records. + content: + application/json: + schema: + allOf: + - $ref: '#/components/schemas/OrganizationEnvelope' + - type: object + properties: + data: + type: array + items: { $ref: '#/components/schemas/Enrollment' } + examples: + default: + value: + organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + as_of: '2026-08-13T06:15:00Z' + total_count: 163 + outcomes_withheld_count: 38 + data: + - learner_id: 3e1a9c74-5b2d-4f88-9a01-7c6de2b4f019 + email: rgarcia@contoso.example + full_name: R. Garcia + organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + contract_id: 6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6 + contract_name: Contoso 2026 Site Licence + courserun_id: course-v1:MITxT+14.310x+2T2026 + courserun_title: Data Analysis for Social Scientists + courserun_start_on: '2026-02-01T00:00:00Z' + courserun_end_on: '2026-06-30T23:59:59Z' + enrolled_on: '2026-02-03T14:22:11Z' + enrollment_is_active: true + enrollment_mode: verified + enrollment_status: null + outcomes_shared: true + completion_status: certified + is_passing: true + grade: 0.91 + letter_grade: A + certificate_issued_on: '2026-07-02T11:00:00Z' + certificate_is_revoked: false + last_active_on: '2026-08-11' + days_active: 34 + videos_watched: 212 + problems_attempted: 88 + chatbot_interactions: 14 + - learner_id: c04e8a17-3d62-4b95-a7e8-51fb2c8d9042 + email: jlin@contoso.example + full_name: J. Lin + organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + contract_id: 6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6 + contract_name: Contoso 2026 Site Licence + courserun_id: course-v1:MITxT+14.310x+2T2026 + courserun_title: Data Analysis for Social Scientists + courserun_start_on: '2026-02-01T00:00:00Z' + courserun_end_on: '2026-06-30T23:59:59Z' + enrolled_on: '2026-03-11T10:02:44Z' + enrollment_is_active: true + enrollment_mode: verified + enrollment_status: null + outcomes_shared: false + completion_status: null + is_passing: null + grade: null + letter_grade: null + certificate_issued_on: null + certificate_is_revoked: null + last_active_on: null + days_active: null + videos_watched: null + problems_attempted: null + chatbot_interactions: null + '400': { $ref: '#/components/responses/BadRequest' } + '401': { $ref: '#/components/responses/Unauthorized' } + '403': { $ref: '#/components/responses/Forbidden' } + '429': { $ref: '#/components/responses/TooManyRequests' } + '503': { $ref: '#/components/responses/ServiceUnavailable' } + + /organizations/{organization_id}/courses: + get: + tags: [catalog] + operationId: listCourses + summary: Contracts and course runs covered by the organization's licence + description: Slow-changing and free of personal data. Cache it rather than joining per learner. + parameters: + - $ref: '#/components/parameters/OrganizationId' + - $ref: '#/components/parameters/ContractId' + - $ref: '#/components/parameters/Limit' + - $ref: '#/components/parameters/Offset' + responses: + '200': + description: A page of contract/course-run records. + content: + application/json: + schema: + allOf: + - $ref: '#/components/schemas/OrganizationEnvelope' + - type: object + properties: + data: + type: array + items: { $ref: '#/components/schemas/CourseRun' } + examples: + default: + value: + organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + as_of: '2026-08-13T06:15:00Z' + total_count: 12 + outcomes_withheld_count: 0 + data: + - organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + organization_name: Contoso Manufacturing + contract_id: 6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6 + contract_name: Contoso 2026 Site Licence + contract_is_active: true + contract_start_date: '2026-01-01' + contract_end_date: '2026-12-31' + seat_limit: 250 + courserun_id: course-v1:MITxT+14.310x+2T2026 + courserun_title: Data Analysis for Social Scientists + courserun_start_on: '2026-02-01T00:00:00Z' + courserun_end_on: '2026-06-30T23:59:59Z' + '400': { $ref: '#/components/responses/BadRequest' } + '401': { $ref: '#/components/responses/Unauthorized' } + '403': { $ref: '#/components/responses/Forbidden' } + '429': { $ref: '#/components/responses/TooManyRequests' } + '503': { $ref: '#/components/responses/ServiceUnavailable' } + + /organizations/{organization_id}/exports: + get: + tags: [exports] + operationId: listExports + summary: Bulk export files available for this organization + description: >- + Manifest of the S3/SFTP bulk exports, one set per collection per + refresh. Records inside carry the same schemas as this API. + parameters: + - $ref: '#/components/parameters/OrganizationId' + - $ref: '#/components/parameters/UpdatedSince' + - $ref: '#/components/parameters/Limit' + - $ref: '#/components/parameters/Offset' + responses: + '200': + description: A page of export manifest entries. + content: + application/json: + schema: + allOf: + - $ref: '#/components/schemas/OrganizationEnvelope' + - type: object + properties: + data: + type: array + items: { $ref: '#/components/schemas/ExportFile' } + examples: + default: + value: + organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + as_of: '2026-08-13T06:15:00Z' + total_count: 3 + outcomes_withheld_count: 0 + data: + - collection: enrollments + as_of: '2026-08-13T06:15:00Z' + format: jsonl + record_count: 163 + size_bytes: 214880 + checksum_sha256: 9f2c1b7ae4d05c8831fbb2e6a0d47c3915ee8b6042d1f7c9a3b508e2d6417f0a + uri: s3://ol-b2b-exports/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz + expires_on: '2026-09-12T06:15:00Z' + '400': { $ref: '#/components/responses/BadRequest' } + '401': { $ref: '#/components/responses/Unauthorized' } + '403': { $ref: '#/components/responses/Forbidden' } + '429': { $ref: '#/components/responses/TooManyRequests' } + '503': { $ref: '#/components/responses/ServiceUnavailable' } + +components: + + securitySchemes: + oauth2ClientCredentials: + type: oauth2 + description: >- + Keycloak client credentials, validated at the gateway. Scopes bound to a + client are a contractual limit on that credential; they are not the + consent mechanism. + flows: + clientCredentials: + tokenUrl: https://sso.mit.edu/realms/olapps/protocol/openid-connect/token + scopes: + learner-records:read: Read records; `email` and `full_name` return null. + learner-records:read-pii: Additionally populate `email` and `full_name`. + + parameters: + + OrganizationId: + name: organization_id + in: path + required: true + description: The organization's Keycloak organization UUID. Slugs and names are not accepted. + schema: { type: string, format: uuid } + example: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 + + ContractId: + name: contract_id + in: query + required: false + schema: { type: string } + example: 6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6 + + CourseRunId: + name: courserun_id + in: query + required: false + schema: { type: string } + example: course-v1:MITxT+14.310x+2T2026 + + LearnerId: + name: learner_id + in: query + required: false + description: Repeat for several; maximum 100 per request. + schema: + type: array + maxItems: 100 + items: { type: string, format: uuid } + style: form + explode: true + + CompletionStatusFilter: + name: completion_status + in: query + required: false + description: >- + Repeat for several. Matches only records with shared outcomes; pass + `unknown` to select records whose outcomes are withheld. + schema: + type: array + items: + type: string + enum: [not_started, in_progress, passed, certified, unknown] + style: form + explode: true + + UpdatedSince: + name: updated_since + in: query + required: false + description: Return only records changed at or after this instant. Pass the previous sync's `as_of`. + schema: { type: string, format: date-time } + example: '2026-08-12T06:15:00Z' + + IncludeInactive: + name: include_inactive + in: query + required: false + description: Include deactivated enrollments (unenrolled, refunded, transferred). + schema: { type: boolean, default: false } + + Limit: + name: limit + in: query + required: false + schema: { type: integer, minimum: 1, maximum: 1000, default: 100 } + + Offset: + name: offset + in: query + required: false + schema: { type: integer, minimum: 0, default: 0 } + + schemas: + + OrganizationEnvelope: + type: object + description: Shared response envelope; `data` is typed per endpoint. + required: [organization_id, as_of, total_count, outcomes_withheld_count, data] + properties: + organization_id: + type: string + format: uuid + as_of: + type: [string, 'null'] + format: date-time + description: Last refresh of the backing data. Null before an organization's first refresh. + total_count: + type: integer + minimum: 0 + description: Matching records across all pages, not this page. + outcomes_withheld_count: + type: integer + minimum: 0 + description: >- + Records in `total_count` carrying `outcomes_shared: false`. Always 0 + on `/courses` and `/exports`. + data: + type: array + items: {} + + Learner: + type: object + description: >- + One learner's association with the organization. Fields below + `outcomes_shared` are null unless it is true. + required: + - learner_id + - organization_id + - organization_name + - membership_source + - is_organization_manager + - courses_enrolled + - outcomes_shared + properties: + learner_id: + type: string + format: uuid + description: Stable opaque identifier, consistent across endpoints and stable across email changes. Use as the join key. + x-data-readiness: available + email: + type: [string, 'null'] + format: email + description: Null without `learner-records:read-pii`. Not a join key — may differ from the enrolling address. + x-data-readiness: available + full_name: + type: [string, 'null'] + description: Null without `learner-records:read-pii`, and often null regardless. + x-data-readiness: available + organization_id: + type: string + format: uuid + x-data-readiness: available + organization_name: + type: string + x-data-readiness: available + membership_source: + $ref: '#/components/schemas/MembershipSource' + is_organization_manager: + type: boolean + description: Administers the organization in MITx Online. + x-data-readiness: available + first_enrolled_on: + type: [string, 'null'] + format: date-time + description: Null for a roster member with no enrollments. + x-data-readiness: available + last_enrolled_on: + type: [string, 'null'] + format: date-time + x-data-readiness: available + courses_enrolled: + type: integer + minimum: 0 + description: Distinct course runs under the organization's contracts. Not consent-gated. + x-data-readiness: derived + outcomes_shared: + type: boolean + description: >- + Whether this learner has opted in to sharing their course status. + False means every field below is null. + x-data-readiness: pending-model + outcomes_consent_on: + type: [string, 'null'] + format: date-time + description: When consent was recorded. Null when `outcomes_shared` is false. + x-data-readiness: pending-model + last_active_on: + type: [string, 'null'] + format: date + description: Most recent day with recorded course activity. A date, not a timestamp — activity is aggregated per day. + x-data-readiness: pending-model + courses_in_progress: + type: [integer, 'null'] + minimum: 0 + x-data-readiness: pending-model + courses_passed: + type: [integer, 'null'] + minimum: 0 + x-data-readiness: derived + courses_certified: + type: [integer, 'null'] + minimum: 0 + x-data-readiness: derived + certificates_earned: + type: [integer, 'null'] + minimum: 0 + description: Includes program certificates, which have no course run and so are not in `courses_certified`. + x-data-readiness: derived + + Enrollment: + type: object + description: >- + One learner's enrollment in one course run. Fields below + `outcomes_shared` are null unless it is true. + required: + - learner_id + - organization_id + - contract_id + - courserun_id + - courserun_title + - enrolled_on + - enrollment_is_active + - outcomes_shared + properties: + learner_id: + type: string + format: uuid + x-data-readiness: available + email: + type: [string, 'null'] + format: email + description: Null without `learner-records:read-pii`. + x-data-readiness: available + full_name: + type: [string, 'null'] + description: Null without `learner-records:read-pii`. + x-data-readiness: available + organization_id: + type: string + format: uuid + x-data-readiness: available + contract_id: + type: string + x-data-readiness: available + contract_name: + type: string + x-data-readiness: available + courserun_id: + type: string + description: 'Readable course-run identifier, e.g. `course-v1:MITxT+14.310x+2T2026`.' + x-data-readiness: available + courserun_title: + type: string + description: Mutable display title — key on `courserun_id`. + x-data-readiness: available + courserun_start_on: + type: [string, 'null'] + format: date-time + x-data-readiness: available + courserun_end_on: + type: [string, 'null'] + format: date-time + description: Null for self-paced runs. + x-data-readiness: available + enrolled_on: + type: string + format: date-time + description: This run's enrollment, not an earlier run of the same course. Not consent-gated. + x-data-readiness: available + enrollment_is_active: + type: boolean + description: Not consent-gated. + x-data-readiness: available + enrollment_mode: + type: [string, 'null'] + description: 'e.g. `verified`, `audit`. Determines whether the run is certificate-bearing.' + x-data-readiness: available + enrollment_status: + type: [string, 'null'] + description: Deactivation reason where one was recorded. + x-data-readiness: available + outcomes_shared: + type: boolean + description: Whether the learner has opted in to sharing their course status. False means every field below is null. + x-data-readiness: pending-model + completion_status: + oneOf: + - $ref: '#/components/schemas/CompletionStatus' + - type: 'null' + description: Single derived answer per row. Null when outcomes are withheld. + x-data-readiness: derived + is_passing: + type: [boolean, 'null'] + description: Null where no grade has been computed. + x-data-readiness: available + grade: + type: [number, 'null'] + format: float + minimum: 0 + maximum: 1 + x-data-readiness: available + letter_grade: + type: [string, 'null'] + description: Frequently null — not every platform records one. + x-data-readiness: available + certificate_issued_on: + type: [string, 'null'] + format: date-time + x-data-readiness: available + certificate_is_revoked: + type: [boolean, 'null'] + description: Null where no certificate exists. A revoked certificate leaves `completion_status` at `passed`. + x-data-readiness: available + last_active_on: + type: [string, 'null'] + format: date + description: Most recent day with recorded activity in this course run. + x-data-readiness: pending-model + days_active: + type: [integer, 'null'] + minimum: 0 + description: Distinct days with recorded activity in this course run. + x-data-readiness: pending-model + videos_watched: + type: [integer, 'null'] + minimum: 0 + description: Distinct video blocks played. + x-data-readiness: pending-model + problems_attempted: + type: [integer, 'null'] + minimum: 0 + description: Distinct problem blocks attempted. + x-data-readiness: pending-model + chatbot_interactions: + type: [integer, 'null'] + minimum: 0 + x-data-readiness: pending-model + + CourseRun: + type: object + description: One course run covered by one of the organization's contracts. No personal data. + required: + - organization_id + - organization_name + - contract_id + - contract_name + - contract_is_active + - courserun_id + - courserun_title + properties: + organization_id: { type: string, format: uuid, x-data-readiness: available } + organization_name: { type: string, x-data-readiness: available } + contract_id: { type: string, x-data-readiness: available } + contract_name: { type: string, x-data-readiness: available } + contract_is_active: { type: boolean, x-data-readiness: available } + contract_start_date: + type: [string, 'null'] + format: date + x-data-readiness: available + contract_end_date: + type: [string, 'null'] + format: date + x-data-readiness: available + seat_limit: + type: [integer, 'null'] + minimum: 0 + description: Null means uncapped, not zero. + x-data-readiness: available + courserun_id: { type: string, x-data-readiness: available } + courserun_title: { type: string, x-data-readiness: available } + courserun_start_on: + type: [string, 'null'] + format: date-time + x-data-readiness: available + courserun_end_on: + type: [string, 'null'] + format: date-time + x-data-readiness: available + + ExportFile: + type: object + description: One bulk-export file. Contents use this API's record schemas. + required: [collection, as_of, format, record_count, uri, expires_on] + properties: + collection: + type: string + enum: [learners, enrollments, courses] + as_of: + type: string + format: date-time + description: The refresh this file was written from. + format: + type: string + enum: [jsonl, csv, parquet] + record_count: + type: integer + minimum: 0 + size_bytes: + type: [integer, 'null'] + minimum: 0 + checksum_sha256: + type: [string, 'null'] + pattern: '^[a-f0-9]{64}$' + uri: + type: string + format: uri + description: '`s3://` or `sftp://` location, readable with the credentials issued alongside the API client.' + expires_on: + type: string + format: date-time + description: After this, the file is deleted and must be re-fetched from a newer refresh. + + MembershipSource: + type: string + description: >- + `roster` = on the organization's membership roster with no enrollments + (an assigned, unstarted seat). `enrollment` = enrolled under a contract + but absent from the roster (usually a provisioning lag). `both` = the + expected state. + enum: [roster, enrollment, both] + x-data-readiness: available + + CompletionStatus: + type: string + description: >- + `passed` without `certified` is normal — certificates are issued on a + schedule after grading, and audit-mode enrollments never certify. + enum: [not_started, in_progress, passed, certified] + x-data-readiness: derived + + Error: + type: object + required: [detail] + properties: + detail: { type: string } + + responses: + + BadRequest: + description: Malformed parameter. + content: + application/json: + schema: { $ref: '#/components/schemas/Error' } + example: { detail: 'limit must be between 1 and 1000' } + + Unauthorized: + description: Missing, malformed or expired token. + content: + application/json: + schema: { $ref: '#/components/schemas/Error' } + example: { detail: 'Invalid or expired access token' } + + Forbidden: + description: >- + Token valid but lacks a grant for this organization, or lacks the + required scope. Identical whether the organization is ungranted or does + not exist, so this cannot be used to enumerate organizations. + content: + application/json: + schema: { $ref: '#/components/schemas/Error' } + example: { detail: 'No grant for the requested organization' } + + TooManyRequests: + description: Per-client rate limit exceeded. + headers: + Retry-After: + schema: { type: integer } + content: + application/json: + schema: { $ref: '#/components/schemas/Error' } + example: { detail: 'Rate limit exceeded' } + + ServiceUnavailable: + description: Analytics store unreachable or saturated. + headers: + Retry-After: + schema: { type: integer } + content: + application/json: + schema: { $ref: '#/components/schemas/Error' } + example: { detail: 'Analytics store unavailable' } +``` From 2cd7a2396bbdceca6e55d0599d3330f1af4de3c8 Mon Sep 17 00:00:00 2001 From: Tobias Macey Date: Thu, 10 Sep 2026 14:55:21 -0400 Subject: [PATCH 2/7] docs: record the learner-records access decision, fix the OpenAPI file Access to the M2M API is settled when the contract is signed, not mediated by mitxonline at request time, and the partner handles per-user authorization in its own LMS. That replaces the grant-record proposal: the per-contract Keycloak client is the whole record, with its organizations as a hardcoded claim and identity as the read-pii scope. The OpenAPI file was wrapped in markdown code fences, so it was not parseable YAML as committed. Stripping them makes it pass check-yaml and openapi-spec-validator. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EBTqmr3ki2pAAP1EeWH7wg --- docs/b2b-learner-records-design.md | 3 +- docs/b2b-learner-records-onepager.md | 15 +- ...-learner-records-provider-authorization.md | 278 ++++++------------ docs/openapi/b2b-learner-records-v1.yaml | 9 +- 4 files changed, 100 insertions(+), 205 deletions(-) diff --git a/docs/b2b-learner-records-design.md b/docs/b2b-learner-records-design.md index 563acbc..714d126 100644 --- a/docs/b2b-learner-records-design.md +++ b/docs/b2b-learner-records-design.md @@ -329,5 +329,6 @@ org self-service authorization is more work but puts the data relationship where it belongs. This is an operational gap, not a configuration detail, and it blocks onboarding the first provider rather than the first organization. -Proposed answer, pending sign-off: +Decided: the contract settles access, and MIT issues per-contract client +credentials that encode it. See [`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). diff --git a/docs/b2b-learner-records-onepager.md b/docs/b2b-learner-records-onepager.md index 4869472..216e80a 100644 --- a/docs/b2b-learner-records-onepager.md +++ b/docs/b2b-learner-records-onepager.md @@ -203,13 +203,11 @@ visible record of who can read its data; org self-service authorization is more work but puts the data relationship where it belongs. This is an operational gap, not a configuration detail. -*Proposed answer, pending sign-off:* +*Decided:* [`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). -The organization authorizes, and MIT records it. The provider's credential -(Pulumi) is separate from the organization grant (mitxonline). Phase 1 grants -are staff-recorded on written authorization, visible to the organization, and -revocable by it. Self-service authorization waits for manager designation via -Keycloak Organization Groups. +The contract settles access. MIT issues one Keycloak client per contracted +integration, carrying its organizations as a claim and identity as the +`read-pii` scope. The partner handles per-user authorization in its own LMS. **4. Is consent per-organization or global?** A learner holding seats under two organizations should be able to share with one and not the other. A single @@ -221,5 +219,6 @@ the consent design rather than a question this service can answer. Both are degrading rather than blocking — the service ships without either and fills in as they land. The learner-consent field, without which every record reads `outcomes_shared: false`; and a per-learner activity model, without which -"last active" and the engagement counters are null. A provider-authorization -workflow is the one genuine blocker, and only for onboarding the first provider. +"last active" and the engagement counters are null. Onboarding the first +partner also needs the per-contract Keycloak client template and a bearer-only +gateway route. diff --git a/docs/b2b-learner-records-provider-authorization.md b/docs/b2b-learner-records-provider-authorization.md index 9453453..6893818 100644 --- a/docs/b2b-learner-records-provider-authorization.md +++ b/docs/b2b-learner-records-provider-authorization.md @@ -1,197 +1,93 @@ # B2B Learner Records: who authorizes a training provider Companion to [`b2b-learner-records-design.md`](b2b-learner-records-design.md). -Status: **proposal, pending product-owner sign-off.** Answers open question 3 of -the [one-pager](b2b-learner-records-onepager.md). Questions 1, 2 and 4 stay -open; §6 notes how this changes question 2. - -## 1. The question - -A contracted training provider calls the API with its own client credentials -and reads individually identifying records for an organization's learners. -Something has to say "this provider may read this organization's learners, on -these terms, until this date". The question is who says it and where it is -recorded. - -This is what gates onboarding the first provider. The rest of the service -(tenant, dbt models, consent field) can be built without it. - -## 2. What exists today - -Nothing models the relationship. There is no provider, partner or API-client -record attached to an organization or contract in any system. - -| Piece | Where it lives today | Relevance | -| --- | --- | --- | -| Organization | mitxonline `b2b.models.OrganizationPage`, linked to Keycloak by `sso_organization_id` | The grant's subject. The API's `organization_id` is this UUID. | -| Org manager | mitxonline `b2b.models.UserOrganization.is_manager` | Set only in Django admin (`UserOrganizationAdmin`), row by row, by MIT staff. Never reaches the Keycloak token. | -| Manager designation, future | Keycloak Organization Groups, per mitodl/hq#10594. Labelled C4 in the capability glossary of the provisioning spec (mitodl/mitxonline#3922), from the onboarding RFC mitodl/hq#12784 | Gives the organization a manager role it controls, with an audit trail. Not built. | -| Service-account clients | Pulumi, `ol-infrastructure` `substructure/keycloak/olapps.py` (e.g. `mitxonline-b2b-client`), scopes via `ClientDefaultScopes` | The pattern for issuing a provider's credential. | -| Ownership split | mitxonline PR #3922, `docs/source/b2b/provisioning_api.md` | Pulumi keeps realm, flows, client scopes, **clients** and service-account grants. The mitxonline provisioning API owns everything per customer. | -| Token validation | APISIX in front of ol-analytics-api; the app decodes `X-Userinfo` (`core/auth/userinfo.py`) | No inbound client-credentials principal exists yet. | -| Service-to-service check | `b2b_dashboard` asks mitxonline whether a user is a manager with its own client-credentials token, scope `b2b:manager-check` (`tenants/b2b_dashboard/mitxonline_client.py`, mitxonline `b2b/views/v0/service.py`) | The pattern for checking a grant at request time. | -| Manager UI | mit-learn contract admin page, `app-pages/ContractAdminPage`, which already has a revoke/confirm flow for seat codes | Where an organization would see and revoke grants. | - -## 3. Separate the credential from the grant - -The two shapes in the original question ("MIT issues credentials by ticket" vs -"the org authorizes from its dashboard") conflate two different objects: - -- **The credential** identifies the provider. One Keycloak client per provider, - whatever the number of organizations it works for. It carries no - organization. Issuing it is rare, provider-level, and fits the onboarding - split: clients are substrate, so they stay in Pulumi under review. -- **The grant** is the relationship: provider X may read organization Y's - learners, with or without identity, until a date. It is per customer, it - changes whenever a contract starts or ends, and it is the thing the - organization needs to see. Under the onboarding split it belongs in - mitxonline, next to `OrganizationPage` and `ContractPage`. - -A credential without a grant reads nothing: every request returns the same 403 -as an unknown organization. So issuing credentials can stay an MIT operations -step without MIT deciding who reads whose data. - -## 4. Recommendation - -**The organization authorizes. Never the provider, and never MIT alone.** MIT -records and enforces the organization's decision. - -### Phase 1: staff-recorded grants, org-visible, org-revocable - -Enough to onboard the first provider. - -1. The provider gets a Keycloak client from a reviewed Pulumi change. It holds - no grants. -2. The organization authorizes the provider in writing. MIT confirms with the - organization's contact of record, not a contact the provider supplies. That - closes the obvious social-engineering route of a provider asking MIT for - access on the organization's behalf. -3. MIT staff record the grant in mitxonline (Django admin), with the - authorizing person and a reference to the written authorization. -4. The organization's managers see active and past grants on the mit-learn - contract admin page, and can **revoke** one there. - -Self-service revocation ships in phase 1. Self-service authorization waits for -phase 2, and the asymmetry is deliberate. Revoking can only reduce disclosure, -so the current manager flag (set by MIT staff in Django admin) is authority -enough to act on it. Authorizing increases disclosure of individually -identifying data. That should rest on a manager role the organization controls -and that leaves an audit trail, which is what C4 provides. - -The visible grant list is what separates this from "MIT-issued-by-ticket". The -objection to the ticket-only shape was that the organization has no record of -who can read its learners' data. Phase 1 fixes that without building a -self-service authorization flow. - -### Phase 2: organization self-service authorization - -Blocked by C4. The provider (or MIT staff on its behalf) creates a *pending* -grant naming the organization, and an organization manager approves it in -mit-learn. The provider can request but never activate. MIT staff keep the -ability to revoke, but no longer record authorizations. - -## 5. The grant record - -A mitxonline model in the `b2b` app. Working name `OrganizationDataGrant`. - -| Field | Note | +Status: **decided 2026-09-10.** Answers open question 3 of the +[one-pager](b2b-learner-records-onepager.md). + +## Decision + +Access is settled when the contract is signed. MIT does not decide it or check +it against another system at request time. + +The contract with the organization says who may read its learners' records +(the organization itself, or a training provider it has contracted), and +whether names and email addresses are included. MIT then issues client +credentials that encode those terms. The partner builds the integration into +its own LMS and handles per-user authorization on its side. MIT does not model +the partner's users, and a credential reads everything its contract covers. + +## How the terms are encoded + +One Keycloak client per contracted integration, defined in Pulumi +(`ol-infrastructure`, `substructure/keycloak`) alongside the existing +service-account clients such as `mitxonline-b2b-client`. The reviewed change +that creates the client is MIT's record of the access. + +| Contract term | On the client | | --- | --- | -| `organization` | FK `OrganizationPage`. | -| `client_id` | The provider's Keycloak client ID. | -| `provider_name` | Display name for the manager UI. | -| `includes_identity` | Whether records carry `email` and `full_name`. See §6. | -| `status` | `pending`, `active`, `revoked`. | -| `expires_on` | Required. Not later than the end of the organization's latest active contract. | -| `authorized_by_name`, `authorized_by_email` | The organization-side person who authorized it. | -| `authorization_reference` | Ticket or document reference for the written authorization (phase 1). | -| `recorded_by`, `activated_on` | MIT staff user (phase 1) or the approving manager (phase 2). | -| `revoked_by`, `revoked_on`, `revocation_reason` | | - -Two rules, the same ones the design already imposes on consent: - -- **Rows are never deleted.** Revocation is a retained state change. A deleted - grant cannot answer "who could read our data in March", which is the first - question an organization asks after a dispute with a provider. -- **Expiry is computed at check time,** so a grant is active only if - `status = active` and `expires_on` is in the future. A contract ending needs - no job to revoke access. - -## 6. How the API checks a grant - -Per request, in `tenants/b2b_learner_records/auth.py`: - -1. Read the client ID from the validated token. -2. Resolve the client's active grants from mitxonline through a new service - endpoint. Use the same pattern as `b2b:manager-check`, under a new scope - (e.g. `b2b:data-grant-check`). Cache per client for a short TTL (proposed - 60 s). -3. If there is no active grant for the path's `organization_id`, return the - existing 403, identical to the one for an organization that does not exist. -4. If mitxonline is unreachable and the cache is expired, return 503. Never - serve from a stale grant list, because that would keep a revoked grant - readable. - -Revocation takes effect within one TTL. - -### Identity moves onto the grant - -The draft contract gates `email`/`full_name` on the client scope -(`learner-records:read` vs `learner-records:read-pii`). A scope belongs to the -client, and a provider working for two organizations has one client, so a -scope cannot express "identity for A, not for B". Proposed rule: identity is -returned only when **both** the client holds `learner-records:read-pii` **and** -the grant has `includes_identity`. The scope becomes MIT's ceiling for that -provider, and the grant flag is the organization's choice within it. - -This reframes open question 2. Instead of asking whether a provider's -credential gets PII, it asks whether MIT ever sets the PII ceiling for -providers, and what `includes_identity` defaults to. It still needs a policy -owner. - -### Exports need the same gate - -The draft says export files are "readable with the credentials issued -alongside the API client". Those would be standing S3/SFTP credentials outside -the grant check, so revoking a grant would leave the export readable. Proposed: -`/exports` returns short-lived presigned HTTPS URLs, minted per request after -the grant check. No standing storage credentials in v1. A push to a -partner-owned SFTP endpoint can come later, keyed off the same grant. - -## 7. Rejected alternatives - -- **Grant as Keycloak client configuration** (a hard-coded claim listing - organization IDs, or per-organization client scopes). This puts a - per-customer resource on a Pulumi-owned client, contrary to the onboarding - split. It makes every grant change an infrastructure deploy, and the - organization still sees nothing. -- **Service account as a member of the Keycloak organization,** so the - existing organization-membership mapper lists the organization in the token. - It reuses machinery, but mitxonline's org sync would pull the service account - in as a learner. It would then appear in `bridge_user_organization`, the - roster this API serves. -- **Grants landed in the warehouse and joined like consent.** Revocation would - lag the materialized-view refresh (manual, Dagster-driven), and prompt - revocation is the one property a grant must have. -- **Provider self-service.** The provider is the party whose access is in - question. - -## 8. To verify before building +| Which organizations | A hardcoded claim (working name `learner_records_organizations`) listing the Keycloak organization UUIDs, via `keycloak.openid.HardcodedClaimProtocolMapper` | +| Identity included or not | Default client scopes: `learner-records:read` always; `learner-records:read-pii` only if the contract includes identity | +| API audience | `keycloak.openid.AudienceProtocolMapper`, as the Superset client already uses | +| Client-credentials only | `service_accounts_enabled=True`, standard flow and direct grants off | + +A provider working for two organizations under two contracts holds two clients. +When a contract ends, removing its client revokes exactly that access, which +keeps the property the design already required. + +Recommended (it costs one more claim): carry the contract end date as a claim +too, and have the API refuse tokens once that date has passed. A forgotten +cleanup PR then fails closed. + +## What the API does + +In `tenants/b2b_learner_records/auth.py`, per request: + +1. Read the organization claim and scopes from the validated token. +2. If the path's `organization_id` is not in the claim, return the existing + 403, identical to the response for an organization that does not exist. +3. Populate `email` and `full_name` only when the token holds + `learner-records:read-pii`. + +No call to mitxonline or any other service, and no grant store. The client +definition is the only place access is recorded. + +## Consequences + +- **Open question 2 now has an owner.** Whether a provider receives learner + identity is decided per contract and expressed as the `read-pii` scope. The + draft OpenAPI contract's scope split already supports this unchanged. +- **Revocation is a Pulumi change.** Tokens issued before the client is removed + stay valid until they expire. The access-token lifespan for these clients + bounds that window. It has not been checked for the `olapps` realm yet. +- **Exports need to be covered by the same credential.** The draft says export + files are "readable with the credentials issued alongside the API client". + Standing storage credentials would be a second thing to revoke when a + contract ends. Recommended: `/exports` returns short-lived presigned HTTPS + URLs, so the API client is the only credential. Not yet applied to the + OpenAPI draft. +- **The partner owns end-user access.** Obligations on how the partner + restricts learner records inside its LMS belong in the contract. Nothing on + MIT's side enforces them. + +## Not chosen + +- **A grant record in mitxonline, checked at request time,** with + organization managers able to see and revoke grants. That adds a runtime + dependency and a UI, but the decision it would store is already made in the + contract. +- **Making the service account a member of the Keycloak organization,** so + the existing organization-membership mapper lists it. mitxonline's org sync + would import the service account as a learner, and it would then appear in + the roster this API serves. + +## To verify before building - Which claim carries the client ID in a Keycloak 26.7 client-credentials - token (`azp`, `client_id`, or both), and whether APISIX's `openid-connect` - plugin populates `X-Userinfo` with it for a bearer-only route. The - learner-records mount needs a bearer-only route; today's routes use the - redirect flow. Check on QA before writing `auth.py`. -- Whether `ContractAdminPage` is the right home for the grant list, or whether - it belongs at organization level. A grant spans contracts. - -## 9. Work this creates - -| Repo | Work | Phase | -| --- | --- | --- | -| mitxonline | `OrganizationDataGrant` model, Django admin, service endpoint + OAuth scope, manager list/revoke endpoints | 1 | -| mit-learn | Grant list and revoke action for organization managers | 1 | -| ol-infrastructure | Per-provider Keycloak client template; bearer-only APISIX route for `/api/v1/learner-records`; new service scope on the ol-analytics-api client | 1 | -| ol-analytics-api | Grant resolution with TTL cache, fail-closed; identity rule; presigned export URLs | 1 | -| mitxonline, mit-learn | Pending grants, manager approval | 2, blocked by C4 | -| ol-analytics-api | OpenAPI changes from §6 (PII rule text, export `uri` semantics) | on sign-off | + token (`azp`, `client_id`, or both). It's needed for audit logging and rate + limits. +- Whether APISIX's `openid-connect` plugin passes the hardcoded claim and + scopes through in `X-Userinfo` on a bearer-only route. The learner-records + mount needs a bearer-only route, but today's routes use the redirect flow. + Check on QA before writing `auth.py`. +- The access-token lifespan these clients will get, since it is the + revocation window. diff --git a/docs/openapi/b2b-learner-records-v1.yaml b/docs/openapi/b2b-learner-records-v1.yaml index 67b78c4..67ca99f 100644 --- a/docs/openapi/b2b-learner-records-v1.yaml +++ b/docs/openapi/b2b-learner-records-v1.yaml @@ -1,4 +1,3 @@ -```yaml openapi: 3.1.0 info: @@ -336,9 +335,10 @@ components: oauth2ClientCredentials: type: oauth2 description: >- - Keycloak client credentials, validated at the gateway. Scopes bound to a - client are a contractual limit on that credential; they are not the - consent mechanism. + Keycloak client credentials, validated at the gateway. One client is + issued per contract and lists the organizations it may read. Scopes + bound to a client are a contractual limit on that credential; they are + not the consent mechanism. flows: clientCredentials: tokenUrl: https://sso.mit.edu/realms/olapps/protocol/openid-connect/token @@ -813,4 +813,3 @@ components: application/json: schema: { $ref: '#/components/schemas/Error' } example: { detail: 'Analytics store unavailable' } -``` From dd6606ac3094b958efb3945b248a3dc05ad16015 Mon Sep 17 00:00:00 2001 From: Tobias Macey Date: Thu, 10 Sep 2026 15:49:41 -0400 Subject: [PATCH 3/7] docs(learner-records): type contract_id as an integer, fix stale design facts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit contract_id is mitxonline's ContractPage id, exposed by dim_contract as b2b_contract_id: an integer. The spec typed it as a string with a UUID example, the same mismatch #39 fixed in b2b_dashboard after production rejected input_value=695. A partner generating a client from this contract would have expected UUID strings. Also corrects three stale statements in the design doc: §4 still titled the provider-authorization question "Open" above its decision, the b2b_analytics MV count is eight rather than six, and the roster's manager column is userorganization_is_manager. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EBTqmr3ki2pAAP1EeWH7wg --- docs/b2b-learner-records-design.md | 18 +++++++----------- docs/b2b-learner-records-onepager.md | 4 ++-- docs/openapi/b2b-learner-records-v1.yaml | 14 +++++++------- 3 files changed, 16 insertions(+), 20 deletions(-) diff --git a/docs/b2b-learner-records-design.md b/docs/b2b-learner-records-design.md index 714d126..9b1a48a 100644 --- a/docs/b2b-learner-records-design.md +++ b/docs/b2b-learner-records-design.md @@ -22,7 +22,7 @@ names explicitly — is the gap.** | --- | --- | --- | | Stable learner id | `dim_user.user_global_id` | Keycloak `sub`; already the cross-system identifier. Survives email changes. | | Email, full name | `dim_user.email`, `.full_name` | `email` is a *coalesce* across platform accounts — the most recently active address, not necessarily the one the org enrolled them under. Not a safe join key. | -| Org membership roster | `bridge_user_organization` | `(user_fk, organization_fk, is_manager)`. Includes members with zero enrollments. | +| Org membership roster | `bridge_user_organization` | `(user_fk, organization_fk, userorganization_is_manager)`. Includes members with zero enrollments. | | Org identity | `dim_organization` | `sso_organization_id` is the Keycloak org UUID the existing tenant already filters on. | | Contract | `dim_contract` | Name, term, `b2b_contract_max_learners` (seat limit), membership type. | | Contract → course run | `bridge_organization_courserun` | Grain `(org, contract, courserun)`. | @@ -32,7 +32,7 @@ names explicitly — is the gap.** ### The gaps -**1. No learner-grain view exists.** All six `b2b_analytics` MVs are +**1. No learner-grain view exists.** All eight `b2b_analytics` MVs are pre-aggregated by design. A learner roster needs a new dbt model — call it `mv_b2b_learner_enrollment` at `(org × contract × courserun × learner)` — plus a learner-grain rollup. This is the bulk of the upstream work, but it is @@ -321,14 +321,10 @@ Identity is available to the *organization*. A provider is a different principal, and the API supports either answer today via the `read` / `read-pii` scope split (§3). It needs a policy owner rather than a default. -### Open: who authorizes a provider, through what workflow? +### Settled: who authorizes a provider, through what workflow? -No provisioning design exists. MIT-issued-by-ticket is simplest and leaves the -organization with no visible record of who can read its learners' data; -org self-service authorization is more work but puts the data relationship where -it belongs. This is an operational gap, not a configuration detail, and it -blocks onboarding the first provider rather than the first organization. - -Decided: the contract settles access, and MIT issues per-contract client -credentials that encode it. See +The contract settles access. MIT issues one Keycloak client per contracted +integration, with the organizations it may read carried as a claim and +identity carried as the `read-pii` scope. The partner handles per-user +authorization in its own LMS. See [`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). diff --git a/docs/b2b-learner-records-onepager.md b/docs/b2b-learner-records-onepager.md index 216e80a..27b766b 100644 --- a/docs/b2b-learner-records-onepager.md +++ b/docs/b2b-learner-records-onepager.md @@ -115,7 +115,7 @@ Every collection returns the same envelope, with `data` typed to its record: "email": "rgarcia@contoso.example", "full_name": "R. Garcia", "organization_id": "8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21", - "contract_id": "6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6", + "contract_id": 42, "contract_name": "Contoso 2026 Site Licence", "courserun_id": "course-v1:MITxT+14.310x+2T2026", "courserun_title": "Data Analysis for Social Scientists", @@ -146,7 +146,7 @@ Every collection returns the same envelope, with `data` typed to its record: { "organization_id": "8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21", "organization_name": "Contoso Manufacturing", - "contract_id": "6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6", + "contract_id": 42, "contract_name": "Contoso 2026 Site Licence", "contract_is_active": true, "contract_start_date": "2026-01-01", diff --git a/docs/openapi/b2b-learner-records-v1.yaml b/docs/openapi/b2b-learner-records-v1.yaml index 67ca99f..d9bd25a 100644 --- a/docs/openapi/b2b-learner-records-v1.yaml +++ b/docs/openapi/b2b-learner-records-v1.yaml @@ -177,7 +177,7 @@ paths: email: rgarcia@contoso.example full_name: R. Garcia organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 - contract_id: 6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6 + contract_id: 42 contract_name: Contoso 2026 Site Licence courserun_id: course-v1:MITxT+14.310x+2T2026 courserun_title: Data Analysis for Social Scientists @@ -203,7 +203,7 @@ paths: email: jlin@contoso.example full_name: J. Lin organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 - contract_id: 6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6 + contract_id: 42 contract_name: Contoso 2026 Site Licence courserun_id: course-v1:MITxT+14.310x+2T2026 courserun_title: Data Analysis for Social Scientists @@ -265,7 +265,7 @@ paths: data: - organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 organization_name: Contoso Manufacturing - contract_id: 6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6 + contract_id: 42 contract_name: Contoso 2026 Site Licence contract_is_active: true contract_start_date: '2026-01-01' @@ -360,8 +360,8 @@ components: name: contract_id in: query required: false - schema: { type: string } - example: 6a2f0b9d-77c4-4e1a-bb52-08f3ad91c7e6 + schema: { type: integer } + example: 42 CourseRunId: name: courserun_id @@ -574,7 +574,7 @@ components: format: uuid x-data-readiness: available contract_id: - type: string + type: integer x-data-readiness: available contract_name: type: string @@ -684,7 +684,7 @@ components: properties: organization_id: { type: string, format: uuid, x-data-readiness: available } organization_name: { type: string, x-data-readiness: available } - contract_id: { type: string, x-data-readiness: available } + contract_id: { type: integer, x-data-readiness: available } contract_name: { type: string, x-data-readiness: available } contract_is_active: { type: boolean, x-data-readiness: available } contract_start_date: From aeb40403466cca093b0e7ec6c74ef649632583f5 Mon Sep 17 00:00:00 2001 From: Tobias Macey Date: Thu, 10 Sep 2026 15:54:08 -0400 Subject: [PATCH 4/7] docs(learner-records): declare contract_id as int64 dim_contract.contract_id and the mitxonline source column b2b_contractpage.page_ptr_id are both BIGINT in the production warehouse, so the contract states the width a partner's client should use rather than leaving it to generator defaults. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EBTqmr3ki2pAAP1EeWH7wg --- docs/openapi/b2b-learner-records-v1.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/openapi/b2b-learner-records-v1.yaml b/docs/openapi/b2b-learner-records-v1.yaml index d9bd25a..da6b63d 100644 --- a/docs/openapi/b2b-learner-records-v1.yaml +++ b/docs/openapi/b2b-learner-records-v1.yaml @@ -360,7 +360,7 @@ components: name: contract_id in: query required: false - schema: { type: integer } + schema: { type: integer, format: int64 } example: 42 CourseRunId: @@ -575,6 +575,7 @@ components: x-data-readiness: available contract_id: type: integer + format: int64 x-data-readiness: available contract_name: type: string @@ -684,7 +685,7 @@ components: properties: organization_id: { type: string, format: uuid, x-data-readiness: available } organization_name: { type: string, x-data-readiness: available } - contract_id: { type: integer, x-data-readiness: available } + contract_id: { type: integer, format: int64, x-data-readiness: available } contract_name: { type: string, x-data-readiness: available } contract_is_active: { type: boolean, x-data-readiness: available } contract_start_date: From 61f3d7f9de655f2227c5bc6921e9a973d9290618 Mon Sep 17 00:00:00 2001 From: Tobias Macey Date: Thu, 10 Sep 2026 16:39:14 -0400 Subject: [PATCH 5/7] docs(learner-records): address Copilot review of the spec Drops the read/read-pii scope split. The organization already holds its learners' identity because they are its employees or students, so redacting it protects nothing. A single per-organization export could not have honoured the split anyway. Records and exports always carry identity, and one scope remains. Pins down contract semantics the review found ambiguous: - Enrollment grain includes the contract, matching the backing model. No org/run pair in production links to more than one contract today (0 of 2468), so this adds no duplicates. - Every list endpoint has a unique order, so offset paging is stable. - /learners?contract_id= recomputes rollups from that contract's enrollments only. - Fixed-shape nullable fields are required, so omission is not a second way to say null. - The x-data-readiness claim is limited to the record schemas, and the security scheme says "per contracted integration", matching the decision doc. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EBTqmr3ki2pAAP1EeWH7wg --- docs/b2b-learner-records-design.md | 38 ++++----- docs/b2b-learner-records-onepager.md | 13 ++- ...-learner-records-provider-authorization.md | 19 ++--- docs/openapi/b2b-learner-records-v1.yaml | 79 ++++++++++++++----- 4 files changed, 93 insertions(+), 56 deletions(-) diff --git a/docs/b2b-learner-records-design.md b/docs/b2b-learner-records-design.md index 9b1a48a..1f5436b 100644 --- a/docs/b2b-learner-records-design.md +++ b/docs/b2b-learner-records-design.md @@ -179,8 +179,9 @@ Three collections, all org-scoped, all read-only: - `GET /organizations/{organization_id}/learners` — learner grain, roster plus progress rollup. Answers "who is on our licence and how are they doing". -- `GET /organizations/{organization_id}/enrollments` — `(learner × course run)` - grain. Answers "did this learner complete this course". The primary endpoint; +- `GET /organizations/{organization_id}/enrollments` — `(learner × contract × + course run)` grain, matching `mv_b2b_learner_enrollment` above. Answers "did + this learner complete this course". The primary endpoint; `/learners` is a convenience rollup over the same records. - `GET /organizations/{organization_id}/courses` — the contracts and course runs the identifiers refer to. Small, slow-changing, no personal data, cacheable. @@ -212,22 +213,18 @@ rendering a dashboard should use the API. add a router over the same StarRocks views under its own gate. Keeping the concerns separate now is cheaper than un-merging them later. -### Identity is a contractual limit, not a consent one +### Identity is not redacted -`learner-records:read` returns records with `email` and `full_name` as `null`; -`learner-records:read-pii` populates them. Both scopes return the same progress -data. +Every record carries `email` and `full_name`, and there is one scope, +`learner-records:read`. -This split is **not** the consent mechanism — consent governs outcomes (§4), and -the organization already holds its learners' names and addresses because it -assigned the seats. The split exists because a contracted training provider is a -different principal from the organization: "the org may see it" does not settle -"the provider may see it." Since `learner_id` is stable, a provider that enrolled -the learners can join on an identifier it already supplied and never receive -contact details from MIT at all. +Redacting identity would protect nothing. The organization already holds its +learners' names and addresses: they are its employees or students, and it +assigned the seats. A contracted provider reads on the organization's behalf +under the contract and handles per-user access in its own LMS. Consent governs +outcomes (§4), not identity. -Whether any given provider gets the PII scope is a contractual call, and one -nobody has made yet — see §4's open questions. +One read scope also means one bulk export per organization serves every client. ## 4. Learner consent @@ -315,16 +312,13 @@ The available directions are: Worth putting to whoever owns the consent language early, since the third option changes their work rather than ours. -### Open: does a contracted provider receive identity at all? +### Settled: a contracted provider receives identity -Identity is available to the *organization*. A provider is a different -principal, and the API supports either answer today via the `read` / -`read-pii` scope split (§3). It needs a policy owner rather than a default. +Yes. Identity is not redacted for any client (§3). ### Settled: who authorizes a provider, through what workflow? The contract settles access. MIT issues one Keycloak client per contracted -integration, with the organizations it may read carried as a claim and -identity carried as the `read-pii` scope. The partner handles per-user -authorization in its own LMS. See +integration, with the organizations it may read carried as a claim. The +partner handles per-user authorization in its own LMS. See [`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). diff --git a/docs/b2b-learner-records-onepager.md b/docs/b2b-learner-records-onepager.md index 27b766b..6057166 100644 --- a/docs/b2b-learner-records-onepager.md +++ b/docs/b2b-learner-records-onepager.md @@ -191,11 +191,10 @@ action on non-participation), or design-side (report participation only in aggregate above a threshold, and accept degraded seat reporting). Unresolved, and the sharpest issue in this design. -**2. Does a contracted provider receive learner identity at all?** Identity is -available to the *organization*. A provider is a different principal, and -"the org may see it" does not settle "the provider may see it." Cheap to support -either way — credentials can be scoped to pseudonymous records — but it needs a -policy owner, not a default. +**2. Does a contracted provider receive learner identity at all?** *Decided: +yes.* The organization already holds its learners' identity (they are its +employees or students), so redacting it gains nothing. There is one read scope, +and records and exports always carry identity. **3. Who authorizes a provider, through what workflow?** No provisioning design exists. MIT-issued-by-ticket is simplest and leaves the organization with no @@ -206,8 +205,8 @@ not a configuration detail. *Decided:* [`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). The contract settles access. MIT issues one Keycloak client per contracted -integration, carrying its organizations as a claim and identity as the -`read-pii` scope. The partner handles per-user authorization in its own LMS. +integration, carrying its organizations as a claim. The partner handles +per-user authorization in its own LMS. **4. Is consent per-organization or global?** A learner holding seats under two organizations should be able to share with one and not the other. A single diff --git a/docs/b2b-learner-records-provider-authorization.md b/docs/b2b-learner-records-provider-authorization.md index 6893818..4de1a4a 100644 --- a/docs/b2b-learner-records-provider-authorization.md +++ b/docs/b2b-learner-records-provider-authorization.md @@ -10,9 +10,8 @@ Access is settled when the contract is signed. MIT does not decide it or check it against another system at request time. The contract with the organization says who may read its learners' records -(the organization itself, or a training provider it has contracted), and -whether names and email addresses are included. MIT then issues client -credentials that encode those terms. The partner builds the integration into +(the organization itself, or a training provider it has contracted). MIT then +issues client credentials that encode those terms. The partner builds the integration into its own LMS and handles per-user authorization on its side. MIT does not model the partner's users, and a credential reads everything its contract covers. @@ -26,7 +25,7 @@ that creates the client is MIT's record of the access. | Contract term | On the client | | --- | --- | | Which organizations | A hardcoded claim (working name `learner_records_organizations`) listing the Keycloak organization UUIDs, via `keycloak.openid.HardcodedClaimProtocolMapper` | -| Identity included or not | Default client scopes: `learner-records:read` always; `learner-records:read-pii` only if the contract includes identity | +| Read access | Default client scope `learner-records:read`. There is no separate identity scope; see Consequences | | API audience | `keycloak.openid.AudienceProtocolMapper`, as the Superset client already uses | | Client-credentials only | `service_accounts_enabled=True`, standard flow and direct grants off | @@ -45,17 +44,19 @@ In `tenants/b2b_learner_records/auth.py`, per request: 1. Read the organization claim and scopes from the validated token. 2. If the path's `organization_id` is not in the claim, return the existing 403, identical to the response for an organization that does not exist. -3. Populate `email` and `full_name` only when the token holds - `learner-records:read-pii`. +3. Require the `learner-records:read` scope. Identity fields are always + populated. No call to mitxonline or any other service, and no grant store. The client definition is the only place access is recorded. ## Consequences -- **Open question 2 now has an owner.** Whether a provider receives learner - identity is decided per contract and expressed as the `read-pii` scope. The - draft OpenAPI contract's scope split already supports this unchanged. +- **No identity split.** The organization already holds its learners' names + and addresses (they are its employees or students), so redacting them gains + nothing. The draft's `read-pii` scope is dropped. Records and exports always + carry identity, and one export set per organization serves every client. This + settles open question 2. - **Revocation is a Pulumi change.** Tokens issued before the client is removed stay valid until they expire. The access-token lifespan for these clients bounds that window. It has not been checked for the `olapps` realm yet. diff --git a/docs/openapi/b2b-learner-records-v1.yaml b/docs/openapi/b2b-learner-records-v1.yaml index da6b63d..dce8b7f 100644 --- a/docs/openapi/b2b-learner-records-v1.yaml +++ b/docs/openapi/b2b-learner-records-v1.yaml @@ -30,9 +30,10 @@ info: normal changed record with `outcomes_shared: false` and outcome fields nulled, which the client upserts to drop the data it held. - Every schema property carries `x-data-readiness`: `available` (backed - today), `derived` (computable from existing sources), `pending-model` - (needs upstream work; ships `null` until then). + Every property of the `Learner`, `Enrollment` and `CourseRun` record + schemas carries `x-data-readiness`: `available` (backed today), + `derived` (computable from existing sources), `pending-model` (needs + upstream work; ships `null` until then). contact: name: MIT Open Learning — Data Engineering @@ -64,7 +65,11 @@ paths: tags: [learners] operationId: listLearners summary: Learner roster with progress rollups - description: One row per learner associated with the organization, rolled up across all its contracts. + description: >- + One row per learner associated with the organization, rolled up across + all its contracts. With `contract_id`, returns learners holding that + contract and recomputes every rollup from that contract's enrollments + only. Ordered by `learner_id`, which is unique. parameters: - $ref: '#/components/parameters/OrganizationId' - $ref: '#/components/parameters/ContractId' @@ -140,8 +145,9 @@ paths: operationId: listEnrollments summary: Learner-by-course-run enrollment and completion records description: >- - One row per (learner, course run) under the organization's contracts. - Ordered by `(learner_id, courserun_id)`. + One row per (learner, contract, course run) under the organization's + contracts. Ordered by `(learner_id, contract_id, courserun_id)`, which + is unique, so offset paging is stable. parameters: - $ref: '#/components/parameters/OrganizationId' - $ref: '#/components/parameters/ContractId' @@ -236,7 +242,9 @@ paths: tags: [catalog] operationId: listCourses summary: Contracts and course runs covered by the organization's licence - description: Slow-changing and free of personal data. Cache it rather than joining per learner. + description: >- + Slow-changing and free of personal data. Cache it rather than joining + per learner. Ordered by `(contract_id, courserun_id)`, which is unique. parameters: - $ref: '#/components/parameters/OrganizationId' - $ref: '#/components/parameters/ContractId' @@ -288,7 +296,9 @@ paths: summary: Bulk export files available for this organization description: >- Manifest of the S3/SFTP bulk exports, one set per collection per - refresh. Records inside carry the same schemas as this API. + refresh. Records inside carry the same schemas as this API, identity + fields included. Ordered by `as_of` descending, then `collection` and + `format`, which is unique. parameters: - $ref: '#/components/parameters/OrganizationId' - $ref: '#/components/parameters/UpdatedSince' @@ -336,15 +346,15 @@ components: type: oauth2 description: >- Keycloak client credentials, validated at the gateway. One client is - issued per contract and lists the organizations it may read. Scopes + issued per contracted integration and lists the organizations it may + read. Scopes bound to a client are a contractual limit on that credential; they are not the consent mechanism. flows: clientCredentials: tokenUrl: https://sso.mit.edu/realms/olapps/protocol/openid-connect/token scopes: - learner-records:read: Read records; `email` and `full_name` return null. - learner-records:read-pii: Additionally populate `email` and `full_name`. + learner-records:read: Read records, identity fields included. parameters: @@ -459,12 +469,22 @@ components: `outcomes_shared` are null unless it is true. required: - learner_id + - email + - full_name - organization_id - organization_name - membership_source - is_organization_manager + - first_enrolled_on + - last_enrolled_on - courses_enrolled - outcomes_shared + - outcomes_consent_on + - last_active_on + - courses_in_progress + - courses_passed + - courses_certified + - certificates_earned properties: learner_id: type: string @@ -474,11 +494,11 @@ components: email: type: [string, 'null'] format: email - description: Null without `learner-records:read-pii`. Not a join key — may differ from the enrolling address. + description: Not a join key; may differ from the enrolling address. x-data-readiness: available full_name: type: [string, 'null'] - description: Null without `learner-records:read-pii`, and often null regardless. + description: Often null. x-data-readiness: available organization_id: type: string @@ -544,17 +564,36 @@ components: Enrollment: type: object description: >- - One learner's enrollment in one course run. Fields below - `outcomes_shared` are null unless it is true. + One learner's enrollment in one course run, attributed to one of the + organization's contracts. Fields below `outcomes_shared` are null + unless it is true. required: - learner_id + - email + - full_name - organization_id - contract_id + - contract_name - courserun_id - courserun_title + - courserun_start_on + - courserun_end_on - enrolled_on - enrollment_is_active + - enrollment_mode + - enrollment_status - outcomes_shared + - completion_status + - is_passing + - grade + - letter_grade + - certificate_issued_on + - certificate_is_revoked + - last_active_on + - days_active + - videos_watched + - problems_attempted + - chatbot_interactions properties: learner_id: type: string @@ -563,11 +602,10 @@ components: email: type: [string, 'null'] format: email - description: Null without `learner-records:read-pii`. x-data-readiness: available full_name: type: [string, 'null'] - description: Null without `learner-records:read-pii`. + description: Often null. x-data-readiness: available organization_id: type: string @@ -680,8 +718,13 @@ components: - contract_id - contract_name - contract_is_active + - contract_start_date + - contract_end_date + - seat_limit - courserun_id - courserun_title + - courserun_start_on + - courserun_end_on properties: organization_id: { type: string, format: uuid, x-data-readiness: available } organization_name: { type: string, x-data-readiness: available } @@ -715,7 +758,7 @@ components: ExportFile: type: object description: One bulk-export file. Contents use this API's record schemas. - required: [collection, as_of, format, record_count, uri, expires_on] + required: [collection, as_of, format, record_count, size_bytes, checksum_sha256, uri, expires_on] properties: collection: type: string From 1c46b950cce22bdf22ed66f2bba6470d99618b14 Mon Sep 17 00:00:00 2001 From: Tobias Macey Date: Fri, 11 Sep 2026 13:07:37 -0400 Subject: [PATCH 6/7] docs(learner-records): settle the declined-learner question as out of scope The organization can see which learners declined by diffing its roster, and no response shape changes that. How it may use that knowledge is a contract term owned by legal and contracting, and issuing a client presumes those terms exist, so the spec drops the options list and keeps the per-learner record shape. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014zeAAspVVGJDPn944Gt9vz --- docs/b2b-learner-records-design.md | 30 +++++++++------------------- docs/b2b-learner-records-onepager.md | 14 ++++++------- 2 files changed, 15 insertions(+), 29 deletions(-) diff --git a/docs/b2b-learner-records-design.md b/docs/b2b-learner-records-design.md index 1f5436b..62033b5 100644 --- a/docs/b2b-learner-records-design.md +++ b/docs/b2b-learner-records-design.md @@ -288,29 +288,17 @@ The existing `b2b_dashboard` tenant's k-anonymized org-level views disclose no individual and continue to cover the whole cohort. No consent join, no change to `mv_b2b_*`. -### Open: does withholding outcomes actually protect the learner? +### Settled: the organization can see who declined -This is the sharpest unresolved issue in the design. +The organization holds the full roster and the identities, so it can see exactly +which of its learners carry `outcomes_shared: false`. Field suppression conceals +a learner's *outcomes*, not their *decision*, and no response shape changes that +while records stay individually identifiable. -Because the organization holds the full roster and the identities, it can see -exactly which of its learners carry `outcomes_shared: false`. The mechanism -therefore conceals a learner's *outcomes* but publishes their *decision* — and -an employee visibly declining to share progress with their employer may be more -exposed than one whose completion data was simply shared. - -No response shape fixes this while the record remains individually identifiable. -The available directions are: - -* **Contractual** — terms of use forbidding adverse action on non-participation. - Cheap, and the only option that preserves the current record shape. -* **Aggregate-only participation reporting** — expose consent coverage as a - floored count per organization and never per learner, accepting that seat - reconciliation degrades. -* **Revisit what the learner is asked** — if consent cannot be declined without - the employer knowing, the consent language should probably say so. - -Worth putting to whoever owns the consent language early, since the third option -changes their work rather than ours. +This service doesn't try to. How an organization may use learner data is set by +its contract, which legal and contracting own. Issuing a client presumes those +terms are already in place. The record shape stays as specified: per-learner, +identity intact, outcomes nulled when `outcomes_shared` is false. ### Settled: a contracted provider receives identity diff --git a/docs/b2b-learner-records-onepager.md b/docs/b2b-learner-records-onepager.md index 6057166..690721c 100644 --- a/docs/b2b-learner-records-onepager.md +++ b/docs/b2b-learner-records-onepager.md @@ -182,14 +182,12 @@ consent withdrawal arrives on the sync cursor. ## Open questions -**1. Does withholding outcomes actually protect the learner?** Because the -organization holds the full roster, it can identify exactly who did not share by -set difference — whatever the response reports. The mechanism therefore hides a -learner's *outcomes* but not their *decision*, which exposes them to a different -pressure from their employer. Options are contractual (terms forbidding adverse -action on non-participation), or design-side (report participation only in -aggregate above a threshold, and accept degraded seat reporting). Unresolved, -and the sharpest issue in this design. +**1. Does withholding outcomes actually protect the learner?** *Decided: out of +scope for this service.* The organization holds the full roster, so it can tell +who did not share by set difference, whatever the response reports. How an +organization may use learner data is set by its contract, which legal and +contracting own, and issuing a client presumes those terms are in place. The +record shape stays per-learner: identity intact, outcomes suppressed. **2. Does a contracted provider receive learner identity at all?** *Decided: yes.* The organization already holds its learners' identity (they are its From 555ed57ccc94567ee7454e0b68d2d844cc0c9caf Mon Sep 17 00:00:00 2001 From: Tobias Macey Date: Wed, 16 Sep 2026 17:16:15 -0400 Subject: [PATCH 7/7] docs(learner-records): address mbertrand's review of the spec Settles the questions the review raised that the docs left implicit: - Access is organization-wide: a client reads every contract under the organizations it lists, and contract_id is a filter. A contract outside the path's organization returns an empty page. - Removals reach a syncing client as is_current: false on the learner, built from the userorganization snapshot. updated_since is a per-record change time, not as_of, and syncing needs include_inactive=true to see deactivations. Erasure is named as an open gap. - Exports are written to S3 by a batch job outside the API and read through cross-account IAM, SFTP, or a presigned URL from /exports. - Error bodies carry a code enum, and validation failures are documented as a deliberate 400. - Contract expiry is left open for pdpinch. Also: "access" replaces "grant", servers and token URL match ol-infrastructure, CSV nulls are empty fields, the revoked-certificate description matches #56 and #59, and the aggregates exemption is qualified. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01CUMWN5wuNmLj5VMUUpt3rV --- docs/b2b-learner-records-design.md | 80 +++++++-- docs/b2b-learner-records-onepager.md | 76 +++++--- ...-learner-records-provider-authorization.md | 74 ++++++-- docs/openapi/b2b-learner-records-v1.yaml | 170 ++++++++++++++---- 4 files changed, 300 insertions(+), 100 deletions(-) diff --git a/docs/b2b-learner-records-design.md b/docs/b2b-learner-records-design.md index 62033b5..9fed3d4 100644 --- a/docs/b2b-learner-records-design.md +++ b/docs/b2b-learner-records-design.md @@ -101,8 +101,38 @@ Out of scope; say so to partners rather than letting them discover it. **8. Freshness.** The MVs are `refresh_method='manual'`, driven by a Dagster asset. "Last active" is as stale as the last refresh. The `as_of` envelope the -existing tenant already returns carries this honestly and doubles as the -`updated_since` cursor for incremental sync. +existing tenant already returns carries this honestly. + +**9. Incremental sync needs a per-row change timestamp.** `as_of` is one +timestamp for the whole view, so it can't say which rows changed. Each +learner-grain model needs a `record_updated_on` column that `updated_since` is +compared against: the latest of the enrollment's created and updated times, the +grade's updated time and the certificate's updated time, rolled up per learner +on the learner model. Consent's change timestamp (§4) and removal time (gap 10) +feed it when they land. Activity doesn't: a day's activity first appears at a +refresh after that day began, so a timestamp derived from it would already sort +below the `updated_since` a client passes. Activity changes reach a partner only +through a full reload, and the spec says so on each activity field. + +**10. Removals don't reach a syncing client.** A record that stops appearing +sends no signal to a client applying changes: the partner's copy keeps it +forever. Three cases: + +- *Deactivated enrollment* (unenrolled, refunded, reassigned seat). Already a + state change: `enrollment_is_active` flips and `record_updated_on` moves. + It is filtered out by default, so a syncing client has to pass + `include_inactive=true` to receive it. +- *Learner removed from the organization.* `bridge_user_organization` is + current state, so the row vanishes. A learner removed from the roster who + still holds a contract enrollment stays on `/learners` as + `membership_source: enrollment`. A roster-only learner drops out entirely. + The learner model needs to keep that row with `is_current: false`, using + `snapshot_mitxonline_b2b_userorganization`, which closes a removed + membership's version (`hard_deletes: invalidate`, one-run resolution) rather + than deleting it. `dbt_valid_to` becomes the row's `record_updated_on`. +- *Erasure request.* Not covered. A non-current row still carries email and + name, which an erasure has to remove. How retirement reaches the warehouse, + and what a departed row keeps afterward, is an open gap. ## 2. Existing tenant, or a new one? @@ -128,9 +158,10 @@ exists to return. `require_org_manager` needs three things a client-credentials token does not have: an `organization` claim to check membership against, a `sub` to name in the MITx Online round-trip, and a human whose `is_manager` flag was curated in -Django admin. There is no user in this flow. The org grant has to come from the -*client's* registration, which is a different check, reached by a different -code path, with different failure modes. +Django admin. The warehouse carries that flag (§1); the token doesn't. There is +no user in this flow. Organization access has to come from the *client's* +registration, which is a different check, reached by a different code path, +with different failure modes. ### The blast radius attaches to a different principal @@ -139,9 +170,12 @@ permits learner-level disclosure all attach to the partner client, not to a logged-in org manager. Those are exactly the knobs the architecture already puts in each tenant's own `config.py` and `auth.py`. -### It costs one package and one registry line +### It costs one package and a little wiring -`main.py`'s `TENANTS` list is the documented extension point. A new tenant gets +`main.py`'s `TENANTS` list is the documented extension point. Beyond the +package, a tenant needs its import and registry entry in `main.py`, +`add_shared_error_handlers` in its `create_app`, and a readiness registration. +A new tenant gets its own OpenAPI document at its own mount — which is precisely the artifact you want to hand to a partner, without the aggregate dashboard's endpoints in it. @@ -156,7 +190,7 @@ now means the learner-grain schema can be granted to that role alone later. src/ol_analytics_api/tenants/b2b_learner_records/ app.py # create_app(); title/description say "identifiable learner records" config.py # own StarRocks schema, own page caps, own audit settings - auth.py # client-credentials principal, org-grant check, scope gating + auth.py # client-credentials principal, organization access check, scope gating models.py # Learner, Enrollment, CourseRun — no CohortPolicy routers/ organizations.py @@ -188,9 +222,13 @@ Three collections, all org-scoped, all read-only: Plus `updated_since` incremental sync on the two record collections, because a partner mirroring into their own LMS should not re-read the whole licence daily. +Sync delivers changes and removals (§1 gaps 9 and 10), not activity, so partners +still reload in full periodically. **Two delivery channels over one schema.** The REST API above, and a -per-organization bulk export (S3/SFTP) written on each refresh, discoverable via +per-organization bulk export written to S3 on each refresh by a batch job +outside the API. Partners read it through a cross-account IAM role limited to +their organization's prefix, over SFTP, or by a short-lived presigned URL from `/organizations/{id}/exports`. The export is a second encoding of the same records under the same field names, produced by the same query with the same consent enforcement — not a second data product with its own semantics. A @@ -261,6 +299,8 @@ is worse on every axis that matters here: suppression a withdrawal is simply a changed record with `outcomes_shared: false` and nulled outcomes; a client that upserts normally drops the data it held, with no extra object type and no retention window to reason about. + Learners who leave the organization follow the same pattern: a changed record + with `is_current: false` (§1 gap 10). That third point is the one that would have been expensive to discover late. @@ -284,9 +324,20 @@ That third point is the one that would have been expensive to discover late. ### Settled: aggregates are exempt -The existing `b2b_dashboard` tenant's k-anonymized org-level views disclose no -individual and continue to cover the whole cohort. No consent join, no change to -`mv_b2b_*`. +The existing `b2b_dashboard` tenant's k-anonymized org-level views continue to +cover the whole cohort. No consent join, no change to `mv_b2b_*`. + +That is not the same as saying the aggregates disclose no individual. Take ten +learners, nine sharing. The funnel says 6 certified; the records show 5 +certified among the nine who share, all named. The sixth certificate belongs to +the one who declined, whose name is in the same response. Both counts clear the +floor of 5, which protects small cohorts, not a small remainder. + +Different credentials reach the two tenants today: a logged-in manager for the +aggregates, a contracted client for the records. Nothing stops one organization +holding both, and the organization is who consent protects the learner from. +This doesn't change the design. The organization can already tell who declined +(below), and its use of learner data is governed by its contract. ### Settled: the organization can see who declined @@ -307,6 +358,7 @@ Yes. Identity is not redacted for any client (§3). ### Settled: who authorizes a provider, through what workflow? The contract settles access. MIT issues one Keycloak client per contracted -integration, with the organizations it may read carried as a claim. The -partner handles per-user authorization in its own LMS. See +integration, with the organizations it may read carried as a claim. Access +covers every contract under those organizations. The partner handles per-user +authorization in its own LMS. See [`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). diff --git a/docs/b2b-learner-records-onepager.md b/docs/b2b-learner-records-onepager.md index 690721c..893bcc8 100644 --- a/docs/b2b-learner-records-onepager.md +++ b/docs/b2b-learner-records-onepager.md @@ -23,18 +23,21 @@ existing tenant resolves a logged-in user's manager status, and there is no user in a machine-to-machine flow. Separate tenants cost one package and one registry entry, and give the partner its own API document. -**Machine-to-machine auth, with per-client organization grants.** Each client is -issued its own credentials and an explicit set of organizations it may read. A -provider working for several organizations holds one grant per organization, so -a contract ending revokes exactly one. Requests outside the grant are refused -identically to requests for organizations that do not exist, so the interface -cannot be used to enumerate MIT's customers. +**Machine-to-machine auth, one client per contracted integration.** Each client +is issued its own credentials and an explicit list of organizations it may +read, covering every contract under each. A provider working under several +contracts holds one client per contract, so a contract ending revokes exactly +that client's access. Requests for an organization the client doesn't list are +refused identically to requests for organizations that do not exist, so the +interface cannot be used to enumerate MIT's customers. **Two delivery channels over one schema.** A paged REST API for dashboards and -incremental sync, and a per-organization bulk export (S3/SFTP) on the refresh -cadence for partners loading into their own systems. Same records, same field -names, same consent enforcement — the export is a second encoding of the API's -schema, not a second data product. +incremental sync, and a per-organization bulk export on the refresh cadence for +partners loading into their own systems. A batch job writes the export to S3, +outside the API. Partners read it through cross-account IAM, over SFTP, or from +a presigned URL that `/exports` returns. Same records, same field names, same +consent enforcement — the export is a second encoding of the API's schema, not +a second data product. **Consent gates outcomes, not identity.** Organizations already hold their learners' names and addresses; they assigned the seats. What a learner opts into @@ -42,9 +45,11 @@ sharing is their *course status* — completion, progress, activity. So consent enforced on the outcome fields, and identity is governed by the contract with the partner rather than by the learner's consent choice. -**Aggregates are exempt.** The existing k-anonymized org-level views disclose no -individual and continue to cover the whole cohort. No consent join, no change to -the aggregate models. +**Aggregates are exempt.** The existing k-anonymized org-level views continue to +cover the whole cohort. No consent join, no change to the aggregate models. The +floor protects small cohorts, not a small remainder: an organization holding +both the aggregates and the records could subtract one from the other to learn +a declining learner's outcome. See design §4. **Consent enforcement fails closed.** No recorded opt-in means no outcome data. Because suppression is per-field rather than per-record, the service can ship @@ -61,10 +66,11 @@ timestamp, so the withdrawal never reaches the partner holding a copy. Three collections, all organization-scoped and read-only: **learners** (roster and rollup), **enrollments** (learner × course, the grain that answers "did they complete it"), and **courses** (the contracts and course runs the identifiers -refer to), plus an **exports** manifest listing the bulk files. Incremental sync -is driven by a refresh cursor; a consent withdrawal arrives on it as an ordinary -changed record with outcomes nulled, so a client that upserts normally drops the -data it held. +refer to), plus an **exports** manifest returning presigned URLs to the bulk +files. Incremental sync compares each record's change time with the previous +sync's `as_of`. A consent withdrawal arrives as an ordinary changed record with +outcomes nulled, and a learner who leaves the organization arrives with +`is_current: false`, so a client that upserts normally drops the data it held. Out of scope for v1: program-level progress, per-assessment detail, any write operation, and non-MITx-Online platforms. @@ -93,6 +99,7 @@ Every collection returns the same envelope, with `data` typed to its record: "organization_id": "8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21", "organization_name": "Contoso Manufacturing", "membership_source": "both", + "is_current": true, "is_organization_manager": false, "first_enrolled_on": "2026-02-03T14:22:11Z", "last_enrolled_on": "2026-05-19T09:04:52Z", @@ -169,7 +176,8 @@ Every collection returns the same envelope, with `data` typed to its record: "record_count": 163, "size_bytes": 214880, "checksum_sha256": "9f2c1b7ae4d05c8831fbb2e6a0d47c3915ee8b6042d1f7c9a3b508e2d6417f0a", - "uri": "s3://ol-b2b-exports/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz", + "uri": "https://ol-b2b-exports.s3.amazonaws.com/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz?X-Amz-Expires=900&X-Amz-Signature=…", + "uri_expires_on": "2026-08-13T09:30:00Z", "expires_on": "2026-09-12T06:15:00Z" } ``` @@ -204,18 +212,28 @@ not a configuration detail. [`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). The contract settles access. MIT issues one Keycloak client per contracted integration, carrying its organizations as a claim. The partner handles -per-user authorization in its own LMS. +per-user authorization in its own LMS. The contract is also the organization's +record of who can read its data: it signed the contract naming the provider, so +no organization-facing view is planned. -**4. Is consent per-organization or global?** A learner holding seats under two -organizations should be able to share with one and not the other. A single -global flag works mechanically but makes withdrawal all-or-nothing. An input to -the consent design rather than a question this service can answer. +**4. Is consent per-organization or global?** *Decided: per `(learner, +organization)`* (design §4). A learner holding seats under two organizations can +share with one and not the other. A global flag would make withdrawal +all-or-nothing. + +**5. Does access expire with the contract?** *Open, for pdpinch.* Nothing +checks a contract end date today, so a client nobody removes keeps working. +Options: an end-date claim on the client, or a warehouse check against +`dim_contract.contract_is_active`. See the +[authorization decision](b2b-learner-records-provider-authorization.md). ## Dependencies -Both are degrading rather than blocking — the service ships without either and -fills in as they land. The learner-consent field, without which every record -reads `outcomes_shared: false`; and a per-learner activity model, without which -"last active" and the engagement counters are null. Onboarding the first -partner also needs the per-contract Keycloak client template and a bearer-only -gateway route. +Degrading, not blocking — the service ships without these and fills in as they +land: the learner-consent field, without which every record reads +`outcomes_shared: false`; a per-learner activity model, without which "last +active" and the engagement counters are null; and learner removal on +`mv_b2b_learner`, without which `is_current` is always true. + +Blocking the first partner: the per-contract Keycloak client template and a +bearer-only gateway route. No partner can authenticate without both. diff --git a/docs/b2b-learner-records-provider-authorization.md b/docs/b2b-learner-records-provider-authorization.md index 4de1a4a..fdcc3ef 100644 --- a/docs/b2b-learner-records-provider-authorization.md +++ b/docs/b2b-learner-records-provider-authorization.md @@ -15,6 +15,12 @@ issues client credentials that encode those terms. The partner builds the integr its own LMS and handles per-user authorization on its side. MIT does not model the partner's users, and a credential reads everything its contract covers. +Access is organization-wide. A client reads every learner under every contract +of each organization it lists, including learners on contracts another +provider serves. `contract_id` on the API is a filter the caller chooses, not a +limit on the credential. The contract is with the organization, and it grants +access to the organization's learners, not to one contract's cohort. + ## How the terms are encoded One Keycloak client per contracted integration, defined in Pulumi @@ -24,30 +30,38 @@ that creates the client is MIT's record of the access. | Contract term | On the client | | --- | --- | -| Which organizations | A hardcoded claim (working name `learner_records_organizations`) listing the Keycloak organization UUIDs, via `keycloak.openid.HardcodedClaimProtocolMapper` | +| Which organizations | A hardcoded claim (working name `learner_records_organizations`) listing the Keycloak organization UUIDs, via `keycloak.openid.HardcodedClaimProtocolMapper` with `jsonType.label` set to `JSON`, so the claim arrives as an array rather than a string | | Read access | Default client scope `learner-records:read`. There is no separate identity scope; see Consequences | | API audience | `keycloak.openid.AudienceProtocolMapper`, as the Superset client already uses | | Client-credentials only | `service_accounts_enabled=True`, standard flow and direct grants off | A provider working for two organizations under two contracts holds two clients. -When a contract ends, removing its client revokes exactly that access, which -keeps the property the design already required. +When a contract ends, removing its client revokes that credential. Any other +client listing the same organization keeps reading the same rows, so ending one +provider's access means removing that provider's clients, not the +organization's. + +**Open, for pdpinch: does access expire with the contract?** Today it doesn't. +The per-request steps below check no end date, so a client whose cleanup PR is +forgotten keeps issuing tokens. Two ways to enforce it: -Recommended (it costs one more claim): carry the contract end date as a claim -too, and have the API refuse tokens once that date has passed. A forgotten -cleanup PR then fails closed. +- Carry the contract end date as a claim, and refuse tokens past it. +- Check the warehouse instead. `dim_contract` already has `contract_is_active` + and `contract_end_date`, and the API serves both on `/courses`. ## What the API does In `tenants/b2b_learner_records/auth.py`, per request: -1. Read the organization claim and scopes from the validated token. +1. Read the organization claim and scopes from the validated token. A missing + claim, or one that isn't an array of UUIDs, lists no organizations, as + `b2b_dashboard/auth.py` already treats a malformed organization claim. 2. If the path's `organization_id` is not in the claim, return the existing 403, identical to the response for an organization that does not exist. 3. Require the `learner-records:read` scope. Identity fields are always populated. -No call to mitxonline or any other service, and no grant store. The client +No call to mitxonline or any other service, and no access store. The client definition is the only place access is recorded. ## Consequences @@ -60,22 +74,26 @@ definition is the only place access is recorded. - **Revocation is a Pulumi change.** Tokens issued before the client is removed stay valid until they expire. The access-token lifespan for these clients bounds that window. It has not been checked for the `olapps` realm yet. -- **Exports need to be covered by the same credential.** The draft says export - files are "readable with the credentials issued alongside the API client". - Standing storage credentials would be a second thing to revoke when a - contract ends. Recommended: `/exports` returns short-lived presigned HTTPS - URLs, so the API client is the only credential. Not yet applied to the - OpenAPI draft. +- **Exports are written out of band and reached three ways.** A batch job + writes each organization's export to its own prefix in S3. A partner reads it + through a cross-account IAM role limited to that organization's prefix (no + `ListBucket` beyond it), over SFTP, or through `/exports`, which returns a + short-lived presigned HTTPS URL to the same file. The IAM role and SFTP + account are credentials separate from the API client, so ending a contract + means revoking those too. `/exports` needs only the API client. - **The partner owns end-user access.** Obligations on how the partner restricts learner records inside its LMS belong in the contract. Nothing on MIT's side enforces them. +- **The contract is the organization's record of who reads its data.** The + organization signed the contract that names the provider, so it already + knows. No organization-facing view of clients is planned. ## Not chosen -- **A grant record in mitxonline, checked at request time,** with - organization managers able to see and revoke grants. That adds a runtime +- **An access record in mitxonline, checked at request time,** with + organization managers able to see and revoke access. That adds a runtime dependency and a UI, but the decision it would store is already made in the - contract. + contract, which is also where the organization sees it. - **Making the service account a member of the Keycloak organization,** so the existing organization-membership mapper lists it. mitxonline's org sync would import the service account as a learner, and it would then appear in @@ -89,6 +107,26 @@ definition is the only place access is recorded. - Whether APISIX's `openid-connect` plugin passes the hardcoded claim and scopes through in `X-Userinfo` on a bearer-only route. The learner-records mount needs a bearer-only route, but today's routes use the redirect flow. - Check on QA before writing `auth.py`. + Check on QA. +- That APISIX *overwrites* a caller-supplied `X-Userinfo` rather than passing + it through. `core/auth/userinfo.py` decodes whatever header arrives without + validating a token, and the organization check reads from it. Also confirm + the pod can't be reached except through the gateway route: `k8s/` defines no + NetworkPolicy. Check both on QA. - The access-token lifespan these clients will get, since it is the revocation window. + +## Follow-ups + +Not needed to write the tenant, but each needs an owner before partners +multiply: + +- **Client lifecycle.** One client per contract covers creation only. Who + delivers the secret to the partner, who rotates it, and who removes it when + the contract ends are unassigned. +- **Audit logging, before the first partner.** `auth.py` logs the client ID and + organization on every authorized request. It doesn't log how many records + were returned, and nothing yet says where those logs are kept or who reviews + them. +- **Rate limits.** The contract documents a 429 but no limits, and hasn't + decided whether APISIX or the app enforces them. diff --git a/docs/openapi/b2b-learner-records-v1.yaml b/docs/openapi/b2b-learner-records-v1.yaml index dce8b7f..3f49106 100644 --- a/docs/openapi/b2b-learner-records-v1.yaml +++ b/docs/openapi/b2b-learner-records-v1.yaml @@ -22,13 +22,22 @@ info: seat. Enforcement fails closed, so every record reads `outcomes_shared: false` until the upstream consent field ships. * **The same schemas are delivered two ways.** This REST API, and a - per-organization bulk export (S3/SFTP) written on each refresh. The export - carries the same records under the same field names. - * **`as_of` is the data's freshness and the sync cursor.** Records come from - materialized views refreshed on a schedule. Pass a previous `as_of` as - `updated_since` for incremental sync; a consent withdrawal arrives as a - normal changed record with `outcomes_shared: false` and outcome fields - nulled, which the client upserts to drop the data it held. + per-organization bulk export written to S3 on each refresh. The export + carries the same records under the same field names. It is readable + through a cross-account IAM role, over SFTP, or from the presigned URL + `/exports` returns. + * **`as_of` is the data's freshness.** Records come from materialized views + refreshed on a schedule. For incremental sync, pass the previous sync's + `as_of` as `updated_since` together with `include_inactive=true`. A + consent withdrawal arrives as a changed record with + `outcomes_shared: false` and outcome fields nulled, a deactivated + enrollment with `enrollment_is_active: false`, and a learner who left the + organization with `is_current: false`. A client that upserts drops what + it held. Activity changes don't arrive on sync; reload in full + periodically. + * **Paging is stable within one refresh.** Every collection has a unique + order, but a refresh between two pages can shift rows across the offset. + If `as_of` changes between pages, restart from offset 0. Every property of the `Learner`, `Enrollment` and `CourseRun` record schemas carries `x-data-readiness`: `available` (backed today), @@ -43,9 +52,9 @@ info: identifier: BSD-3-Clause servers: - - url: https://api.mitxonline.mit.edu/api/v1/learner-records + - url: https://analytics.ol.mit.edu/api/v1/learner-records description: Production (planned) - - url: https://api.qa.mitxonline.mit.edu/api/v1/learner-records + - url: https://analytics-qa.ol.mit.edu/api/v1/learner-records description: QA — partner integration testing (planned) security: @@ -105,6 +114,7 @@ paths: organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 organization_name: Contoso Manufacturing membership_source: both + is_current: true is_organization_manager: false first_enrolled_on: '2026-02-03T14:22:11Z' last_enrolled_on: '2026-05-19T09:04:52Z' @@ -122,6 +132,7 @@ paths: organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 organization_name: Contoso Manufacturing membership_source: both + is_current: true is_organization_manager: false first_enrolled_on: '2026-03-11T10:02:44Z' last_enrolled_on: '2026-03-11T10:02:44Z' @@ -295,8 +306,12 @@ paths: operationId: listExports summary: Bulk export files available for this organization description: >- - Manifest of the S3/SFTP bulk exports, one set per collection per - refresh. Records inside carry the same schemas as this API, identity + Manifest of the bulk exports, one set per collection per refresh, each + with a short-lived presigned HTTPS URL. A batch job writes the files to + S3 outside this API; the same files are also readable through a + cross-account IAM role limited to the organization's prefix, or over + SFTP. This endpoint is the path that needs no credential beyond the API + client. Records inside carry the same schemas as this API, identity fields included. Ordered by `as_of` descending, then `collection` and `format`, which is unique. parameters: @@ -331,7 +346,8 @@ paths: record_count: 163 size_bytes: 214880 checksum_sha256: 9f2c1b7ae4d05c8831fbb2e6a0d47c3915ee8b6042d1f7c9a3b508e2d6417f0a - uri: s3://ol-b2b-exports/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz + uri: https://ol-b2b-exports.s3.amazonaws.com/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz?X-Amz-Expires=900&X-Amz-Signature=… + uri_expires_on: '2026-08-13T09:30:00Z' expires_on: '2026-09-12T06:15:00Z' '400': { $ref: '#/components/responses/BadRequest' } '401': { $ref: '#/components/responses/Unauthorized' } @@ -347,12 +363,12 @@ components: description: >- Keycloak client credentials, validated at the gateway. One client is issued per contracted integration and lists the organizations it may - read. Scopes - bound to a client are a contractual limit on that credential; they are - not the consent mechanism. + read, every contract under each included. Scopes bound to a client are + a contractual limit on that credential; they are not the consent + mechanism. flows: clientCredentials: - tokenUrl: https://sso.mit.edu/realms/olapps/protocol/openid-connect/token + tokenUrl: https://sso.ol.mit.edu/realms/olapps/protocol/openid-connect/token scopes: learner-records:read: Read records, identity fields included. @@ -370,6 +386,13 @@ components: name: contract_id in: query required: false + description: >- + Narrows the result to one of the organization's contracts. A filter, + not an access boundary: a client reads every contract under the + organizations it lists. A contract not under the path's organization + matches nothing and returns an empty page, not a 403; the result is the + same whether the contract belongs to another organization or doesn't + exist. schema: { type: integer, format: int64 } example: 42 @@ -411,7 +434,14 @@ components: name: updated_since in: query required: false - description: Return only records changed at or after this instant. Pass the previous sync's `as_of`. + description: >- + Return only records changed at or after this instant. Pass the previous + sync's `as_of`. A record's change time is the latest of its enrollment, + grade, certificate, organization membership and (once it exists) + consent changes. Activity doesn't move it. Deactivated enrollments and + learners no longer in the organization are changes too, but + `include_inactive` still filters them, so pass + `include_inactive=true` when syncing. schema: { type: string, format: date-time } example: '2026-08-12T06:15:00Z' @@ -419,7 +449,14 @@ components: name: include_inactive in: query required: false - description: Include deactivated enrollments (unenrolled, refunded, transferred). + description: >- + On `/enrollments`, includes deactivated enrollments (unenrolled, + refunded, transferred). On `/learners`: rollups count deactivated + enrollments too; learners whose only enrollments are deactivated are + included; learners with `is_current: false` are included; and a roster + member whose only enrollment is deactivated reads + `membership_source: both` rather than `roster`. Roster members with no + enrollments at all are included either way. schema: { type: boolean, default: false } Limit: @@ -432,6 +469,10 @@ components: name: offset in: query required: false + description: >- + Stable only within one refresh. If the envelope's `as_of` differs from + the previous page's, restart from 0; otherwise rows can be skipped or + repeated. schema: { type: integer, minimum: 0, default: 0 } schemas: @@ -474,6 +515,7 @@ components: - organization_id - organization_name - membership_source + - is_current - is_organization_manager - first_enrolled_on - last_enrolled_on @@ -509,6 +551,16 @@ components: x-data-readiness: available membership_source: $ref: '#/components/schemas/MembershipSource' + is_current: + type: boolean + description: >- + False once the learner is no longer associated with the + organization: off the roster and holding no enrollment under its + contracts. The record stays so a syncing client learns of the + removal; it is returned only with `include_inactive=true`. + Identity is still populated. Erasure requests are handled + separately and not yet specified. + x-data-readiness: pending-model is_organization_manager: type: boolean description: Administers the organization in MITx Online. @@ -541,7 +593,7 @@ components: last_active_on: type: [string, 'null'] format: date - description: Most recent day with recorded course activity. A date, not a timestamp — activity is aggregated per day. + description: Most recent day with recorded course activity. A date, not a timestamp — activity is aggregated per day. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model courses_in_progress: type: [integer, 'null'] @@ -558,7 +610,12 @@ components: certificates_earned: type: [integer, 'null'] minimum: 0 - description: Includes program certificates, which have no course run and so are not in `courses_certified`. + description: >- + Includes program certificates, which have no course run and so are + not in `courses_certified`. This is the only program-level figure in + v1; program progress is otherwise out of scope, so nothing else in + the payload reconciles against it. With `contract_id`, program + certificates belong to no single contract and are excluded. x-data-readiness: derived Enrollment: @@ -682,31 +739,35 @@ components: x-data-readiness: available certificate_is_revoked: type: [boolean, 'null'] - description: Null where no certificate exists. A revoked certificate leaves `completion_status` at `passed`. + description: >- + Null where no certificate exists. A revoked certificate doesn't + count as certified: `completion_status` then follows the grade, so + it can read `passed`, `in_progress` or `not_started`. x-data-readiness: available last_active_on: type: [string, 'null'] format: date - description: Most recent day with recorded activity in this course run. + description: Most recent day with recorded activity in this course run. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model days_active: type: [integer, 'null'] minimum: 0 - description: Distinct days with recorded activity in this course run. + description: Distinct days with recorded activity in this course run. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model videos_watched: type: [integer, 'null'] minimum: 0 - description: Distinct video blocks played. + description: Distinct video blocks played. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model problems_attempted: type: [integer, 'null'] minimum: 0 - description: Distinct problem blocks attempted. + description: Distinct problem blocks attempted. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model chatbot_interactions: type: [integer, 'null'] minimum: 0 + description: Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model CourseRun: @@ -758,7 +819,7 @@ components: ExportFile: type: object description: One bulk-export file. Contents use this API's record schemas. - required: [collection, as_of, format, record_count, size_bytes, checksum_sha256, uri, expires_on] + required: [collection, as_of, format, record_count, size_bytes, checksum_sha256, uri, uri_expires_on, expires_on] properties: collection: type: string @@ -769,6 +830,9 @@ components: description: The refresh this file was written from. format: type: string + description: >- + In `csv`, null is an empty unquoted field and an empty string is + `""`. `jsonl` and `parquet` carry null natively. enum: [jsonl, csv, parquet] record_count: type: integer @@ -782,11 +846,20 @@ components: uri: type: string format: uri - description: '`s3://` or `sftp://` location, readable with the credentials issued alongside the API client.' + description: >- + Presigned HTTPS URL for the file. Needs no credential beyond itself; + valid until `uri_expires_on`. List the manifest again for a fresh + URL. + uri_expires_on: + type: string + format: date-time + description: When `uri` stops working. The file itself remains until `expires_on`. expires_on: type: string format: date-time - description: After this, the file is deleted and must be re-fetched from a newer refresh. + description: >- + File retention. After this, the file is deleted and must be + re-fetched from a newer refresh. MembershipSource: type: string @@ -808,35 +881,54 @@ components: Error: type: object - required: [detail] + required: [code, detail] properties: - detail: { type: string } + code: + type: string + description: >- + Stable, machine-readable reason. Branch on this, not on `detail`, + whose wording may change. + enum: + - invalid_parameter + - unauthorized + - missing_scope + - no_organization_access + - rate_limited + - unavailable + detail: + type: string + description: Human-readable explanation. responses: BadRequest: - description: Malformed parameter. + description: >- + Malformed parameter, including a non-UUID `organization_id` in the path. + Deliberately 400, not FastAPI's default 422: the service normalizes + validation errors to this body. content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'limit must be between 1 and 1000' } + example: { code: invalid_parameter, detail: 'limit: Input should be less than or equal to 1000' } Unauthorized: description: Missing, malformed or expired token. content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'Invalid or expired access token' } + example: { code: unauthorized, detail: 'Invalid or expired access token' } Forbidden: description: >- - Token valid but lacks a grant for this organization, or lacks the - required scope. Identical whether the organization is ungranted or does - not exist, so this cannot be used to enumerate organizations. + Token valid but lacks access to this organization + (`no_organization_access`) or lacks the required scope + (`missing_scope`). Identical whether the client lacks access or the + organization does not exist, so this cannot be used to enumerate + organizations. content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'No grant for the requested organization' } + example: { code: no_organization_access, detail: 'No access to the requested organization' } TooManyRequests: description: Per-client rate limit exceeded. @@ -846,7 +938,7 @@ components: content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'Rate limit exceeded' } + example: { code: rate_limited, detail: 'Rate limit exceeded' } ServiceUnavailable: description: Analytics store unreachable or saturated. @@ -856,4 +948,4 @@ components: content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'Analytics store unavailable' } + example: { code: unavailable, detail: 'Analytics store unavailable' }