diff --git a/docs/b2b-learner-records-design.md b/docs/b2b-learner-records-design.md index 5f750aa..9fed3d4 100644 --- a/docs/b2b-learner-records-design.md +++ b/docs/b2b-learner-records-design.md @@ -101,8 +101,38 @@ Out of scope; say so to partners rather than letting them discover it. **8. Freshness.** The MVs are `refresh_method='manual'`, driven by a Dagster asset. "Last active" is as stale as the last refresh. The `as_of` envelope the -existing tenant already returns carries this honestly and doubles as the -`updated_since` cursor for incremental sync. +existing tenant already returns carries this honestly. + +**9. Incremental sync needs a per-row change timestamp.** `as_of` is one +timestamp for the whole view, so it can't say which rows changed. Each +learner-grain model needs a `record_updated_on` column that `updated_since` is +compared against: the latest of the enrollment's created and updated times, the +grade's updated time and the certificate's updated time, rolled up per learner +on the learner model. Consent's change timestamp (§4) and removal time (gap 10) +feed it when they land. Activity doesn't: a day's activity first appears at a +refresh after that day began, so a timestamp derived from it would already sort +below the `updated_since` a client passes. Activity changes reach a partner only +through a full reload, and the spec says so on each activity field. + +**10. Removals don't reach a syncing client.** A record that stops appearing +sends no signal to a client applying changes: the partner's copy keeps it +forever. Three cases: + +- *Deactivated enrollment* (unenrolled, refunded, reassigned seat). Already a + state change: `enrollment_is_active` flips and `record_updated_on` moves. + It is filtered out by default, so a syncing client has to pass + `include_inactive=true` to receive it. +- *Learner removed from the organization.* `bridge_user_organization` is + current state, so the row vanishes. A learner removed from the roster who + still holds a contract enrollment stays on `/learners` as + `membership_source: enrollment`. A roster-only learner drops out entirely. + The learner model needs to keep that row with `is_current: false`, using + `snapshot_mitxonline_b2b_userorganization`, which closes a removed + membership's version (`hard_deletes: invalidate`, one-run resolution) rather + than deleting it. `dbt_valid_to` becomes the row's `record_updated_on`. +- *Erasure request.* Not covered. A non-current row still carries email and + name, which an erasure has to remove. How retirement reaches the warehouse, + and what a departed row keeps afterward, is an open gap. ## 2. Existing tenant, or a new one? @@ -128,9 +158,10 @@ exists to return. `require_org_manager` needs three things a client-credentials token does not have: an `organization` claim to check membership against, a `sub` to name in the MITx Online round-trip, and a human whose `is_manager` flag was curated in -Django admin. There is no user in this flow. The org grant has to come from the -*client's* registration, which is a different check, reached by a different -code path, with different failure modes. +Django admin. The warehouse carries that flag (§1); the token doesn't. There is +no user in this flow. Organization access has to come from the *client's* +registration, which is a different check, reached by a different code path, +with different failure modes. ### The blast radius attaches to a different principal @@ -139,9 +170,12 @@ permits learner-level disclosure all attach to the partner client, not to a logged-in org manager. Those are exactly the knobs the architecture already puts in each tenant's own `config.py` and `auth.py`. -### It costs one package and one registry line +### It costs one package and a little wiring -`main.py`'s `TENANTS` list is the documented extension point. A new tenant gets +`main.py`'s `TENANTS` list is the documented extension point. Beyond the +package, a tenant needs its import and registry entry in `main.py`, +`add_shared_error_handlers` in its `create_app`, and a readiness registration. +A new tenant gets its own OpenAPI document at its own mount — which is precisely the artifact you want to hand to a partner, without the aggregate dashboard's endpoints in it. @@ -156,7 +190,7 @@ now means the learner-grain schema can be granted to that role alone later. src/ol_analytics_api/tenants/b2b_learner_records/ app.py # create_app(); title/description say "identifiable learner records" config.py # own StarRocks schema, own page caps, own audit settings - auth.py # client-credentials principal, org-grant check, scope gating + auth.py # client-credentials principal, organization access check, scope gating models.py # Learner, Enrollment, CourseRun — no CohortPolicy routers/ organizations.py @@ -188,9 +222,13 @@ Three collections, all org-scoped, all read-only: Plus `updated_since` incremental sync on the two record collections, because a partner mirroring into their own LMS should not re-read the whole licence daily. +Sync delivers changes and removals (§1 gaps 9 and 10), not activity, so partners +still reload in full periodically. **Two delivery channels over one schema.** The REST API above, and a -per-organization bulk export (S3/SFTP) written on each refresh, discoverable via +per-organization bulk export written to S3 on each refresh by a batch job +outside the API. Partners read it through a cross-account IAM role limited to +their organization's prefix, over SFTP, or by a short-lived presigned URL from `/organizations/{id}/exports`. The export is a second encoding of the same records under the same field names, produced by the same query with the same consent enforcement — not a second data product with its own semantics. A @@ -241,15 +279,10 @@ null; identity, contract, course run and enrollment facts are unaffected. The envelope's `outcomes_withheld_count` reports how many records in the result are in that state. -The field does not exist upstream yet. Until it ships, the tenant's -`consent_fail_open` setting (`OL_ANALYTICS_API_B2B_LEARNER_RECORDS_CONSENT_FAIL_OPEN`) -decides every record. It defaults to false, which fails closed: every record -reads `outcomes_shared: false` and the outcome columns are uniformly null. That -is a degraded response rather than an empty one, which means partner -integration can proceed against real records. A deployment that sets it to -true discloses outcomes for learners with no recorded decision. Once the field -lands, a recorded decision always wins and the setting covers only learners -with none. +The field does not exist upstream yet. Enforcement fails closed, so until it +ships every record reads `outcomes_shared: false` and the outcome columns are +uniformly null. That is a degraded response rather than an empty one, which +means partner integration can proceed against real records. ### Why suppression rather than exclusion @@ -266,6 +299,8 @@ is worse on every axis that matters here: suppression a withdrawal is simply a changed record with `outcomes_shared: false` and nulled outcomes; a client that upserts normally drops the data it held, with no extra object type and no retention window to reason about. + Learners who leave the organization follow the same pattern: a changed record + with `is_current: false` (§1 gap 10). That third point is the one that would have been expensive to discover late. @@ -289,9 +324,20 @@ That third point is the one that would have been expensive to discover late. ### Settled: aggregates are exempt -The existing `b2b_dashboard` tenant's k-anonymized org-level views disclose no -individual and continue to cover the whole cohort. No consent join, no change to -`mv_b2b_*`. +The existing `b2b_dashboard` tenant's k-anonymized org-level views continue to +cover the whole cohort. No consent join, no change to `mv_b2b_*`. + +That is not the same as saying the aggregates disclose no individual. Take ten +learners, nine sharing. The funnel says 6 certified; the records show 5 +certified among the nine who share, all named. The sixth certificate belongs to +the one who declined, whose name is in the same response. Both counts clear the +floor of 5, which protects small cohorts, not a small remainder. + +Different credentials reach the two tenants today: a logged-in manager for the +aggregates, a contracted client for the records. Nothing stops one organization +holding both, and the organization is who consent protects the learner from. +This doesn't change the design. The organization can already tell who declined +(below), and its use of learner data is governed by its contract. ### Settled: the organization can see who declined @@ -312,6 +358,7 @@ Yes. Identity is not redacted for any client (§3). ### Settled: who authorizes a provider, through what workflow? The contract settles access. MIT issues one Keycloak client per contracted -integration, with the organizations it may read carried as a claim. The -partner handles per-user authorization in its own LMS. See +integration, with the organizations it may read carried as a claim. Access +covers every contract under those organizations. The partner handles per-user +authorization in its own LMS. See [`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). diff --git a/docs/b2b-learner-records-onepager.md b/docs/b2b-learner-records-onepager.md index 20b8fce..893bcc8 100644 --- a/docs/b2b-learner-records-onepager.md +++ b/docs/b2b-learner-records-onepager.md @@ -23,18 +23,21 @@ existing tenant resolves a logged-in user's manager status, and there is no user in a machine-to-machine flow. Separate tenants cost one package and one registry entry, and give the partner its own API document. -**Machine-to-machine auth, with per-client organization grants.** Each client is -issued its own credentials and an explicit set of organizations it may read. A -provider working for several organizations holds one grant per organization, so -a contract ending revokes exactly one. Requests outside the grant are refused -identically to requests for organizations that do not exist, so the interface -cannot be used to enumerate MIT's customers. +**Machine-to-machine auth, one client per contracted integration.** Each client +is issued its own credentials and an explicit list of organizations it may +read, covering every contract under each. A provider working under several +contracts holds one client per contract, so a contract ending revokes exactly +that client's access. Requests for an organization the client doesn't list are +refused identically to requests for organizations that do not exist, so the +interface cannot be used to enumerate MIT's customers. **Two delivery channels over one schema.** A paged REST API for dashboards and -incremental sync, and a per-organization bulk export (S3/SFTP) on the refresh -cadence for partners loading into their own systems. Same records, same field -names, same consent enforcement — the export is a second encoding of the API's -schema, not a second data product. +incremental sync, and a per-organization bulk export on the refresh cadence for +partners loading into their own systems. A batch job writes the export to S3, +outside the API. Partners read it through cross-account IAM, over SFTP, or from +a presigned URL that `/exports` returns. Same records, same field names, same +consent enforcement — the export is a second encoding of the API's schema, not +a second data product. **Consent gates outcomes, not identity.** Organizations already hold their learners' names and addresses; they assigned the seats. What a learner opts into @@ -42,17 +45,17 @@ sharing is their *course status* — completion, progress, activity. So consent enforced on the outcome fields, and identity is governed by the contract with the partner rather than by the learner's consent choice. -**Aggregates are exempt.** The existing k-anonymized org-level views disclose no -individual and continue to cover the whole cohort. No consent join, no change to -the aggregate models. - -**Consent enforcement fails closed by default.** No recorded opt-in means no -outcome data, unless a deployment sets `consent_fail_open` to disclose outcomes -for learners with no recorded decision. Because suppression is per-field rather -than per-record, the service can ship before the consent field exists — failing -closed, every record simply reads `outcomes_shared: false` with outcomes null, -which is a degraded response rather than an empty one, so partner integration -can proceed against real records. One upstream +**Aggregates are exempt.** The existing k-anonymized org-level views continue to +cover the whole cohort. No consent join, no change to the aggregate models. The +floor protects small cohorts, not a small remainder: an organization holding +both the aggregates and the records could subtract one from the other to learn +a declining learner's outcome. See design §4. + +**Consent enforcement fails closed.** No recorded opt-in means no outcome data. +Because suppression is per-field rather than per-record, the service can ship +before the consent field exists — every record simply reads `outcomes_shared: +false` with outcomes null, which is a degraded response rather than an empty +one, so partner integration can proceed against real records. One upstream property is a hard requirement and cheap only if specified now: *withdrawal must be a retained state change, not a deleted row*. A deleted consent row is indistinguishable from one never granted and does not move the record's change @@ -63,10 +66,11 @@ timestamp, so the withdrawal never reaches the partner holding a copy. Three collections, all organization-scoped and read-only: **learners** (roster and rollup), **enrollments** (learner × course, the grain that answers "did they complete it"), and **courses** (the contracts and course runs the identifiers -refer to), plus an **exports** manifest listing the bulk files. Incremental sync -is driven by a refresh cursor; a consent withdrawal arrives on it as an ordinary -changed record with outcomes nulled, so a client that upserts normally drops the -data it held. +refer to), plus an **exports** manifest returning presigned URLs to the bulk +files. Incremental sync compares each record's change time with the previous +sync's `as_of`. A consent withdrawal arrives as an ordinary changed record with +outcomes nulled, and a learner who leaves the organization arrives with +`is_current: false`, so a client that upserts normally drops the data it held. Out of scope for v1: program-level progress, per-assessment detail, any write operation, and non-MITx-Online platforms. @@ -95,6 +99,7 @@ Every collection returns the same envelope, with `data` typed to its record: "organization_id": "8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21", "organization_name": "Contoso Manufacturing", "membership_source": "both", + "is_current": true, "is_organization_manager": false, "first_enrolled_on": "2026-02-03T14:22:11Z", "last_enrolled_on": "2026-05-19T09:04:52Z", @@ -171,7 +176,8 @@ Every collection returns the same envelope, with `data` typed to its record: "record_count": 163, "size_bytes": 214880, "checksum_sha256": "9f2c1b7ae4d05c8831fbb2e6a0d47c3915ee8b6042d1f7c9a3b508e2d6417f0a", - "uri": "s3://ol-b2b-exports/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz", + "uri": "https://ol-b2b-exports.s3.amazonaws.com/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz?X-Amz-Expires=900&X-Amz-Signature=…", + "uri_expires_on": "2026-08-13T09:30:00Z", "expires_on": "2026-09-12T06:15:00Z" } ``` @@ -206,18 +212,28 @@ not a configuration detail. [`b2b-learner-records-provider-authorization.md`](b2b-learner-records-provider-authorization.md). The contract settles access. MIT issues one Keycloak client per contracted integration, carrying its organizations as a claim. The partner handles -per-user authorization in its own LMS. +per-user authorization in its own LMS. The contract is also the organization's +record of who can read its data: it signed the contract naming the provider, so +no organization-facing view is planned. -**4. Is consent per-organization or global?** A learner holding seats under two -organizations should be able to share with one and not the other. A single -global flag works mechanically but makes withdrawal all-or-nothing. An input to -the consent design rather than a question this service can answer. +**4. Is consent per-organization or global?** *Decided: per `(learner, +organization)`* (design §4). A learner holding seats under two organizations can +share with one and not the other. A global flag would make withdrawal +all-or-nothing. + +**5. Does access expire with the contract?** *Open, for pdpinch.* Nothing +checks a contract end date today, so a client nobody removes keeps working. +Options: an end-date claim on the client, or a warehouse check against +`dim_contract.contract_is_active`. See the +[authorization decision](b2b-learner-records-provider-authorization.md). ## Dependencies -Both are degrading rather than blocking — the service ships without either and -fills in as they land. The learner-consent field, without which every record -reads `outcomes_shared: false`; and a per-learner activity model, without which -"last active" and the engagement counters are null. Onboarding the first -partner also needs the per-contract Keycloak client template and a bearer-only -gateway route. +Degrading, not blocking — the service ships without these and fills in as they +land: the learner-consent field, without which every record reads +`outcomes_shared: false`; a per-learner activity model, without which "last +active" and the engagement counters are null; and learner removal on +`mv_b2b_learner`, without which `is_current` is always true. + +Blocking the first partner: the per-contract Keycloak client template and a +bearer-only gateway route. No partner can authenticate without both. diff --git a/docs/b2b-learner-records-provider-authorization.md b/docs/b2b-learner-records-provider-authorization.md index 4de1a4a..fdcc3ef 100644 --- a/docs/b2b-learner-records-provider-authorization.md +++ b/docs/b2b-learner-records-provider-authorization.md @@ -15,6 +15,12 @@ issues client credentials that encode those terms. The partner builds the integr its own LMS and handles per-user authorization on its side. MIT does not model the partner's users, and a credential reads everything its contract covers. +Access is organization-wide. A client reads every learner under every contract +of each organization it lists, including learners on contracts another +provider serves. `contract_id` on the API is a filter the caller chooses, not a +limit on the credential. The contract is with the organization, and it grants +access to the organization's learners, not to one contract's cohort. + ## How the terms are encoded One Keycloak client per contracted integration, defined in Pulumi @@ -24,30 +30,38 @@ that creates the client is MIT's record of the access. | Contract term | On the client | | --- | --- | -| Which organizations | A hardcoded claim (working name `learner_records_organizations`) listing the Keycloak organization UUIDs, via `keycloak.openid.HardcodedClaimProtocolMapper` | +| Which organizations | A hardcoded claim (working name `learner_records_organizations`) listing the Keycloak organization UUIDs, via `keycloak.openid.HardcodedClaimProtocolMapper` with `jsonType.label` set to `JSON`, so the claim arrives as an array rather than a string | | Read access | Default client scope `learner-records:read`. There is no separate identity scope; see Consequences | | API audience | `keycloak.openid.AudienceProtocolMapper`, as the Superset client already uses | | Client-credentials only | `service_accounts_enabled=True`, standard flow and direct grants off | A provider working for two organizations under two contracts holds two clients. -When a contract ends, removing its client revokes exactly that access, which -keeps the property the design already required. +When a contract ends, removing its client revokes that credential. Any other +client listing the same organization keeps reading the same rows, so ending one +provider's access means removing that provider's clients, not the +organization's. + +**Open, for pdpinch: does access expire with the contract?** Today it doesn't. +The per-request steps below check no end date, so a client whose cleanup PR is +forgotten keeps issuing tokens. Two ways to enforce it: -Recommended (it costs one more claim): carry the contract end date as a claim -too, and have the API refuse tokens once that date has passed. A forgotten -cleanup PR then fails closed. +- Carry the contract end date as a claim, and refuse tokens past it. +- Check the warehouse instead. `dim_contract` already has `contract_is_active` + and `contract_end_date`, and the API serves both on `/courses`. ## What the API does In `tenants/b2b_learner_records/auth.py`, per request: -1. Read the organization claim and scopes from the validated token. +1. Read the organization claim and scopes from the validated token. A missing + claim, or one that isn't an array of UUIDs, lists no organizations, as + `b2b_dashboard/auth.py` already treats a malformed organization claim. 2. If the path's `organization_id` is not in the claim, return the existing 403, identical to the response for an organization that does not exist. 3. Require the `learner-records:read` scope. Identity fields are always populated. -No call to mitxonline or any other service, and no grant store. The client +No call to mitxonline or any other service, and no access store. The client definition is the only place access is recorded. ## Consequences @@ -60,22 +74,26 @@ definition is the only place access is recorded. - **Revocation is a Pulumi change.** Tokens issued before the client is removed stay valid until they expire. The access-token lifespan for these clients bounds that window. It has not been checked for the `olapps` realm yet. -- **Exports need to be covered by the same credential.** The draft says export - files are "readable with the credentials issued alongside the API client". - Standing storage credentials would be a second thing to revoke when a - contract ends. Recommended: `/exports` returns short-lived presigned HTTPS - URLs, so the API client is the only credential. Not yet applied to the - OpenAPI draft. +- **Exports are written out of band and reached three ways.** A batch job + writes each organization's export to its own prefix in S3. A partner reads it + through a cross-account IAM role limited to that organization's prefix (no + `ListBucket` beyond it), over SFTP, or through `/exports`, which returns a + short-lived presigned HTTPS URL to the same file. The IAM role and SFTP + account are credentials separate from the API client, so ending a contract + means revoking those too. `/exports` needs only the API client. - **The partner owns end-user access.** Obligations on how the partner restricts learner records inside its LMS belong in the contract. Nothing on MIT's side enforces them. +- **The contract is the organization's record of who reads its data.** The + organization signed the contract that names the provider, so it already + knows. No organization-facing view of clients is planned. ## Not chosen -- **A grant record in mitxonline, checked at request time,** with - organization managers able to see and revoke grants. That adds a runtime +- **An access record in mitxonline, checked at request time,** with + organization managers able to see and revoke access. That adds a runtime dependency and a UI, but the decision it would store is already made in the - contract. + contract, which is also where the organization sees it. - **Making the service account a member of the Keycloak organization,** so the existing organization-membership mapper lists it. mitxonline's org sync would import the service account as a learner, and it would then appear in @@ -89,6 +107,26 @@ definition is the only place access is recorded. - Whether APISIX's `openid-connect` plugin passes the hardcoded claim and scopes through in `X-Userinfo` on a bearer-only route. The learner-records mount needs a bearer-only route, but today's routes use the redirect flow. - Check on QA before writing `auth.py`. + Check on QA. +- That APISIX *overwrites* a caller-supplied `X-Userinfo` rather than passing + it through. `core/auth/userinfo.py` decodes whatever header arrives without + validating a token, and the organization check reads from it. Also confirm + the pod can't be reached except through the gateway route: `k8s/` defines no + NetworkPolicy. Check both on QA. - The access-token lifespan these clients will get, since it is the revocation window. + +## Follow-ups + +Not needed to write the tenant, but each needs an owner before partners +multiply: + +- **Client lifecycle.** One client per contract covers creation only. Who + delivers the secret to the partner, who rotates it, and who removes it when + the contract ends are unassigned. +- **Audit logging, before the first partner.** `auth.py` logs the client ID and + organization on every authorized request. It doesn't log how many records + were returned, and nothing yet says where those logs are kept or who reviews + them. +- **Rate limits.** The contract documents a 429 but no limits, and hasn't + decided whether APISIX or the app enforces them. diff --git a/docs/openapi/b2b-learner-records-v1.yaml b/docs/openapi/b2b-learner-records-v1.yaml index fc918fb..3f49106 100644 --- a/docs/openapi/b2b-learner-records-v1.yaml +++ b/docs/openapi/b2b-learner-records-v1.yaml @@ -19,18 +19,25 @@ info: activity fields are populated only where the learner has opted in to sharing their course status. `outcomes_shared` says which. Identity and enrollment facts are not consent-gated — the organization assigned the - seat. Until the upstream consent field ships, a learner with no - recorded decision reads `outcomes_shared: false` where enforcement fails - closed (the default) and `outcomes_shared: true` where the deployment - fails open. + seat. Enforcement fails closed, so every record reads + `outcomes_shared: false` until the upstream consent field ships. * **The same schemas are delivered two ways.** This REST API, and a - per-organization bulk export (S3/SFTP) written on each refresh. The export - carries the same records under the same field names. - * **`as_of` is the data's freshness and the sync cursor.** Records come from - materialized views refreshed on a schedule. Pass a previous `as_of` as - `updated_since` for incremental sync; a consent withdrawal arrives as a - normal changed record with `outcomes_shared: false` and outcome fields - nulled, which the client upserts to drop the data it held. + per-organization bulk export written to S3 on each refresh. The export + carries the same records under the same field names. It is readable + through a cross-account IAM role, over SFTP, or from the presigned URL + `/exports` returns. + * **`as_of` is the data's freshness.** Records come from materialized views + refreshed on a schedule. For incremental sync, pass the previous sync's + `as_of` as `updated_since` together with `include_inactive=true`. A + consent withdrawal arrives as a changed record with + `outcomes_shared: false` and outcome fields nulled, a deactivated + enrollment with `enrollment_is_active: false`, and a learner who left the + organization with `is_current: false`. A client that upserts drops what + it held. Activity changes don't arrive on sync; reload in full + periodically. + * **Paging is stable within one refresh.** Every collection has a unique + order, but a refresh between two pages can shift rows across the offset. + If `as_of` changes between pages, restart from offset 0. Every property of the `Learner`, `Enrollment` and `CourseRun` record schemas carries `x-data-readiness`: `available` (backed today), @@ -45,9 +52,9 @@ info: identifier: BSD-3-Clause servers: - - url: https://api.mitxonline.mit.edu/api/v1/learner-records + - url: https://analytics.ol.mit.edu/api/v1/learner-records description: Production (planned) - - url: https://api.qa.mitxonline.mit.edu/api/v1/learner-records + - url: https://analytics-qa.ol.mit.edu/api/v1/learner-records description: QA — partner integration testing (planned) security: @@ -107,6 +114,7 @@ paths: organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 organization_name: Contoso Manufacturing membership_source: both + is_current: true is_organization_manager: false first_enrolled_on: '2026-02-03T14:22:11Z' last_enrolled_on: '2026-05-19T09:04:52Z' @@ -124,6 +132,7 @@ paths: organization_id: 8f14e45f-ceea-467a-9c1b-2f4b9c0a3d21 organization_name: Contoso Manufacturing membership_source: both + is_current: true is_organization_manager: false first_enrolled_on: '2026-03-11T10:02:44Z' last_enrolled_on: '2026-03-11T10:02:44Z' @@ -297,8 +306,12 @@ paths: operationId: listExports summary: Bulk export files available for this organization description: >- - Manifest of the S3/SFTP bulk exports, one set per collection per - refresh. Records inside carry the same schemas as this API, identity + Manifest of the bulk exports, one set per collection per refresh, each + with a short-lived presigned HTTPS URL. A batch job writes the files to + S3 outside this API; the same files are also readable through a + cross-account IAM role limited to the organization's prefix, or over + SFTP. This endpoint is the path that needs no credential beyond the API + client. Records inside carry the same schemas as this API, identity fields included. Ordered by `as_of` descending, then `collection` and `format`, which is unique. parameters: @@ -333,7 +346,8 @@ paths: record_count: 163 size_bytes: 214880 checksum_sha256: 9f2c1b7ae4d05c8831fbb2e6a0d47c3915ee8b6042d1f7c9a3b508e2d6417f0a - uri: s3://ol-b2b-exports/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz + uri: https://ol-b2b-exports.s3.amazonaws.com/8f14e45f/2026-08-13T06-15-00Z/enrollments.jsonl.gz?X-Amz-Expires=900&X-Amz-Signature=… + uri_expires_on: '2026-08-13T09:30:00Z' expires_on: '2026-09-12T06:15:00Z' '400': { $ref: '#/components/responses/BadRequest' } '401': { $ref: '#/components/responses/Unauthorized' } @@ -349,12 +363,12 @@ components: description: >- Keycloak client credentials, validated at the gateway. One client is issued per contracted integration and lists the organizations it may - read. Scopes - bound to a client are a contractual limit on that credential; they are - not the consent mechanism. + read, every contract under each included. Scopes bound to a client are + a contractual limit on that credential; they are not the consent + mechanism. flows: clientCredentials: - tokenUrl: https://sso.mit.edu/realms/olapps/protocol/openid-connect/token + tokenUrl: https://sso.ol.mit.edu/realms/olapps/protocol/openid-connect/token scopes: learner-records:read: Read records, identity fields included. @@ -372,6 +386,13 @@ components: name: contract_id in: query required: false + description: >- + Narrows the result to one of the organization's contracts. A filter, + not an access boundary: a client reads every contract under the + organizations it lists. A contract not under the path's organization + matches nothing and returns an empty page, not a 403; the result is the + same whether the contract belongs to another organization or doesn't + exist. schema: { type: integer, format: int64 } example: 42 @@ -413,7 +434,14 @@ components: name: updated_since in: query required: false - description: Return only records changed at or after this instant. Pass the previous sync's `as_of`. + description: >- + Return only records changed at or after this instant. Pass the previous + sync's `as_of`. A record's change time is the latest of its enrollment, + grade, certificate, organization membership and (once it exists) + consent changes. Activity doesn't move it. Deactivated enrollments and + learners no longer in the organization are changes too, but + `include_inactive` still filters them, so pass + `include_inactive=true` when syncing. schema: { type: string, format: date-time } example: '2026-08-12T06:15:00Z' @@ -421,7 +449,14 @@ components: name: include_inactive in: query required: false - description: Include deactivated enrollments (unenrolled, refunded, transferred). + description: >- + On `/enrollments`, includes deactivated enrollments (unenrolled, + refunded, transferred). On `/learners`: rollups count deactivated + enrollments too; learners whose only enrollments are deactivated are + included; learners with `is_current: false` are included; and a roster + member whose only enrollment is deactivated reads + `membership_source: both` rather than `roster`. Roster members with no + enrollments at all are included either way. schema: { type: boolean, default: false } Limit: @@ -434,6 +469,10 @@ components: name: offset in: query required: false + description: >- + Stable only within one refresh. If the envelope's `as_of` differs from + the previous page's, restart from 0; otherwise rows can be skipped or + repeated. schema: { type: integer, minimum: 0, default: 0 } schemas: @@ -476,6 +515,7 @@ components: - organization_id - organization_name - membership_source + - is_current - is_organization_manager - first_enrolled_on - last_enrolled_on @@ -511,6 +551,16 @@ components: x-data-readiness: available membership_source: $ref: '#/components/schemas/MembershipSource' + is_current: + type: boolean + description: >- + False once the learner is no longer associated with the + organization: off the roster and holding no enrollment under its + contracts. The record stays so a syncing client learns of the + removal; it is returned only with `include_inactive=true`. + Identity is still populated. Erasure requests are handled + separately and not yet specified. + x-data-readiness: pending-model is_organization_manager: type: boolean description: Administers the organization in MITx Online. @@ -543,7 +593,7 @@ components: last_active_on: type: [string, 'null'] format: date - description: Most recent day with recorded course activity. A date, not a timestamp — activity is aggregated per day. + description: Most recent day with recorded course activity. A date, not a timestamp — activity is aggregated per day. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model courses_in_progress: type: [integer, 'null'] @@ -560,7 +610,12 @@ components: certificates_earned: type: [integer, 'null'] minimum: 0 - description: Includes program certificates, which have no course run and so are not in `courses_certified`. + description: >- + Includes program certificates, which have no course run and so are + not in `courses_certified`. This is the only program-level figure in + v1; program progress is otherwise out of scope, so nothing else in + the payload reconciles against it. With `contract_id`, program + certificates belong to no single contract and are excluded. x-data-readiness: derived Enrollment: @@ -684,31 +739,35 @@ components: x-data-readiness: available certificate_is_revoked: type: [boolean, 'null'] - description: Null where no certificate exists. A revoked certificate leaves `completion_status` at `passed`. + description: >- + Null where no certificate exists. A revoked certificate doesn't + count as certified: `completion_status` then follows the grade, so + it can read `passed`, `in_progress` or `not_started`. x-data-readiness: available last_active_on: type: [string, 'null'] format: date - description: Most recent day with recorded activity in this course run. + description: Most recent day with recorded activity in this course run. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model days_active: type: [integer, 'null'] minimum: 0 - description: Distinct days with recorded activity in this course run. + description: Distinct days with recorded activity in this course run. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model videos_watched: type: [integer, 'null'] minimum: 0 - description: Distinct video blocks played. + description: Distinct video blocks played. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model problems_attempted: type: [integer, 'null'] minimum: 0 - description: Distinct problem blocks attempted. + description: Distinct problem blocks attempted. Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model chatbot_interactions: type: [integer, 'null'] minimum: 0 + description: Changes to it do not move `updated_since`; a full reload picks them up. x-data-readiness: pending-model CourseRun: @@ -760,7 +819,7 @@ components: ExportFile: type: object description: One bulk-export file. Contents use this API's record schemas. - required: [collection, as_of, format, record_count, size_bytes, checksum_sha256, uri, expires_on] + required: [collection, as_of, format, record_count, size_bytes, checksum_sha256, uri, uri_expires_on, expires_on] properties: collection: type: string @@ -771,6 +830,9 @@ components: description: The refresh this file was written from. format: type: string + description: >- + In `csv`, null is an empty unquoted field and an empty string is + `""`. `jsonl` and `parquet` carry null natively. enum: [jsonl, csv, parquet] record_count: type: integer @@ -784,11 +846,20 @@ components: uri: type: string format: uri - description: '`s3://` or `sftp://` location, readable with the credentials issued alongside the API client.' + description: >- + Presigned HTTPS URL for the file. Needs no credential beyond itself; + valid until `uri_expires_on`. List the manifest again for a fresh + URL. + uri_expires_on: + type: string + format: date-time + description: When `uri` stops working. The file itself remains until `expires_on`. expires_on: type: string format: date-time - description: After this, the file is deleted and must be re-fetched from a newer refresh. + description: >- + File retention. After this, the file is deleted and must be + re-fetched from a newer refresh. MembershipSource: type: string @@ -810,35 +881,54 @@ components: Error: type: object - required: [detail] + required: [code, detail] properties: - detail: { type: string } + code: + type: string + description: >- + Stable, machine-readable reason. Branch on this, not on `detail`, + whose wording may change. + enum: + - invalid_parameter + - unauthorized + - missing_scope + - no_organization_access + - rate_limited + - unavailable + detail: + type: string + description: Human-readable explanation. responses: BadRequest: - description: Malformed parameter. + description: >- + Malformed parameter, including a non-UUID `organization_id` in the path. + Deliberately 400, not FastAPI's default 422: the service normalizes + validation errors to this body. content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'limit must be between 1 and 1000' } + example: { code: invalid_parameter, detail: 'limit: Input should be less than or equal to 1000' } Unauthorized: description: Missing, malformed or expired token. content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'Invalid or expired access token' } + example: { code: unauthorized, detail: 'Invalid or expired access token' } Forbidden: description: >- - Token valid but lacks a grant for this organization, or lacks the - required scope. Identical whether the organization is ungranted or does - not exist, so this cannot be used to enumerate organizations. + Token valid but lacks access to this organization + (`no_organization_access`) or lacks the required scope + (`missing_scope`). Identical whether the client lacks access or the + organization does not exist, so this cannot be used to enumerate + organizations. content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'No grant for the requested organization' } + example: { code: no_organization_access, detail: 'No access to the requested organization' } TooManyRequests: description: Per-client rate limit exceeded. @@ -848,7 +938,7 @@ components: content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'Rate limit exceeded' } + example: { code: rate_limited, detail: 'Rate limit exceeded' } ServiceUnavailable: description: Analytics store unreachable or saturated. @@ -858,4 +948,4 @@ components: content: application/json: schema: { $ref: '#/components/schemas/Error' } - example: { detail: 'Analytics store unavailable' } + example: { code: unavailable, detail: 'Analytics store unavailable' }