diff --git a/CHANGELOG.md b/CHANGELOG.md index 506b73aa4..e3c33eaba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ All notable changes to UnoSim. Versions follow [Semantic Versioning](https://semver.org/). +## Unreleased + +### Fixed + +- **Tutor keeps answering while GitHub is unavailable.** When the Course + Content source cannot be refreshed (outage, rate limit or load limit), Tutor + questions and dialogs fall back to the free Tutor without a Course Content + session instead of failing with `INVALID_REQUEST` (Course Content SSOT §11, + case G). + ## v1.0.0 – 2026-10-09 First versioned release of UnoSim. It covers everything merged since the last @@ -25,7 +35,7 @@ stays at `1.4.0`. - **Capacity:** the application default admits 25 running plus waiting simulations, at most 5 per subject. The Ubuntu/VBox installer writes the value measured on its reference VM, `SIMULATION_ADMISSION_MAX=40`. Read - [CAPACITY_VALIDATION_PLAN.md](docs/CAPACITY_VALIDATION_PLAN.md) before + [CAPACITY_MODEL.md](docs/CAPACITY_MODEL.md) before raising it on other hosts. - **New lifetime limits:** a sandbox ends after `SANDBOX_MAX_LIFETIME_SECONDS` (default 7200 s) and a run after `SIMULATION_MAX_PAUSED_SECONDS` of total diff --git a/README.md b/README.md index f49feb91c..6d8e79b84 100644 --- a/README.md +++ b/README.md @@ -180,7 +180,7 @@ then share an UnoSim identity and its limits. This IP-identified mode grants access to every client that can reach the proxy and is not user authentication. See [the deployment security contract](docs/SECURITY.md) and the [Docker installation guide](docs/INSTALL_SERVER.md). For a tested -Ubuntu Desktop setup, see [README_UbuntuServer.md](README_UbuntuServer.md). +Ubuntu Desktop setup, see [docs/INSTALL_UBUNTU_VBOX.md](docs/INSTALL_UBUNTU_VBOX.md). Sandbox execution remains dynamic and uses the Docker socket at runtime. If you need SonarQube, run it separately in its own stack or service; the UnoSim compose file does not include SonarQube or MCP. @@ -211,12 +211,12 @@ On macOS, make sure your project directory is allowed under Docker Desktop file | `RUN_HEAVY_TESTS=1 ./run-tests.sh` | Full pipeline inklusive langsamem Parallel-Instanz-Stresstest | For host-specific Capacity sizing, use `npm run capacity:calibrate` and follow -the review workflow in [`docs/CAPACITY_VALIDATION_PLAN.md`](docs/CAPACITY_VALIDATION_PLAN.md). +the review workflow in [`docs/CAPACITY_MODEL.md`](docs/CAPACITY_MODEL.md). ### Architecture Overview - **Sandbox Runner Pool** — Manages runner leases for sketch execution. In the documented production path, each simulation runs in a short-lived isolated Docker sandbox with stdout/stderr capture for serial output and pin state reporting. -- **Compilation Worker Pool** — In production mode, the configured Node.js Worker Thread pool handles compilations in parallel via `CompilerWithFallback`. The Compose reference uses 8 workers (the pool runs at most 8); a full pool answers `503 SYSTEM_BUSY`. Current capacity guidance is in [`docs/CAPACITY_VALIDATION_PLAN.md`](docs/CAPACITY_VALIDATION_PLAN.md); [`docs/SCALABILITY.md`](docs/SCALABILITY.md) keeps the historical measurement. +- **Compilation Worker Pool** — In production mode, the configured Node.js Worker Thread pool handles compilations in parallel via `CompilerWithFallback`. The Compose reference uses 8 workers (the pool runs at most 8); a full pool answers `503 SYSTEM_BUSY`. Current capacity guidance is in [`docs/CAPACITY_MODEL.md`](docs/CAPACITY_MODEL.md); [`docs/CAPACITY_HISTORICAL_PHASE_3_4.md`](docs/CAPACITY_HISTORICAL_PHASE_3_4.md) keeps the historical measurement. - **WebSocket Layer** — Real-time communication between client and server for serial output, pin state batches, and simulation control (start/stop/pause/resume). - **SonarQube Integration** — Optional SonarQube scans are wired into the pre-push hook and `./run-tests.sh` when `SONAR_TOKEN` and a reachable SonarQube service are available. Release blocking is controlled by `REQUIRE_RELEASE_GATE=1`. @@ -234,7 +234,7 @@ The repository contains a **robust, fast test pipeline**: 3. Heavy stress tests are opt-in. Use `RUN_HEAVY_TESTS=1 ./run-tests.sh` when validating Docker sandbox isolation and cleanup. Load-test commands are listed in [`docs/RELEASE_RUNBOOK.md`](docs/RELEASE_RUNBOOK.md) and capacity - conclusions belong in [`docs/CAPACITY_VALIDATION_PLAN.md`](docs/CAPACITY_VALIDATION_PLAN.md). + conclusions belong in [`docs/CAPACITY_MODEL.md`](docs/CAPACITY_MODEL.md). Local quick‑check example: diff --git a/docker-compose.yml b/docker-compose.yml index bb675881d..de617bebc 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -56,7 +56,7 @@ services: # Logical simulation capacity grows on-demand; idle logical runners shrink after IDLE_TIMEOUT_MS. # Admission caps running plus waiting simulations. Without a host-specific value it # stays at the application default 25; the Ubuntu/VBox installer writes the value - # measured on its reference VM. See docs/CAPACITY_VALIDATION_PLAN.md before raising. + # measured on its reference VM. See docs/CAPACITY_MODEL.md before raising. - SIMULATION_ADMISSION_MAX=${SIMULATION_ADMISSION_MAX:-25} - SIMULATION_MAX_CONCURRENT=200 # Total paused time of one run; a paused run still holds its capacity. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 4d8d570ec..f6e7b9e73 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -393,5 +393,5 @@ Diese Metriken sind über `/api/status` und WebSocket-Events verfügbar. `/api/h - [`adr/0006-unified-course-content-and-tutor-strategy.md`](adr/0006-unified-course-content-and-tutor-strategy.md) – aktueller einheitlicher Course-Content-/Tutor-Vertrag - [`../ssot/ssot_function_definition_CourseContent.md`](../ssot/ssot_function_definition_CourseContent.md) – Normativer Course-Content-Vertrag - [`../ssot/ssot_function_definition_ExternalExamples.md`](../ssot/ssot_function_definition_ExternalExamples.md) – Fachlicher Examples-Vertrag -- [`SCALABILITY.md`](SCALABILITY.md) – historische Kapazitätsmessung; aktuelle Werte in [`CAPACITY_VALIDATION_PLAN.md`](CAPACITY_VALIDATION_PLAN.md) +- [`CAPACITY_HISTORICAL_PHASE_3_4.md`](CAPACITY_HISTORICAL_PHASE_3_4.md) – historische Kapazitätsmessung; aktuelle Werte in [`CAPACITY_MODEL.md`](CAPACITY_MODEL.md) - [`TESTING_STANDARDS.md`](TESTING_STANDARDS.md) – Teststrategie und -konventionen diff --git a/docs/SCALABILITY.md b/docs/CAPACITY_HISTORICAL_PHASE_3_4.md similarity index 98% rename from docs/SCALABILITY.md rename to docs/CAPACITY_HISTORICAL_PHASE_3_4.md index 847e45301..eea90c064 100644 --- a/docs/SCALABILITY.md +++ b/docs/CAPACITY_HISTORICAL_PHASE_3_4.md @@ -4,7 +4,7 @@ Dieses Dokument bewahrt die Messung aus Phase 3.4 vor dem aktuellen Kapazitätsmodell. Es ist keine normative Aussage über die heutigen Produktionsgrenzen. Das aktuelle Modell, die aktuellen Defaults und die Dell- Referenzmessungen stehen in -[CAPACITY_VALIDATION_PLAN.md](CAPACITY_VALIDATION_PLAN.md). +[CAPACITY_MODEL.md](CAPACITY_MODEL.md). ## Gemessene Grenzen diff --git a/docs/MIXED_CAPACITY_TESTING.md b/docs/CAPACITY_MIXED_LOAD_TESTING.md similarity index 100% rename from docs/MIXED_CAPACITY_TESTING.md rename to docs/CAPACITY_MIXED_LOAD_TESTING.md diff --git a/docs/CAPACITY_VALIDATION_PLAN.md b/docs/CAPACITY_MODEL.md similarity index 99% rename from docs/CAPACITY_VALIDATION_PLAN.md rename to docs/CAPACITY_MODEL.md index 47be1aae8..4cc0d1bea 100644 --- a/docs/CAPACITY_VALIDATION_PLAN.md +++ b/docs/CAPACITY_MODEL.md @@ -1,4 +1,4 @@ -# Capacity validation plan and operating model +# Capacity model and validation This is the authoritative description of UnoSim's simulation capacity model, its timeout boundaries, and the validation evidence. Validation profiles are diff --git a/docs/CAPACITY_TEST_RESULTS.md b/docs/CAPACITY_TEST_RESULTS.md index e90e4f5ae..976cfdf9f 100644 --- a/docs/CAPACITY_TEST_RESULTS.md +++ b/docs/CAPACITY_TEST_RESULTS.md @@ -2,7 +2,7 @@ This document summarizes the measured reference results. The operating model, timeout boundaries, production defaults, and acceptance procedure are -authoritative in CAPACITY_VALIDATION_PLAN.md. +authoritative in CAPACITY_MODEL.md. ## Historical harness baseline @@ -35,7 +35,7 @@ That run used 80 sandbox-start slots. On current `main` with 8 start slots and the 30 s start-slot timeout, a synchronous 80-burst started only 64; the rest timed out waiting for a start slot. A 43 s timeout or 16 start slots brought it back to 80/80. See the 2026-10-09 delta check in -CAPACITY_VALIDATION_PLAN.md. +CAPACITY_MODEL.md. ## Dell classroom validation diff --git a/docs/INSTALL_SERVER.md b/docs/INSTALL_SERVER.md index c43dd409f..16ee6be01 100644 --- a/docs/INSTALL_SERVER.md +++ b/docs/INSTALL_SERVER.md @@ -35,7 +35,7 @@ Gateway-Identität und Docker-Simulation automatisch. Der Server prüft Docker-Daemon, Sandbox-Image und Runner-Pool vor der Readiness-Freigabe. Die vollständige Kapazitäts- und Timeout-Semantik steht in -[`CAPACITY_VALIDATION_PLAN.md`](CAPACITY_VALIDATION_PLAN.md). Die dortigen +[`CAPACITY_MODEL.md`](CAPACITY_MODEL.md). Die dortigen Validierungsprofile sind Testprofile und keine automatischen Produktionsvorgaben. @@ -166,7 +166,7 @@ zusammen mit `SIMULATION_ADMISSION_MAX`: Ohne diesen Wert lässt die Admission höchstens 25 laufende und wartende Simulationen zu. Der Ubuntu/VBox-Installer schreibt `SIMULATION_ADMISSION_MAX=40` in die `.env`; dieser Wert ist auf der Referenz-VBox (8 vCPU, 11 GiB, kein Swap) für Klassen von etwa 30 Studierenden -gemessen (siehe `CAPACITY_VALIDATION_PLAN.md`). Auf anderer Hardware vor einer +gemessen (siehe `CAPACITY_MODEL.md`). Auf anderer Hardware vor einer Übernahme erneut messen. Ist der Compile-Worker-Pool ausgelastet (500 wartende Compiles oder 30 s @@ -185,7 +185,7 @@ ist nur ein Review-Vorschlag; fehlende Werte müssen geprüft und ausdrücklich aufgelöst werden. Die Kalibrierung ändert keine Produktionskonfiguration automatisch. Die technische Referenz mit Messphasen, Timeout-Grenzen und Akzeptanzkriterien steht in -[`CAPACITY_VALIDATION_PLAN.md`](CAPACITY_VALIDATION_PLAN.md). +[`CAPACITY_MODEL.md`](CAPACITY_MODEL.md). Frühere Topologie- und Kompatibilitätsschalter werden beim Start abgelehnt: `UNOSIM_SIMULATION_MODE`, `UNOSIM_TRUST_MODE`, `FORCE_DOCKER` und diff --git a/README_UbuntuServer.md b/docs/INSTALL_UBUNTU_VBOX.md similarity index 98% rename from README_UbuntuServer.md rename to docs/INSTALL_UBUNTU_VBOX.md index 831217285..44ceec503 100644 --- a/README_UbuntuServer.md +++ b/docs/INSTALL_UBUNTU_VBOX.md @@ -417,6 +417,6 @@ https://127.0.0.1 https://127.0.0.1:8443 ``` -Weitere Details stehen in [docs/INSTALL_SERVER.md](docs/INSTALL_SERVER.md), -[docs/SECURITY.md](docs/SECURITY.md) und -[ADR 0008](docs/adr/0008-optional-gateway-authentication.md). +Weitere Details stehen in [INSTALL_SERVER.md](INSTALL_SERVER.md), +[SECURITY.md](SECURITY.md) und +[ADR 0008](adr/0008-optional-gateway-authentication.md). diff --git a/docs/OPEN_POINTS.md b/docs/OPEN_POINTS.md new file mode 100644 index 000000000..2d0cb4613 --- /dev/null +++ b/docs/OPEN_POINTS.md @@ -0,0 +1,36 @@ +# UnoSim – Offene Punkte (OPL) + +> **Nicht normativ.** Enthält nur offene Punkte. Jeder Punkt wurde am +> 2026-10-11 gegen `main` @ `4dcc1ae9` geprüft. Erledigte Punkte werden aus +> dieser Liste entfernt; die abgeschlossene Refactoring-Serie mit Verlauf steht +> in [archive/refactoring/REFACTORING_OPL.md](archive/refactoring/REFACTORING_OPL.md). +> Normative Verträge bleiben ADRs und SSOTs. + +Kein offener Punkt ist mit Risiko „hoch“ bewertet. + +Statuswerte: `BLOCKED_DECISION` (wartet auf eine Entscheidung) · `DEFERRED` +(bewusst zurückgestellt) · `OPEN` (entschieden, noch nicht umgesetzt) + +## Sicherheit und Verträge + +| ID | Befund | Risiko | Beleg | Status | Nötige Entscheidung | +|---|---|---|---|---|---| +| R10-PROXY | ADR 0001 §51 verlangt, dass das Backend nur von `UNOSIM_TRUSTED_PROXY` erreichbar ist. Der Code validiert die Einstellung nur und setzt damit `trust proxy`; die Quell-IP eingehender Verbindungen prüft er nicht. | mittel | [access-control.ts:135-142](../server/security/access-control.ts#L135-L142), [index.ts:53](../server/index.ts#L53) | BLOCKED_DECISION | Quell-IP-Prüfung im Code (Rollout-Risiko bei falsch gesetztem Wert) oder ADR als Betreiberpflicht formulieren | +| S1-ASM | Die GAS-Direktive `.include` im Inline-Assembler gibt die ersten Zeichen jeder Zeile einer lesbaren Datei als Fehlermeldung aus. Ein Textfilter lässt sich per String-Konkatenation umgehen. Als Restrisiko in [SECURITY.md](SECURITY.md) dokumentiert. | mittel | Marker-Präfix in der Assemblermeldung reproduziert | BLOCKED_DECISION | REST-Compile ohne Zugriff auf Backend-Dateien (Sandbox/Namespace) – Architekturentscheidung | +| R10-HANDSHAKE | [EXTERNAL_API.md](EXTERNAL_API.md) beschreibt einen initialen `handshake` mit `protocolVersion`. Der Server sendet ihn nur, wenn der Client eine `testRunId` mitschickt. | gering | [simulation.ws.ts:889-895](../server/routes/simulation.ws.ts#L889-L895) | BLOCKED_DECISION | Handshake immer senden (Schemaänderung, `testRunId` optional) oder Vertragstext anpassen | + +## Produktentscheidungen + +| ID | Befund | Risiko | Beleg | Status | Nötige Entscheidung | +|---|---|---|---|---|---| +| T3 | Vom Client gesendete `answerRating`-Werte zählen als Mastery-Evidenz, solange für die Frage keine gespeicherte Evidenz existiert. Wirkt nur auf den eigenen Lernstand, widerspricht aber der deterministischen Mastery aus ADR 0007. | gering | [curriculum-tutor-adapter.ts:493-515](../server/services/tutor/curriculum-tutor-adapter.ts#L493-L515), [learning-planner.ts:193-200](../server/services/tutor/curriculum/learning-planner.ts#L193-L200) | BLOCKED_DECISION | Dürfen Client-Ratings Mastery beeinflussen? | +| D1 | LEARN nach schwacher oder mittlerer Antwort, wenn das aktuelle Concept keine frische Frage mehr hat, das Topic aber noch andere Fragen bietet. Die SSOT regelt nur den Fall, dass das ganze Topic nicht mehr abfragbar ist (`content-exhausted`). | gering | [LearningQuestions §Zustandsübergänge](../ssot/ssot_function_definition_LearningQuestions.md) | BLOCKED_DECISION | Gewünschtes Tutor-Verhalten festlegen, dann SSOT ergänzen | +| D4 | Tutor-Sessions laufen nach einer absoluten TTL ab (Default 1 h, per `TUTOR_SESSION_TTL_MS` einstellbar), auch bei laufender Nutzung. | gering | [config.ts:412](../server/config.ts#L412), [course-content-session.ts:107](../server/services/course-content/course-content-session.ts#L107) | BLOCKED_DECISION | Absolute oder gleitende TTL? | +| D5 | Ein laufender Sketch hält seinen Simulationsslot bis zum Lauf-Timeout (Default 60 s, Client-Maximum 300 s). Pausierte Läufe enden nach `SIMULATION_MAX_PAUSED_SECONDS` (600 s), jeder Container nach `SANDBOX_MAX_LIFETIME_SECONDS` (2 h). Einen Inaktivitäts-Timeout für laufende Simulationen gibt es nicht. | gering | [input-limits.ts:40-42](../shared/input-limits.ts#L40-L42), [config.ts:246-247](../server/config.ts#L246-L247) | BLOCKED_DECISION | Reichen die vorhandenen Grenzen, oder soll eine Simulation ohne Nutzerinteraktion früher enden? | + +## Wartung + +| ID | Befund | Risiko | Beleg | Status | Nötige Entscheidung | +|---|---|---|---|---|---| +| A2-DEAD | Die Cache-Lock-API (`acquireCacheLock`) und `drain()` im Gatekeeper werden im Produktionscode nicht genutzt; 48 Testreferenzen hängen daran. | gering | [unified-gatekeeper.ts:259](../server/services/unified-gatekeeper.ts#L259), [unified-gatekeeper.ts:502](../server/services/unified-gatekeeper.ts#L502) | DEFERRED | Entfernen samt Tests (Teständerung braucht Freigabe) | +| R10-SSOTREF | Die SSOT TutorQualityEvaluation verweist auf `ssot_function_tutor_model_registration.md`, die nicht im Repository liegt. | gering | [TutorQualityEvaluation:23](../ssot/ssot_function_definition_TutorQualityEvaluation.md) | DEFERRED | Datei versionieren oder Verweis entfernen (Owner) | diff --git a/docs/README.md b/docs/README.md index cd85105db..a3e6ec360 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,7 +11,7 @@ und Messläufe, keinen Vertrag. Kompilierung und Simulation. - [INSTALL_SERVER.md](INSTALL_SERVER.md) – Docker-Deployment mit erforderlichem Gateway, optionaler Benutzeranmeldung und Docker-Sandboxen. -- [../README_UbuntuServer.md](../README_UbuntuServer.md) – Ubuntu-Desktop-Setup +- [INSTALL_UBUNTU_VBOX.md](INSTALL_UBUNTU_VBOX.md) – Ubuntu-Desktop-Setup in VirtualBox mit Docker, Nginx, TLS und optionaler Benutzeranmeldung. - [SECURITY.md](SECURITY.md) – Sicherheits- und Isolationsvertrag. - [RELEASE_RUNBOOK.md](RELEASE_RUNBOOK.md) – Release-Gates und Betriebskontrollen. @@ -22,8 +22,19 @@ und Messläufe, keinen Vertrag. Runtime-Profile. - [EXTERNAL_API.md](EXTERNAL_API.md) – versionierter iframe-/postMessage-Vertrag. - [TESTING_STANDARDS.md](TESTING_STANDARDS.md) – Testklassen und Qualitätsgates. -- [SCALABILITY.md](SCALABILITY.md) – historische Kapazitätsmessung; aktuelle - Werte und Kalibrierung in [CAPACITY_VALIDATION_PLAN.md](CAPACITY_VALIDATION_PLAN.md). + +## Kapazität + +- [CAPACITY_MODEL.md](CAPACITY_MODEL.md) – maßgebliches + Kapazitätsmodell, Timeout-Grenzen und Kalibrierung. +- [CAPACITY_VALIDATION_QUICKSTART.md](CAPACITY_VALIDATION_QUICKSTART.md) – + Ablauf der Validierung auf dem Staging-Host. +- [CAPACITY_MIXED_LOAD_TESTING.md](CAPACITY_MIXED_LOAD_TESTING.md) – gemischte + Messung aus REST-Compiles, Simulationsstarts und laufenden Simulationen. +- [CAPACITY_TEST_RESULTS.md](CAPACITY_TEST_RESULTS.md) – gemessene + Referenzergebnisse. +- [CAPACITY_HISTORICAL_PHASE_3_4.md](CAPACITY_HISTORICAL_PHASE_3_4.md) – + historische Kapazitätsmessung aus Phase 3.4 (nicht normativ). ## Entscheidungen und Fachverträge @@ -32,9 +43,25 @@ und Messläufe, keinen Vertrag. Bei Änderungen zuerst die zuständige normative Quelle aktualisieren. -## Analysen und Arbeitsstand (nicht normativ) +## Offene Punkte (nicht normativ) + +- [OPEN_POINTS.md](OPEN_POINTS.md) – geprüfte offene Entscheidungen und + zurückgestellte Punkte. + +## Archiv + +- [archive/refactoring/](archive/refactoring/) – Architektur- und + Gesundheitsbericht (Stand `54cc2950`) und die daraus abgeleitete, + abgeschlossene Refactoring-Serie. +- [archive/tutor-quality/](archive/tutor-quality/) – abgeschlossene Pläne und + Messberichte zur Tutor-Qualität. Normativ bleibt + [../ssot/ssot_function_definition_TutorQualityEvaluation.md](../ssot/ssot_function_definition_TutorQualityEvaluation.md). + +Erledigte Implementierungspläne werden gelöscht, nicht archiviert; sie bleiben +in der Git-Historie. + +## Namenskonvention -- [UNOSIM_TECHNICAL_ARCHITECTURE_HEALTH_REVIEW.md](UNOSIM_TECHNICAL_ARCHITECTURE_HEALTH_REVIEW.md) - – Architektur- und Gesundheitsbericht (Stand `54cc2950`). -- [UNOSIM_REFACTORING_OPL.md](UNOSIM_REFACTORING_OPL.md) – Ergebnis der daraus - abgeleiteten Refactoring-Serie und offene Entscheidungen. +Dateien im Wurzelverzeichnis heißen `THEMA_INHALT.md` in Großbuchstaben. Das +Präfix gruppiert zusammengehörige Dokumente (`INSTALL_`, `CAPACITY_`). ADRs +liegen unter `adr/` als `NNNN-titel.md`. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 276df10eb..30e99d315 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -120,7 +120,7 @@ Remaining risk: inline-assembler directives such as `.include` can still make the assembler report the first characters of each line of a readable file, and string concatenation defeats a textual filter. Do not place secrets as files in the backend container. A sandboxed REST compile is an open decision -([refactoring OPL](UNOSIM_REFACTORING_OPL.md), S1-ASM). The process environment +([open points](OPEN_POINTS.md), S1-ASM). The process environment is not reachable this way: `/proc/*/environ` reads as empty for both compilers in use. diff --git a/docs/adr/0003-scalability-and-ha-model.md b/docs/adr/0003-scalability-and-ha-model.md index f92819e56..569365946 100644 --- a/docs/adr/0003-scalability-and-ha-model.md +++ b/docs/adr/0003-scalability-and-ha-model.md @@ -1,7 +1,7 @@ # ADR 0003: Scalability and HA model - Status: Accepted for the historical Phase 3.4 envelope; current capacity - validation is documented in ../CAPACITY_VALIDATION_PLAN.md + validation is documented in ../CAPACITY_MODEL.md - Date: 2026-09-06 - Owners: UnoSim maintainers and platform operators - Records: accepted scalability / HA decision based on the Phase 3.4 measurements @@ -11,7 +11,7 @@ Phase 3.4 measured UnoSim capacity with real Docker compile and simulation load. This ADR records that pre-refactor envelope. It must be read together with the current model and Dell reference measurements in -../CAPACITY_VALIDATION_PLAN.md; the historical runner values below are not +../CAPACITY_MODEL.md; the historical runner values below are not current operator settings. The measurement profile intentionally kept the existing production-like runner and resource configuration unchanged: diff --git a/docs/UNOSIM_TECHNICAL_ARCHITECTURE_HEALTH_REVIEW.md b/docs/archive/refactoring/ARCHITECTURE_HEALTH_REVIEW.md similarity index 76% rename from docs/UNOSIM_TECHNICAL_ARCHITECTURE_HEALTH_REVIEW.md rename to docs/archive/refactoring/ARCHITECTURE_HEALTH_REVIEW.md index 03e3399db..dca7b138e 100644 --- a/docs/UNOSIM_TECHNICAL_ARCHITECTURE_HEALTH_REVIEW.md +++ b/docs/archive/refactoring/ARCHITECTURE_HEALTH_REVIEW.md @@ -3,14 +3,14 @@ > **Nicht normativ.** Analyse-Stand nach Abschluss der Tutor-SSOT-Arbeit > (`main` @ `54cc2950`, nach #154–#158). Dieser Bericht ist die Grundlage der > anschließenden Refactoring-Serie; deren Arbeitsstand steht in -> [UNOSIM_REFACTORING_OPL.md](UNOSIM_REFACTORING_OPL.md). Normative Verträge -> bleiben die ADRs unter [adr/](adr/) und die SSOTs unter [../ssot/](../ssot/). +> [REFACTORING_OPL.md](REFACTORING_OPL.md). Normative Verträge +> bleiben die ADRs unter [adr/](../../adr/) und die SSOTs unter [../ssot/](../../../ssot/). > > Status der Befunde nach der Refactoring-Serie (bestätigt, falsifiziert, > umgesetzt, offen): siehe OPL. Dieser Bericht bleibt unverändert der > Analyse-Stand. > -> Ältere Planungsdokumente (z. B. `UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md`) +> Ältere Planungsdokumente (z. B. `docs/archive/tutor-quality/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md`) > verweisen unter demselben Dateinamen auf eine frühere, nie versionierte > Review vom 2026-09-29. Diese Datei ersetzt jene Referenz nicht inhaltlich; > sie ist der hier beschriebene neue Analyse-Stand. @@ -47,72 +47,72 @@ Im Tutor gibt es vor allem Übergangsschichten und unklare Zuständigkeiten, abe **S1. Ungeprüfter Compile im privilegierten Backend-Container: hoch [plausibel]** -- `POST /api/compile` lässt im Docker-Modus `arduino-cli` direkt im Backend-Container laufen, nicht in der Sandbox ([cli-runner.ts:60](../server/services/compiler/cli-runner.ts#L60)). Der Backend-Container hat Zugriff auf `docker.sock`. -- Der Spawn erbt die komplette Umgebung des Prozesses ([process-executor.ts:140](../server/services/process-executor.ts#L140)), also auch `UNOSIM_GATEWAY_SECRET`. -- Nicht auflösbare `#include`-Zeilen bleiben unverändert im Quelltext ([source-project.ts:349-359](../shared/source-project.ts#L349-L359)). Absolute Pfade werden nirgends abgewiesen. -- Die stderr-Ausgabe geht ungefiltert an den Client ([cli-runner.ts:133-164](../server/services/compiler/cli-runner.ts#L133-L164)). +- `POST /api/compile` lässt im Docker-Modus `arduino-cli` direkt im Backend-Container laufen, nicht in der Sandbox ([cli-runner.ts:60](../../../server/services/compiler/cli-runner.ts#L60)). Der Backend-Container hat Zugriff auf `docker.sock`. +- Der Spawn erbt die komplette Umgebung des Prozesses ([process-executor.ts:140](../../../server/services/process-executor.ts#L140)), also auch `UNOSIM_GATEWAY_SECRET`. +- Nicht auflösbare `#include`-Zeilen bleiben unverändert im Quelltext ([source-project.ts:349-359](../../../shared/source-project.ts#L349-L359)). Absolute Pfade werden nirgends abgewiesen. +- Die stderr-Ausgabe geht ungefiltert an den Client ([cli-runner.ts:133-164](../../../server/services/compiler/cli-runner.ts#L133-L164)). - Ein `#include "/proc/self/environ"` könnte deshalb das Gateway-Secret in Compiler-Diagnosen sichtbar machen. Das ist nicht ausgeführt, sollte aber in einer isolierten Umgebung verifiziert werden. - `SECURITY.md` beschreibt nur die Sandbox-Grenze der Simulation; der Compile-Pfad kommt dort nicht vor. **S2. Globaler `lastCompiledCode` bricht die Isolation zwischen Nutzern: mittel bis hoch [Code]** -- Ein `start_simulation` ohne `code` führt den zuletzt von *irgendeinem* Nutzer kompilierten Code aus ([routes.ts:143](../server/routes.ts#L143), [simulation.ws.ts:519-527](../server/routes/simulation.ws.ts#L519-L527)). +- Ein `start_simulation` ohne `code` führt den zuletzt von *irgendeinem* Nutzer kompilierten Code aus ([routes.ts:143](../../../server/routes.ts#L143), [simulation.ws.ts:519-527](../../../server/routes/simulation.ws.ts#L519-L527)). - Der aktuelle Client sendet immer Code. Der Pfad wird also nur von manipulierten oder alten Clients erreicht. - Der Isolationstest `concurrent-50-clients` läuft nur im `load`-Projekt, nicht in CI, und deckt nur den Pfad *mit* Code ab. **S3. Sketch-CRUD ohne Besitzer: mittel [Code]** -- `POST/PUT/DELETE /api/sketches` arbeiten auf einem globalen `MemStorage` ([routes.ts:177-225](../server/routes.ts#L177-L225)). -- Der Client lädt `sketches[0]` als Start-Sketch für *alle* Nutzer ([useFileSystem.ts:107-131](../client/src/hooks/useFileSystem.ts#L107-L131)). +- `POST/PUT/DELETE /api/sketches` arbeiten auf einem globalen `MemStorage` ([routes.ts:177-225](../../../server/routes.ts#L177-L225)). +- Der Client lädt `sketches[0]` als Start-Sketch für *alle* Nutzer ([useFileSystem.ts:107-131](../../../client/src/hooks/useFileSystem.ts#L107-L131)). - Jeder angemeldete Nutzer kann so den Startcode aller anderen überschreiben oder löschen. Der Client selbst nutzt nur `GET`. **S4. Race bei Runner-Wiederverwendung (fremde Ausgabe, verwaiste Container): hoch [plausibel]** - Der `ExecutionState` gehört zum Runner und wird von Lauf zu Lauf wiederverwendet. Es gibt keinen Lauf-Token und kein Abbruchsignal. - Ablauf des Problems: - 1. Nutzer A stoppt oder trennt die Verbindung, während er in der Warteschlange für einen Sandbox-Start-Slot steht ([execution-manager.ts:405-414](../server/services/sandbox/execution-manager.ts#L405-L414)). - 2. Der Runner wird zurückgesetzt ([sandbox-runner-pool.ts:381-452](../server/services/sandbox-runner-pool.ts#L381-L452)) und von B übernommen. + 1. Nutzer A stoppt oder trennt die Verbindung, während er in der Warteschlange für einen Sandbox-Start-Slot steht ([execution-manager.ts:405-414](../../../server/services/sandbox/execution-manager.ts#L405-L414)). + 2. Der Runner wird zurückgesetzt ([sandbox-runner-pool.ts:381-452](../../../server/services/sandbox-runner-pool.ts#L381-L452)) und von B übernommen. 3. Die Prüfung nach dem Warten in A's Lauf sieht B's Zustand `STARTING` und läuft weiter. - Folgen: - A's Sketch-Ausgabe läuft über B's Batcher an B's Client. - Der Containername wurde schon *vor* dem Warten gesetzt (Zeile 394f) und dann von B überschrieben. A's Container wird deshalb nie mit `docker rm -f` entfernt. - Bedingungen: mehr als 8 gleichzeitige Starts plus Stop oder Verbindungsabbruch in der Warteschlange. Das ist bei einem Burst zu Kursbeginn realistisch. -- Die WS-Schicht löst dasselbe Problem korrekt über Reservation-IDs ([ws-session-manager.ts:106-152](../server/routes/simulation/ws-session-manager.ts#L106-L152)); der ExecutionManager hat nichts Vergleichbares. +- Die WS-Schicht löst dasselbe Problem korrekt über Reservation-IDs ([ws-session-manager.ts:106-152](../../../server/routes/simulation/ws-session-manager.ts#L106-L152)); der ExecutionManager hat nichts Vergleichbares. ## 3. Architekturprobleme -- **A1. Bis zu fünf überlappende Concurrency-Mechanismen** [Code]: Admission (25), Runner-Pool-Queue (500), Sandbox-Start-Semaphore (8), Compile-Worker-Pool (fest auf ≤8 begrenzt, obwohl die Konfiguration bis 256 erlaubt: [compilation-worker-pool.ts:83-90](../server/services/compilation-worker-pool.ts#L83-L90)) und der UnifiedGatekeeper (im Docker-Betrieb nur noch auf dem Fallback-Pfad aktiv). `/api/status` vermischt die Zahlen: `capacity.compile` zeigt `COMPILE_MAX_CONCURRENT` mit den aktiven Workern, und das Legacy-Feld `compileSlots` zeigt in Wahrheit die Sandbox-Start-Semaphore ([status.routes.ts:59-70](../server/routes/status.routes.ts#L59-L70)). +- **A1. Bis zu fünf überlappende Concurrency-Mechanismen** [Code]: Admission (25), Runner-Pool-Queue (500), Sandbox-Start-Semaphore (8), Compile-Worker-Pool (fest auf ≤8 begrenzt, obwohl die Konfiguration bis 256 erlaubt: [compilation-worker-pool.ts:83-90](../../../server/services/compilation-worker-pool.ts#L83-L90)) und der UnifiedGatekeeper (im Docker-Betrieb nur noch auf dem Fallback-Pfad aktiv). `/api/status` vermischt die Zahlen: `capacity.compile` zeigt `COMPILE_MAX_CONCURRENT` mit den aktiven Workern, und das Legacy-Feld `compileSlots` zeigt in Wahrheit die Sandbox-Start-Semaphore ([status.routes.ts:59-70](../../../server/routes/status.routes.ts#L59-L70)). - **A2. Gatekeeper-Fehler** [Code, ungetestet]: - - Wenn ein Slot per TTL abläuft, wird zwar `slot_released` gesendet, aber kein wartender Auftrag bekommt den Slot ([unified-gatekeeper.ts:425-434](../server/services/unified-gatekeeper.ts#L425-L434)). - - Die spätere echte Freigabe findet keinen Eintrag mehr und gibt die Warteschlange ebenfalls nicht frei ([unified-gatekeeper.ts:370-384](../server/services/unified-gatekeeper.ts#L370-L384)). Die Queue hängt also, während neue Anfragen am Fast-Path vorbeiziehen. - - Die TTL-Tests geben nur frei und prüfen nichts davon ([unified-gatekeeper.test.ts:670-700](../tests/server/services/unified-gatekeeper.test.ts#L670-L700)). + - Wenn ein Slot per TTL abläuft, wird zwar `slot_released` gesendet, aber kein wartender Auftrag bekommt den Slot ([unified-gatekeeper.ts:425-434](../../../server/services/unified-gatekeeper.ts#L425-L434)). + - Die spätere echte Freigabe findet keinen Eintrag mehr und gibt die Warteschlange ebenfalls nicht frei ([unified-gatekeeper.ts:370-384](../../../server/services/unified-gatekeeper.ts#L370-L384)). Die Queue hängt also, während neue Anfragen am Fast-Path vorbeiziehen. + - Die TTL-Tests geben nur frei und prüfen nichts davon ([unified-gatekeeper.test.ts:670-700](../../../tests/server/services/unified-gatekeeper.test.ts#L670-L700)). - Die Cache-Lock-API und `drain()` sind tote Abstraktionen. - **A3. Fallback umgeht die Lastgrenze** [Code]: - - Jeder Fehler des Worker-Pools führt zu einem Compile im Hauptthread ([compiler-with-fallback.ts:65-76](../server/services/compiler-with-fallback.ts#L65-L76)). - - Abgestürzte Worker werden nicht neu gestartet ([compilation-worker-pool.ts:162-171](../server/services/compilation-worker-pool.ts#L162-L171)). Danach läuft dauerhaft alles im Hauptthread. + - Jeder Fehler des Worker-Pools führt zu einem Compile im Hauptthread ([compiler-with-fallback.ts:65-76](../../../server/services/compiler-with-fallback.ts#L65-L76)). + - Abgestürzte Worker werden nicht neu gestartet ([compilation-worker-pool.ts:162-171](../../../server/services/compilation-worker-pool.ts#L162-L171)). Danach läuft dauerhaft alles im Hauptthread. - Die Worker-Queue ist unbegrenzt und hat keine Timeouts. - **A4. Cache-Schlüssel unterscheiden sich** [Code]: - - Der direkte Compiler hasht `code + fqbn + entryFile`, aber **ohne Header** ([arduino-compiler.ts:93-107](../server/services/arduino-compiler.ts#L93-L107)). Der Worker hasht mit Headern ([compile-worker-utils.ts:26-33](../server/services/workers/compile-worker-utils.ts#L26-L33)). + - Der direkte Compiler hasht `code + fqbn + entryFile`, aber **ohne Header** ([arduino-compiler.ts:93-107](../../../server/services/arduino-compiler.ts#L93-L107)). Der Worker hasht mit Headern ([compile-worker-utils.ts:26-33](../../../server/services/workers/compile-worker-utils.ts#L26-L33)). - Folge im lokalen Modus und auf dem Fallback-Pfad: Wird nur eine Header-Datei geändert, liefert der Compile einen veralteten Erfolg aus dem Cache. -- **A5. Das HEX-Binary geht als JSON-Zahlenarray an den Client** [Code]: Es wird mitgeschickt (`Buffer` bzw. `{"0":58,…}` nach `postMessage`), aber vom Client nie genutzt ([compiler.routes.ts:190](../server/routes/compiler.routes.ts#L190)). Es liegt außerdem im LRU-Cache. -- **A6. Der Pool greift in private Felder des Runners** [Code]: Über das Cast `SandboxRunnerInternal` setzt der Pool rund 25 Felder des `ExecutionState` von Hand zurück ([sandbox-runner-pool.ts:25-63](../server/services/sandbox-runner-pool.ts#L25-L63), [381-452](../server/services/sandbox-runner-pool.ts#L381-L452)). Jedes neue Feld im Zustand ist eine mögliche Isolationslücke. Hängt der Reset, wird der Runner ersetzt, ohne dass der Container sicher beendet ist ([sandbox-runner-pool.ts:253-271](../server/services/sandbox-runner-pool.ts#L253-L271)). +- **A5. Das HEX-Binary geht als JSON-Zahlenarray an den Client** [Code]: Es wird mitgeschickt (`Buffer` bzw. `{"0":58,…}` nach `postMessage`), aber vom Client nie genutzt ([compiler.routes.ts:190](../../../server/routes/compiler.routes.ts#L190)). Es liegt außerdem im LRU-Cache. +- **A6. Der Pool greift in private Felder des Runners** [Code]: Über das Cast `SandboxRunnerInternal` setzt der Pool rund 25 Felder des `ExecutionState` von Hand zurück ([sandbox-runner-pool.ts:25-63](../../../server/services/sandbox-runner-pool.ts#L25-L63), [381-452](../../../server/services/sandbox-runner-pool.ts#L381-L452)). Jedes neue Feld im Zustand ist eine mögliche Isolationslücke. Hängt der Reset, wird der Runner ersetzt, ohne dass der Container sicher beendet ist ([sandbox-runner-pool.ts:253-271](../../../server/services/sandbox-runner-pool.ts#L253-L271)). - **A7. Lifecycle-Lücken:** - - Beim Start gibt es keinen Sweep für verwaiste `unosim-sandbox-*`-Container. Bei `uncaughtException` beendet sich der Prozess im Docker-Modus sofort ([index.ts:186-193](../server/index.ts#L186-L193)), laufende Container bleiben stehen. + - Beim Start gibt es keinen Sweep für verwaiste `unosim-sandbox-*`-Container. Bei `uncaughtException` beendet sich der Prozess im Docker-Modus sofort ([index.ts:186-193](../../../server/index.ts#L186-L193)), laufende Container bleiben stehen. - Es gibt keinen WS-Ping/Pong. Halb offene Verbindungen halten ihre Reservierung bis zum Simulations-Timeout. - - Nachrichten einer Verbindung werden nicht serialisiert ([simulation.ws.ts:770-772](../server/routes/simulation.ws.ts#L770-L772)). + - Nachrichten einer Verbindung werden nicht serialisiert ([simulation.ws.ts:770-772](../../../server/routes/simulation.ws.ts#L770-L772)). - Für `serial_input` und `set_pin_value` gibt es kein Rate-Limit pro Nachricht. ## 4. Tutor, Course Content und Evaluation: Rückstände ohne Semantikfehler -- **T1. Übergangsschichten im `CurriculumTutorAdapter`** [Code]: Der Adapter kennt drei Snapshot-Quellen: den vom Route-Handler gelieferten Kontext, `CourseContentSnapshotProvider` und den Legacy-Zweig `DidacticContentRepository` ([curriculum-tutor-adapter.ts:192-217](../server/services/tutor/curriculum-tutor-adapter.ts#L192-L217)). Produktiv wird nur der erste genutzt (Factory ohne Abhängigkeiten). Das Repository ist als „Test-only compatibility port“ markiert, hält aber einen Zweig im Produktionscode und zehn Testdateien am Leben. `TutorCourseContentResolver.getTutorContent` wird nirgends aufgerufen. -- **T2. `match()` hat versteckte Seiteneffekte** [Code]: Die Methode verändert `activeTopicId`, `phase` und `progressionBlockedReason` ([curriculum-tutor-adapter.ts:219-256](../server/services/tutor/curriculum-tutor-adapter.ts#L219-L256)). +- **T1. Übergangsschichten im `CurriculumTutorAdapter`** [Code]: Der Adapter kennt drei Snapshot-Quellen: den vom Route-Handler gelieferten Kontext, `CourseContentSnapshotProvider` und den Legacy-Zweig `DidacticContentRepository` ([curriculum-tutor-adapter.ts:192-217](../../../server/services/tutor/curriculum-tutor-adapter.ts#L192-L217)). Produktiv wird nur der erste genutzt (Factory ohne Abhängigkeiten). Das Repository ist als „Test-only compatibility port“ markiert, hält aber einen Zweig im Produktionscode und zehn Testdateien am Leben. `TutorCourseContentResolver.getTutorContent` wird nirgends aufgerufen. +- **T2. `match()` hat versteckte Seiteneffekte** [Code]: Die Methode verändert `activeTopicId`, `phase` und `progressionBlockedReason` ([curriculum-tutor-adapter.ts:219-256](../../../server/services/tutor/curriculum-tutor-adapter.ts#L219-L256)). - Deshalb muss `planAnswered` vorher klonen. - `planFollowup` ruft `match` zweimal auf. - Ein `catch {}` verschluckt jeden Fehler ohne Log und fällt still auf den freien Tutor zurück. - - `findQuestion` existiert zweimal mit leicht unterschiedlicher Semantik (Adapter [curriculum-tutor-adapter.ts:618](../server/services/tutor/curriculum-tutor-adapter.ts#L618), Planner [learning-planner.ts:203](../server/services/tutor/curriculum/learning-planner.ts#L203)). -- **T3. Der Client hat bei der Mastery-Evidenz mitzureden** [Code]: `sessionHistory` mischt gespeicherte Evidenz mit den vom Client gesendeten `answerRating`/`questionId`. `collectObservations` zählt Client-Ratings für Fragen ohne gespeicherte Evidenz ([learning-planner.ts:193-201](../server/services/tutor/curriculum/learning-planner.ts#L193-L201)). Die Wirkung betrifft nur den eigenen Nutzer. Es widerspricht aber dem Geist von „deterministische Mastery“ aus ADR 0007. Das ist eine **Produkt- und Vertrauensfrage**, kein Bug. -- **T4. Verlorene Updates bei gleichzeitigen Anfragen** [Code]: Ablauf ist Klonen, Provider-Call (bis 30 s), dann Commit, bei dem der Letzte gewinnt ([tutor-service.ts:612-620](../server/services/tutor/tutor-service.ts#L612-L620), [progression-state.ts:52-64](../server/services/tutor/curriculum/progression-state.ts#L52-L64)). Zwei Anfragen auf derselben Session (Doppelklick, Retry nach 504) verlieren Evidenz. Abhilfe wäre ein Versions- oder Sperrmechanismus pro Session. -- **T5. GitHub als harte Abhängigkeit für neue Tutor-Fragen** [Code]: Neue Fragen verlangen `requireFresh`. Fällt GitHub aus, scheitern sie, während Examples mit veraltetem Stand weiterlaufen ([source-provider.ts:66-68](../server/services/examples/source-provider.ts#L66-L68)). Es gibt kein GitHub-Token; die unauthentifizierte API erlaubt 60 Anfragen pro Stunde und IP. Mit vielen Browser-Overrides ist das ein reales Risiko. + - `findQuestion` existiert zweimal mit leicht unterschiedlicher Semantik (Adapter [curriculum-tutor-adapter.ts:618](../../../server/services/tutor/curriculum-tutor-adapter.ts#L618), Planner [learning-planner.ts:203](../../../server/services/tutor/curriculum/learning-planner.ts#L203)). +- **T3. Der Client hat bei der Mastery-Evidenz mitzureden** [Code]: `sessionHistory` mischt gespeicherte Evidenz mit den vom Client gesendeten `answerRating`/`questionId`. `collectObservations` zählt Client-Ratings für Fragen ohne gespeicherte Evidenz ([learning-planner.ts:193-201](../../../server/services/tutor/curriculum/learning-planner.ts#L193-L201)). Die Wirkung betrifft nur den eigenen Nutzer. Es widerspricht aber dem Geist von „deterministische Mastery“ aus ADR 0007. Das ist eine **Produkt- und Vertrauensfrage**, kein Bug. +- **T4. Verlorene Updates bei gleichzeitigen Anfragen** [Code]: Ablauf ist Klonen, Provider-Call (bis 30 s), dann Commit, bei dem der Letzte gewinnt ([tutor-service.ts:612-620](../../../server/services/tutor/tutor-service.ts#L612-L620), [progression-state.ts:52-64](../../../server/services/tutor/curriculum/progression-state.ts#L52-L64)). Zwei Anfragen auf derselben Session (Doppelklick, Retry nach 504) verlieren Evidenz. Abhilfe wäre ein Versions- oder Sperrmechanismus pro Session. +- **T5. GitHub als harte Abhängigkeit für neue Tutor-Fragen** [Code]: Neue Fragen verlangen `requireFresh`. Fällt GitHub aus, scheitern sie, während Examples mit veraltetem Stand weiterlaufen ([source-provider.ts:66-68](../../../server/services/examples/source-provider.ts#L66-L68)). Es gibt kein GitHub-Token; die unauthentifizierte API erlaubt 60 Anfragen pro Stunde und IP. Mit vielen Browser-Overrides ist das ein reales Risiko. - **Kein Handlungsbedarf:** - Die Prompt-Duplikation zwischen Builder und `TUTOR_PROMPT_TEMPLATE_SOURCES` sichert ein Drift-Test ab (`tutor-prompt-revision.test.ts`). - Die Evaluation hängt nur über `canonical.ts` am Produktionscode. @@ -121,29 +121,29 @@ Im Tutor gibt es vor allem Übergangsschichten und unklare Zuständigkeiten, abe | Befund | Status | |---|---| -| Mit 200 Studierenden und 50 aktiven Simulationen liegt die Queue-Wartezeit bei p95 211 s. Die Doku nennt die Queue bei einer UX-Grenze von 240 s „infeasible“. Eine Simulation hält ihren Slot für die gesamte Laufzeit (Default 60 s, maximal 300 s). | [gemessen] ([CAPACITY_VALIDATION_PLAN.md](CAPACITY_VALIDATION_PLAN.md)) | +| Mit 200 Studierenden und 50 aktiven Simulationen liegt die Queue-Wartezeit bei p95 211 s. Die Doku nennt die Queue bei einer UX-Grenze von 240 s „infeasible“. Eine Simulation hält ihren Slot für die gesamte Laufzeit (Default 60 s, maximal 300 s). | [gemessen] ([CAPACITY_MODEL.md](../../CAPACITY_MODEL.md)) | | Compose setzt `SIMULATION_MAX_CONCURRENT=200`, die Admission bleibt beim Default 25. Effektiv laufen also maximal 25 Simulationen; die 200 wirken nicht. Kalibriert empfohlen sind 50/8/200/43000. | [Code] | | Jeder Compile & Run kompiliert zweimal: `arduino-cli` im Backend und `g++` im Sandbox-Container mit 0,25 CPU. Die REST-Compile-Kapazität ist laut Doku „nicht kalibriert“. | [Code] | -| Der globale API-Limiter erlaubt 300 Anfragen pro 15 min **pro IP** ([index.ts:113-124](../server/index.ts#L113-L124)). Hinter Campus-NAT teilt sich eine ganze Klasse dieses Budget; die identitätsbasierten Limiter existieren daneben. | [Code] Risiko, abhängig von der Netztopologie | +| Der globale API-Limiter erlaubt 300 Anfragen pro 15 min **pro IP** ([index.ts:113-124](../../../server/index.ts#L113-L124)). Hinter Campus-NAT teilt sich eine ganze Klasse dieses Budget; die identitätsbasierten Limiter existieren daneben. | [Code] Risiko, abhängig von der Netztopologie | | Gatekeeper hängt nach TTL-Ablauf, der Fallback umgeht die Lastgrenze, die Worker-Queue ist unbegrenzt. | [Code] | | Runner-Race (S4), keine Orphan-Sweeps, kein Heartbeat, Tutor-Lost-Update | Architekturrisiko | | Kein `bufferedAmount`-Backpressure beim WS-Senden, stdin-Puffer bei Eingabe-Floods, Größe des Binary-JSON | [Hypothese] | -| Kleinere Leaks: `compileTimes` wächst unbegrenzt ([compilation-worker-pool.ts:265](../server/services/compilation-worker-pool.ts#L265)), nie gelöschte Race-Timer in [prepare-phase.ts:41-49](../server/services/sandbox/execution-phases/prepare-phase.ts#L41-L49) | [Code], geringe Wirkung | +| Kleinere Leaks: `compileTimes` wächst unbegrenzt ([compilation-worker-pool.ts:265](../../../server/services/compilation-worker-pool.ts#L265)), nie gelöschte Race-Timer in [prepare-phase.ts:41-49](../../../server/services/sandbox/execution-phases/prepare-phase.ts#L41-L49) | [Code], geringe Wirkung | ## 6. Dokumentationsinkonsistenzen | Stelle | Befund | Einordnung | |---|---|---| -| ADR 0001 §51, `SECURITY.md` | „Backend nur von `UNOSIM_TRUSTED_PROXY` erreichbar“. Der Code prüft die Quell-IP nicht, er setzt nur `trust proxy` ([access-control.ts:177-213](../server/security/access-control.ts#L177-L213)). | echte Inkonsistenz | +| ADR 0001 §51, `SECURITY.md` | „Backend nur von `UNOSIM_TRUSTED_PROXY` erreichbar“. Der Code prüft die Quell-IP nicht, er setzt nur `trust proxy` ([access-control.ts:177-213](../../../server/security/access-control.ts#L177-L213)). | echte Inkonsistenz | | `SECURITY.md` Sandbox-Abschnitt | Der unsandboxed REST-Compile (S1) wird nicht erwähnt. | echte Inkonsistenz | -| `EXTERNAL_API.md:23` | „Initialer `handshake` mit `protocolVersion`“. Der Server sendet ihn nur bei `testRunId` ([simulation.ws.ts:762-768](../server/routes/simulation.ws.ts#L762-L768)). | echte Inkonsistenz | +| `EXTERNAL_API.md:23` | „Initialer `handshake` mit `protocolVersion`“. Der Server sendet ihn nur bei `testRunId` ([simulation.ws.ts:762-768](../../../server/routes/simulation.ws.ts#L762-L768)). | echte Inkonsistenz | | ARCHITECTURE, WS-Sicherheit | „Rate-Limiting der Nachrichtenfrequenz“: Es gibt nur ein Limit für `start_simulation`. | echte Inkonsistenz | -| Deprecation-Policy | ARCHITECTURE sagt „nach 2 Major-Releases“, der Code sagt „next major“ ([routes.ts:138-142](../server/routes.ts#L138-L142)). | konkurrierende Aussagen | -| `SECURITY.md` „Secrets/Sketch-Source nicht loggen“ | Auf Debug-Level werden Code und Payloads geloggt ([simulation.ws.ts:467-486](../server/routes/simulation.ws.ts#L467-L486), [ws-message-router.ts:41](../server/routes/simulation/ws-message-router.ts#L41)). | echte Inkonsistenz, nur bei Debug | +| Deprecation-Policy | ARCHITECTURE sagt „nach 2 Major-Releases“, der Code sagt „next major“ ([routes.ts:138-142](../../../server/routes.ts#L138-L142)). | konkurrierende Aussagen | +| `SECURITY.md` „Secrets/Sketch-Source nicht loggen“ | Auf Debug-Level werden Code und Payloads geloggt ([simulation.ws.ts:467-486](../../../server/routes/simulation.ws.ts#L467-L486), [ws-message-router.ts:41](../../../server/routes/simulation/ws-message-router.ts#L41)). | echte Inkonsistenz, nur bei Debug | | `SECURITY.md` „reviewed immutable images“ | Dockerfile und CI installieren `arduino-cli` per `curl … master/install.sh \| sh`. | echte Inkonsistenz | | ARCHITECTURE und docs/README: „Tree enthält nur normative Doku“ | Im Tree liegen rund 15 als historisch oder nicht normativ markierte Pläne und Reports sowie die historische SSOT Stage2B. | absichtlich historisch, aber die Governance-Aussage ist falsch | | SSOT TutorQualityEvaluation und weitere getrackte Dateien | Sie verweisen auf `ssot_function_tutor_model_registration.md` und `UNOSIM_TECHNICAL_ARCHITECTURE_HEALTH_REVIEW.md`. Beide waren zum Analysezeitpunkt nie in Git. | bewusst lokal(?), für einen frischen Clone nicht nachvollziehbar | -| README/ARCHITECTURE nennen `SCALABILITY.md` „gemessene Kapazitätsgrenzen“ | Die Datei bezeichnet sich selbst als historisch. | kosmetisch bis irreführend | +| README/ARCHITECTURE nennen `CAPACITY_HISTORICAL_PHASE_3_4.md` „gemessene Kapazitätsgrenzen“ | Die Datei bezeichnet sich selbst als historisch. | kosmetisch bis irreführend | | ARCHITECTURE: `ArduinoSimulatorPage` „753 Zeilen“, Coverage-Prozente, „In-Memory-Cache“, „Compile Semaphore“, „max. 60 s“ | Tatsächlich 7 Zeilen, drei Cache-Ebenen, Gatekeeper/Pool, maximal 300 s. | veraltet, eher kosmetisch | | Config | `disableGatekeeper` und `compilation.timeoutMs` werden nicht genutzt; der CLI-Timeout ist auf 60 s fest codiert. | tote Wahrheit | diff --git a/docs/UNOSIM_REFACTORING_OPL.md b/docs/archive/refactoring/REFACTORING_OPL.md similarity index 97% rename from docs/UNOSIM_REFACTORING_OPL.md rename to docs/archive/refactoring/REFACTORING_OPL.md index 99adc0db4..e51d3d15e 100644 --- a/docs/UNOSIM_REFACTORING_OPL.md +++ b/docs/archive/refactoring/REFACTORING_OPL.md @@ -1,7 +1,10 @@ # UnoSim – Refactoring-OPL (Open Points List) +> **Archiviert am 2026-10-11.** Die aktuellen offenen Punkte stehen in +> [../../OPEN_POINTS.md](../../OPEN_POINTS.md). + > **Nicht normativ.** Arbeitsstand der Refactoring-Serie, die aus -> [UNOSIM_TECHNICAL_ARCHITECTURE_HEALTH_REVIEW.md](UNOSIM_TECHNICAL_ARCHITECTURE_HEALTH_REVIEW.md) +> [ARCHITECTURE_HEALTH_REVIEW.md](ARCHITECTURE_HEALTH_REVIEW.md) > (Analyse-Stand `54cc2950`) abgeleitet ist. Die Liste wird nach jedem > abgeschlossenen PR aktualisiert. Normative Verträge bleiben ADRs und SSOTs. @@ -33,7 +36,7 @@ Abweichungen werden unter „Reihenfolge-Änderungen“ begründet. | R10b | Doku an den implementierten Stand angleichen (ARCHITECTURE, SECURITY, docs/README, INSTALL_SERVER, README, Compose-Kommentar) | Doku | Abschnitt 6 | gering | Doku beschreibt Isolation, Compile-Grenze inkl. Restrisiko, Lifecycle, Backpressure und offene Entscheidungen | klein | R10a | docs/consistency-after-refactoring | DONE | `check:docs`; keine ADR/SSOT geändert | PR-Merge siehe Verlauf | | R10-PROXY | ADR 0001 §51 „Backend nur vom Trusted Proxy erreichbar“ wird im Code nicht erzwungen | Security/Vertrag | [Code] | mittel | – | – | – | – | BLOCKED_DECISION | – | entweder Quell-IP-Prüfung im Code (Rollout-Risiko bei falsch gesetztem `UNOSIM_TRUSTED_PROXY`) oder ADR-Formulierung als Betreiberpflicht; ADR wird nicht passend geschrieben | | R10-HANDSHAKE | EXTERNAL_API: „initialer Handshake mit `protocolVersion`“; Schema und Server senden ihn nur mit `testRunId` | Vertrag | [Code] | gering | – | – | – | – | BLOCKED_DECISION | – | Handshake für alle erfordert Schemaänderung (`testRunId` optional), sonst Vertragstext anpassen | -| R10-PIN | `arduino-cli` wird in Dockerfile und CI ungepinnt per `curl … master/install.sh` installiert (SECURITY: „reviewed immutable images“) | Supply Chain | [Code] | mittel | – | – | – | – | BLOCKED_DECISION | – | Versionswahl und Update-Prozess sind eine Betriebsentscheidung | +| R10-PIN | `arduino-cli` wird in Dockerfile und CI ungepinnt per `curl … master/install.sh` installiert (SECURITY: „reviewed immutable images“) | Supply Chain | [Code] | mittel | – | – | – | – | DONE | Dockerfile und CI prüfen `arduino-cli version` auf 1.5.1 | `fff4be669` (build(toolchain): pin tested Arduino CLI 1.5.1) | | R10-SSOTREF | SSOT TutorQualityEvaluation verweist auf die geschützte, nicht versionierte `ssot_function_tutor_model_registration.md` | Doku | [Code] | gering | – | – | – | – | DEFERRED | – | geschützte Datei und SSOT bleiben unverändert; Klärung durch den Owner | ## Einzelbefunde @@ -63,7 +66,7 @@ Abweichungen werden unter „Reihenfolge-Änderungen“ begründet. | T2 | `match()` mit Seiteneffekten, doppeltes `findQuestion`, verschluckte Fehler | Tutor | [Code] bestätigt | gering–mittel | – | – | – | R8a | DONE | – | `planAnswered` behält seinen Klon, weil sein Ergebnis die angewandte Sicht braucht | | T3 | Client-Ratings fließen in Mastery-Evidenz ein | Produkt/Trust | [Code] | gering | – | – | – | – | BLOCKED_DECISION | – | Produktentscheidung | | T4 | Lost Update bei parallelen Anfragen einer Session | Tutor | [Code] reproduziert | mittel | – | – | – | R8b | DONE | – | Serialisierung statt 409: kein neuer Client-Vertrag nötig | -| T5 | Neue Tutor-Fragen scheitern bei GitHub-Ausfall (`requireFresh`) | Produkt/Verfügbarkeit | [Code] | mittel | – | – | – | – | BLOCKED_DECISION | – | Produktentscheidung | +| T5 | Neue Tutor-Fragen scheitern bei GitHub-Ausfall (`requireFresh`) | Produkt/Verfügbarkeit | [Code] | mittel | – | – | – | docs/cleanup-plans | DONE | RED→GREEN: `tutor.routes.test.ts` (Quelle nicht verfügbar → freier Tutor ohne Session) | Entscheidung 2026-10-11: freier Tutor (Course Content SSOT 11, Fall G) | ## Produktentscheidungen (nicht implementieren) @@ -71,7 +74,7 @@ Abweichungen werden unter „Reihenfolge-Änderungen“ begründet. |---|---|---|---|---|---|---|---|---|---|---|---| | D1 | LEARN nach schwacher/mittlerer Antwort, wenn das Concept erschöpft, das Topic aber noch probeable ist | Produkt | nicht spezifiziert | – | – | – | – | – | BLOCKED_DECISION | – | – | | D2 | = T3: dürfen Client-Ratings Mastery beeinflussen? | Produkt/Trust | [Code] | – | – | – | – | – | BLOCKED_DECISION | – | – | -| D3 | = T5: Verhalten neuer Tutor-Fragen bei GitHub-Ausfall / Last-known-good | Produkt | [Code] | – | – | – | – | – | BLOCKED_DECISION | – | – | +| D3 | = T5: Verhalten neuer Tutor-Fragen bei GitHub-Ausfall / Last-known-good | Produkt | [Code] | – | – | – | – | – | DONE | – | entschieden 2026-10-11: freier Tutor, siehe T5 | | D4 | Session-TTL (absolut 1 h) als Produktpolitik | Produkt | [Code] | – | – | – | – | – | BLOCKED_DECISION | – | – | | D5 | Simulationsslot-Laufzeit, Inaktivitäts-Timeout, automatische Beendigung | Kapazität/Produkt | [gemessen] | – | – | – | – | – | BLOCKED_DECISION | – | – | diff --git a/docs/UNOSIM_FREETUTOR_QUALITY_PLAN.md b/docs/archive/tutor-quality/UNOSIM_FREETUTOR_QUALITY_PLAN.md similarity index 98% rename from docs/UNOSIM_FREETUTOR_QUALITY_PLAN.md rename to docs/archive/tutor-quality/UNOSIM_FREETUTOR_QUALITY_PLAN.md index 08e66d5a3..696453564 100644 --- a/docs/UNOSIM_FREETUTOR_QUALITY_PLAN.md +++ b/docs/archive/tutor-quality/UNOSIM_FREETUTOR_QUALITY_PLAN.md @@ -3,7 +3,7 @@ Stand: 2026-10-06 Status: Plan, **nicht normativ**. Normativ ist `ssot/ssot_function_definition_TutorQualityEvaluation.md` (Regeln `R-…`). -Baut auf `docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md` und den Berichten vom +Baut auf `docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md` und den Berichten vom 2026-10-03 auf. Scope: nur der **Freetutor** (kein aktives Topic, keine Inline-Steuerung, keine Phasenstrategien). @@ -164,7 +164,7 @@ Vorab Ihre Entscheidung zu den Kriterien, dann SSOT-Änderung vor Code (R-COR-1) - Optional (Variante D): ein anderes Tutor-Modell, falls die Relativierung korrekter Antworten bleibt. Dafür brauche ich von Ihnen eine feste Modell-ID. Das Schema steht in - `docs/tutor-quality-next-architecture-2026-10-03.md` §5. + `docs/archive/tutor-quality/tutor-quality-next-architecture-2026-10-03.md` §5. ### Schritt 6 – Verstetigen diff --git a/docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md b/docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md similarity index 96% rename from docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md rename to docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md index 64a486cf2..84204a23b 100644 --- a/docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md +++ b/docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md @@ -4,9 +4,9 @@ Stand: 2026-10-02 Status: ausführbarer Plan, **nicht normativ**. Normativ ist `ssot/ssot_function_definition_TutorQualityEvaluation.md` (im Folgenden „Evaluation-SSOT“, Regeln als `R-…`). -Ersetzt als Arbeitsgrundlage: `docs/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md`, -`docs/plan-tutor-quality-stage-2a.md`, -`docs/superpowers/plans/2026-09-27-tutor-quality-stage-1.md` (historisch). +Ersetzt als Arbeitsgrundlage: `docs/archive/tutor-quality/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md`, +`docs/archive/tutor-quality/plan-tutor-quality-stage-2a.md`, +`docs/superpowers/plans/2026-09-27-tutor-quality-stage-1.md` (gelöscht, siehe Git-Historie) (historisch). Geprüft gegen: `main` @ `c2c2940a`, lokaler Branch `refactor/tutor-quality-phase2-1-strategy-cases` @ `79032a3f` @@ -19,7 +19,7 @@ Geprüft gegen: `main` @ `c2c2940a`, lokaler Branch **Claude Sonnet / Luna MUST read:** 1. `ssot/ssot_function_definition_TutorQualityEvaluation.md` -2. `docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md` +2. `docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md` **before implementing any Tutor-Quality change.** @@ -303,7 +303,7 @@ Gates grün. 4. `ssot/ssot_function_definition_TutorQuality.md` §7: auf die Evaluation-SSOT verweisen (der Hinweis ist bereits gesetzt; Wortlaut prüfen). 5. Historische Pläne: Kopfzeilen prüfen (bereits gesetzt, siehe Abschnitt 4). -6. `docs/tutor-quality-judge-smoke-2026-10-01.md`: Kopfzeile mit Verweis auf +6. `docs/archive/tutor-quality/tutor-quality-judge-smoke-2026-10-01.md`: Kopfzeile mit Verweis auf das Glossar der Evaluation-SSOT (Begriffe Protocol A/B, G1). 7. `.github/workflows/tutor-quality-real-provider.yml`: Artefaktname `tutor-quality-stage-2a-…` → `tutor-quality-run-…`; Input-Text „leave empty @@ -444,7 +444,7 @@ erweitern. Kein neuer Runner, kein neuer Workflow. damit bleiben die Wochenläufe mit diesen Daten vergleichbar. - **Bekannter Zustand bei Aktivierung:** Baseline und Nachlauf ergeben nach §12 `fail`, getragen von echten Tutor-Befunden - (`docs/tutor-quality-remediation-rerun-2026-10-03.md`): + (`docs/archive/tutor-quality/tutor-quality-remediation-rerun-2026-10-03.md`): `strategy-expand-observation` / `expand-observable-result` (6 / 6 Samples über beide Läufe) und `strategy-learn-strong-answer` mit `learn-feedback-grounds-sketch` (3 / 3) sowie @@ -489,10 +489,10 @@ Nicht in A–G mischen: Als historisch bzw. abgelöst markiert, mit Verweis auf diesen Plan oder die Evaluation-SSOT: -- `docs/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md` -- `docs/plan-tutor-quality-stage-2a.md` -- `docs/superpowers/plans/2026-09-27-tutor-quality-stage-1.md` +- `docs/archive/tutor-quality/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md` +- `docs/archive/tutor-quality/plan-tutor-quality-stage-2a.md` +- `docs/superpowers/plans/2026-09-27-tutor-quality-stage-1.md` (gelöscht, siehe Git-Historie) - `ssot/ssot_function_definition_TutorQualityStage2A.md` (abgelöst; in PR D entfernt) - `docs/tutor-quality-stage-2a.md` (abgelöst; in PR D entfernt, Bedienteil in Appendix B der Evaluation-SSOT) -- `ssot/ssot_function_definition_TutorQualityStage2B.md`, `docs/plan-tutor-quality-stage-2b.md` +- `docs/archive/tutor-quality/ssot_function_definition_TutorQualityStage2B.md`, `docs/archive/tutor-quality/plan-tutor-quality-stage-2b.md` (waren bereits historisch; Verweis ergänzt) diff --git a/docs/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md b/docs/archive/tutor-quality/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md similarity index 98% rename from docs/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md rename to docs/archive/tutor-quality/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md index 92b6d139d..0eb655921 100644 --- a/docs/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md +++ b/docs/archive/tutor-quality/UNOSIM_TUTOR_SIMPLIFICATION_PLAN.md @@ -1,9 +1,9 @@ # UnoSim – Umsetzungsplan zur Vereinfachung von Tutor und Tutor Quality -> **HISTORICAL / SUPERSEDED (2026-10-02).** Dieser Plan ist abgeschlossen bzw. überholt und keine Arbeitsgrundlage mehr. Aktueller Plan: `docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`; normativ: `ssot/ssot_function_definition_TutorQualityEvaluation.md`. +> **HISTORICAL / SUPERSEDED (2026-10-02).** Dieser Plan ist abgeschlossen bzw. überholt und keine Arbeitsgrundlage mehr. Aktueller Plan: `docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`; normativ: `ssot/ssot_function_definition_TutorQualityEvaluation.md`. Stand: 2026-09-29 -Grundlage: `docs/UNOSIM_TECHNICAL_ARCHITECTURE_HEALTH_REVIEW.md` +Grundlage: `docs/archive/refactoring/ARCHITECTURE_HEALTH_REVIEW.md` Geprüft gegen: - `main` @ `59686397817e6dba123374590e00c750d0e41200` @@ -353,7 +353,7 @@ zurückholen · **REMOVE** entfernen ohne Rückholabsicht. 4. **Nach Gate G1:** Branch mit einem Tag archivieren (z. B. `archive/stage-2b-pr2-protocol-a`) und den offenen PR mit Verweis auf diesen Plan und die Messergebnisse schließen. -5. **Der offene Stage-2B-Plan** (`docs/plan-tutor-quality-stage-2b.md`, PR 2–6) wird +5. **Der offene Stage-2B-Plan** (`docs/archive/tutor-quality/plan-tutor-quality-stage-2b.md`, PR 2–6) wird in Phase 2 durch einen kurzen Verweis auf diesen Plan ersetzt (siehe 6, PR 2.4). --- @@ -466,7 +466,7 @@ Stage-2B-Bibliothek ist von `main` entfernt. | **2.1 Strategie-Fälle** | Fixture `progression-deepen` in `anchor-course-content.ts`; je **2–3 Szenarien** für LEARN, DEEPEN und EXPAND mit `judge`-Block. Kriterien pro Phase (Beispiele, final im PR): **LEARN:** Rating passt zur Antwortqualität, Feedback ohne Komplettlösung, bei schwacher Antwort Hinweis bzw. diagnostische Frage gemäß `hintFirst`. **DEEPEN:** Folgefrage verlangt Anwendung oder Transfer auf dasselbe Topic und fragt kein bereits Beherrschtes erneut ab. **EXPAND:** Frage zielt auf eine eigene, am Sketch prüfbare Erweiterung, bezieht sich auf das Expansion-Ziel und liefert keinen Ersatzcode. Deterministische `expected.learningPhase` bleibt für jedes Szenario gesetzt | `anchor-course-content.ts`, `anchor-corpus.yaml`, ggf. 1–2 neue `.ino`-Fixtures | KEEP + Daten | +60 / +40 | | **2.2 Stage 2B entfernen** | Alle Dateien aus 4.4 mit REMOVE/DEFER löschen; `semantic-corpus.yaml` löschen (TQ-SEM-001 ist seit 1.3 migriert); Re-Export `semantic-canonical.ts` entfernen; knip bleibt grün | `server/services/tutor/evaluation/semantic/**`, `tests/…/semantic/**`, `evals/tutor-quality/semantic-corpus.yaml`, `semantic-case-review-matrix.md` | REMOVE/DEFER | −1.206 / −868 | | **2.3 Stage-2A-Runner entschlacken** | `real-provider-evaluation.ts`: Artifact-I/O nach `report.ts`, Rates/`byScenario`-Aggregation durch einfache Zähler ersetzen, No-op-Block entfernen, Typen straffen; Verhalten der Checks unverändert | `real-provider-evaluation.ts`, `report.ts`, zugehörige Tests | SIMPLIFY | −500 bis −650 / −80 bis −120 | -| **2.4 Doku zurückschneiden** | `ssot/ssot_function_definition_TutorQualityStage2B.md` (1.285 Z.) auf „Stage 2B Core“ (≈150–250 Z.) kürzen: Ziel, Datenmodell, Judge-Protokoll nach G1, Evidence-Grenzen, Statusmodell, Report. Ein Abschnitt „Deferred“ verweist auf diesen Plan. `docs/plan-tutor-quality-stage-2b.md` durch einen kurzen Hinweis ersetzen. `docs/tutor-quality-stage-2a.md` um Judge-Nutzung ergänzen | `ssot/…Stage2B.md`, `docs/plan-tutor-quality-stage-2b.md`, `docs/tutor-quality-stage-2a.md` | SIMPLIFY | Doku ≈ −1.500 Zeilen | +| **2.4 Doku zurückschneiden** | `docs/archive/tutor-quality/ssot_function_definition_TutorQualityStage2B.md` (1.285 Z.) auf „Stage 2B Core“ (≈150–250 Z.) kürzen: Ziel, Datenmodell, Judge-Protokoll nach G1, Evidence-Grenzen, Statusmodell, Report. Ein Abschnitt „Deferred“ verweist auf diesen Plan. `docs/archive/tutor-quality/plan-tutor-quality-stage-2b.md` durch einen kurzen Hinweis ersetzen. `docs/tutor-quality-stage-2a.md` um Judge-Nutzung ergänzen | `ssot/…Stage2B.md`, `docs/archive/tutor-quality/plan-tutor-quality-stage-2b.md`, `docs/tutor-quality-stage-2a.md` | SIMPLIFY | Doku ≈ −1.500 Zeilen | **Abhängigkeiten:** Phase 1 abgeschlossen (G1). 2.1 ist unabhängig von 2.2 und 2.3. 2.2 vor 2.3, weil 2.3 dann keine Stage-2B-Importe mehr berücksichtigen muss. 2.4 diff --git a/docs/plan-tutor-quality-stage-2a.md b/docs/archive/tutor-quality/plan-tutor-quality-stage-2a.md similarity index 98% rename from docs/plan-tutor-quality-stage-2a.md rename to docs/archive/tutor-quality/plan-tutor-quality-stage-2a.md index 65b680e46..bc1115d5c 100644 --- a/docs/plan-tutor-quality-stage-2a.md +++ b/docs/archive/tutor-quality/plan-tutor-quality-stage-2a.md @@ -1,6 +1,6 @@ # Tutor Quality – Stage 2A Implementation Plan -> **HISTORICAL / SUPERSEDED — completed plan (2026-10-02).** Not current worker guidance. Current plan: `docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`; normative model: `ssot/ssot_function_definition_TutorQualityEvaluation.md`. +> **HISTORICAL / SUPERSEDED — completed plan (2026-10-02).** Not current worker guidance. Current plan: `docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`; normative model: `ssot/ssot_function_definition_TutorQualityEvaluation.md`. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. diff --git a/docs/plan-tutor-quality-stage-2b.md b/docs/archive/tutor-quality/plan-tutor-quality-stage-2b.md similarity index 98% rename from docs/plan-tutor-quality-stage-2b.md rename to docs/archive/tutor-quality/plan-tutor-quality-stage-2b.md index 833a49c9e..104d79423 100644 --- a/docs/plan-tutor-quality-stage-2b.md +++ b/docs/archive/tutor-quality/plan-tutor-quality-stage-2b.md @@ -1,6 +1,6 @@ # Tutor Quality Stage 2B – Implementation Plan -> Current plan: `docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`; normative model: `ssot/ssot_function_definition_TutorQualityEvaluation.md`. +> Current plan: `docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`; normative model: `ssot/ssot_function_definition_TutorQualityEvaluation.md`. > **Historical plan — inactive.** The Stage-2B semantic implementation described here was removed in commit `ae8e2f53f27a5c8f9dce09649b668e44081fec6b`. This plan is retained as design history, not as current worker guidance. The active path uses `evals/tutor-quality/anchor-corpus.yaml` with the existing Stage-A runner and its optional Minimal Judge. Do not recreate this plan's semantic corpus, adapter, exposure, or transcript-reference architecture unless a later approved plan explicitly reactivates it. @@ -12,9 +12,9 @@ **Tech Stack:** TypeScript/ESM, Node 24.20.0 from .nvmrc, existing yaml and zod dependencies, existing TutorService/LLMProvider/KiconnectProvider boundary, Vitest, tsx, and an optional workflow_dispatch-only GitHub Actions workflow. -**Spec:** ssot/ssot_function_definition_TutorQualityStage2B.md (normative); the former Stage-2A SSOT (removed in PR D; superseded by ssot/ssot_function_definition_TutorQualityEvaluation.md) and ssot/ssot_function_definition_TutorQuality.md were authoritative within their scopes at the time. +**Spec:** docs/archive/tutor-quality/ssot_function_definition_TutorQualityStage2B.md (normative at the time); the former Stage-2A SSOT (removed in PR D; superseded by ssot/ssot_function_definition_TutorQualityEvaluation.md) and ssot/ssot_function_definition_TutorQuality.md were authoritative within their scopes at the time. -This plan follows the existing repository convention at docs/plan-tutor-quality-stage-2a.md. It is non-normative; where it conflicts with the Stage-2B SSOT, the SSOT wins. The implementation branch starts from 68d2703f3513762b521284718f0b5c0b6a4a5de8 and is docs/tutor-quality-stage-2b-implementation-plan. +This plan follows the existing repository convention at docs/archive/tutor-quality/plan-tutor-quality-stage-2a.md. It is non-normative; where it conflicts with the Stage-2B SSOT, the SSOT wins. The implementation branch starts from 68d2703f3513762b521284718f0b5c0b6a4a5de8 and is docs/tutor-quality-stage-2b-implementation-plan. ## Global Constraints @@ -430,7 +430,7 @@ All deterministic Stage-2B test modules live under tests/server/services/tutor/e ## Final Plan Review and Handoff -- [ ] Confirm this follow-up changes only docs/plan-tutor-quality-stage-2b.md; the protected local ssot/ssot_function_tutor_model_registration.md remains untracked and unchanged. +- [ ] Confirm this follow-up changes only docs/archive/tutor-quality/plan-tutor-quality-stage-2b.md; the protected local ssot/ssot_function_tutor_model_registration.md remains untracked and unchanged. - [ ] Cross-check all planned requirements against the Stage-2B, Stage-2A, and Stage-1 SSOTs; resolve any conflict in favor of the applicable SSOT. - [ ] Confirm the Stage-2A Case Adapter is the only primary Tutor-generation path and the old Stage-A anchor corpus/runner/CLI remain untouched. - [ ] Confirm PR 2's deterministic/fake-Judge slice can complete with DG-1/DG-10 open, while FIRST_USEFUL_STAGE_2B requires PR 2 plus both gates and the bounded manual real-Judge smoke; model benchmark MVP arrives after PR 4. @@ -441,6 +441,6 @@ All deterministic Stage-2B test modules live under tests/server/services/tutor/e - [ ] Confirm each future PR is reviewable, testable with fake providers, and revertible without undoing another subsystem. - [ ] Confirm no real providers, Tutor prompts/strategies, implementation code, tests, Gold Case records, PR, or Stage 2C work were created as part of this plan task. - [ ] Run Node 24.20.0 as pinned by .nvmrc when available, then run only npm run check:docs and git diff --check for this documentation-only change. -- [ ] Stage and commit only docs/plan-tutor-quality-stage-2b.md with commit message docs(tutor-quality): refine stage 2b implementation plan. Do not push or open a PR. +- [ ] Stage and commit only docs/archive/tutor-quality/plan-tutor-quality-stage-2b.md with commit message docs(tutor-quality): refine stage 2b implementation plan. Do not push or open a PR. TUTOR_QUALITY_STAGE_2B_IMPLEMENTATION_PLAN_READY_FOR_REVIEW=YES diff --git a/ssot/ssot_function_definition_TutorQualityStage2B.md b/docs/archive/tutor-quality/ssot_function_definition_TutorQualityStage2B.md similarity index 100% rename from ssot/ssot_function_definition_TutorQualityStage2B.md rename to docs/archive/tutor-quality/ssot_function_definition_TutorQualityStage2B.md diff --git a/docs/tutor-course-content-capacity-2026-10-03.md b/docs/archive/tutor-quality/tutor-course-content-capacity-2026-10-03.md similarity index 100% rename from docs/tutor-course-content-capacity-2026-10-03.md rename to docs/archive/tutor-quality/tutor-course-content-capacity-2026-10-03.md diff --git a/docs/tutor-quality-answer-evaluation-analysis-2026-10-03.md b/docs/archive/tutor-quality/tutor-quality-answer-evaluation-analysis-2026-10-03.md similarity index 98% rename from docs/tutor-quality-answer-evaluation-analysis-2026-10-03.md rename to docs/archive/tutor-quality/tutor-quality-answer-evaluation-analysis-2026-10-03.md index f5902f47a..39bbb0491 100644 --- a/docs/tutor-quality-answer-evaluation-analysis-2026-10-03.md +++ b/docs/archive/tutor-quality/tutor-quality-answer-evaluation-analysis-2026-10-03.md @@ -4,8 +4,8 @@ Status: Analyse und Arbeitsplan, **nicht normativ**. Normativ sind `ssot/ssot_function_definition_TutorQualityEvaluation.md` (Regeln `R-…`) und `ssot/ssot_function_definition_LearningQuestions.md`. Datengrundlage sind die beiden realen Läufe vom 2026-10-03: -`docs/tutor-quality-baseline-2026-10-03.md` (im Folgenden „Baseline“, Corpus -v5) und `docs/tutor-quality-remediation-rerun-2026-10-03.md` („Nachlauf“, +`docs/archive/tutor-quality/tutor-quality-baseline-2026-10-03.md` (im Folgenden „Baseline“, Corpus +v5) und `docs/archive/tutor-quality/tutor-quality-remediation-rerun-2026-10-03.md` („Nachlauf“, Corpus v6), jeweils `openai-gpt5.4-mini` als Tutor, `openai-gpt5.5` als Judge, 3 Samples pro Case, Prompt-Revision `tutor-prompts-v1`. Ein geplanter Wochenlauf existierte bei Abschluss dieser Analyse noch nicht. diff --git a/docs/tutor-quality-answer-frame-rerun-2026-10-03.md b/docs/archive/tutor-quality/tutor-quality-answer-frame-rerun-2026-10-03.md similarity index 99% rename from docs/tutor-quality-answer-frame-rerun-2026-10-03.md rename to docs/archive/tutor-quality/tutor-quality-answer-frame-rerun-2026-10-03.md index abc8aebb1..e3b7af79b 100644 --- a/docs/tutor-quality-answer-frame-rerun-2026-10-03.md +++ b/docs/archive/tutor-quality/tutor-quality-answer-frame-rerun-2026-10-03.md @@ -1,7 +1,7 @@ # Tutor Quality – L3-Lauf Antwortbewertungsrahmen (Prompt v3) Status: Auswertung, **nicht normativ**. Prüft die in -`docs/tutor-quality-answer-evaluation-analysis-2026-10-03.md` §7 vorab +`docs/archive/tutor-quality/tutor-quality-answer-evaluation-analysis-2026-10-03.md` §7 vorab festgelegten Hypothesen H1–H5 für den Stack #141 (Prompt v2) und #142 (Prompt v3). Normativ sind `ssot/ssot_function_definition_TutorQualityEvaluation.md` und `ssot/ssot_function_definition_LearningQuestions.md`. Rohartefakte liegen diff --git a/docs/tutor-quality-baseline-2026-10-03.md b/docs/archive/tutor-quality/tutor-quality-baseline-2026-10-03.md similarity index 99% rename from docs/tutor-quality-baseline-2026-10-03.md rename to docs/archive/tutor-quality/tutor-quality-baseline-2026-10-03.md index 4c9edbcbf..782b9024a 100644 --- a/docs/tutor-quality-baseline-2026-10-03.md +++ b/docs/archive/tutor-quality/tutor-quality-baseline-2026-10-03.md @@ -1,6 +1,6 @@ # Tutor Quality – Real-Provider-Baseline (L3) -Status: Auswertung von PR E aus `docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`. +Status: Auswertung von PR E aus `docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`. Normativ ist `ssot/ssot_function_definition_TutorQualityEvaluation.md` (im Folgenden „SSOT“, Regeln als `R-…`). Dieser Bericht legt keine Verdict-Regel fest; Abschnitt 8 ist eine nicht-normative Empfehlung für PR F. diff --git a/docs/tutor-quality-freetutor-B-2026-10-07.md b/docs/archive/tutor-quality/tutor-quality-freetutor-B-2026-10-07.md similarity index 100% rename from docs/tutor-quality-freetutor-B-2026-10-07.md rename to docs/archive/tutor-quality/tutor-quality-freetutor-B-2026-10-07.md diff --git a/docs/tutor-quality-freetutor-C-2026-10-07.md b/docs/archive/tutor-quality/tutor-quality-freetutor-C-2026-10-07.md similarity index 98% rename from docs/tutor-quality-freetutor-C-2026-10-07.md rename to docs/archive/tutor-quality/tutor-quality-freetutor-C-2026-10-07.md index 87ad0246e..960bc5743 100644 --- a/docs/tutor-quality-freetutor-C-2026-10-07.md +++ b/docs/archive/tutor-quality/tutor-quality-freetutor-C-2026-10-07.md @@ -5,7 +5,7 @@ Status: Auswertung, **nicht normativ**. Normativ ist lokal unter `.tutor-quality-output/freetutor-C-2026-10-07/`. Die verdeckte Referenzbewertung steht in `evals/tutor-quality/human-ratings/freetutor-C-2026-10-07.md`; sie stammt von einem LLM-Evaluator, nicht von einer Person. Vorher-Werte stehen in -`docs/tutor-quality-freetutor-B-2026-10-07.md`. +`docs/archive/tutor-quality/tutor-quality-freetutor-B-2026-10-07.md`. ## 1. Lauf und Vergleichsbedingungen diff --git a/docs/tutor-quality-freetutor-D-2026-10-07.md b/docs/archive/tutor-quality/tutor-quality-freetutor-D-2026-10-07.md similarity index 98% rename from docs/tutor-quality-freetutor-D-2026-10-07.md rename to docs/archive/tutor-quality/tutor-quality-freetutor-D-2026-10-07.md index a56547397..0228ec1a8 100644 --- a/docs/tutor-quality-freetutor-D-2026-10-07.md +++ b/docs/archive/tutor-quality/tutor-quality-freetutor-D-2026-10-07.md @@ -4,7 +4,7 @@ Status: Auswertung, **nicht normativ**. Rohartefakte liegen nur lokal unter `.tutor-quality-output/freetutor-D-2026-10-07/`. Die verdeckte Referenzbewertung steht in `evals/tutor-quality/human-ratings/freetutor-D-2026-10-07.md`; sie stammt von einem LLM-Evaluator, nicht von einer Person. Vorläufer: -`docs/tutor-quality-freetutor-B-2026-10-07.md`, `…-C-2026-10-07.md`. +`docs/archive/tutor-quality/tutor-quality-freetutor-B-2026-10-07.md`, `…-C-2026-10-07.md`. ## 1. Lauf und Vergleichsbedingungen diff --git a/docs/tutor-quality-freetutor-E-2026-10-07.md b/docs/archive/tutor-quality/tutor-quality-freetutor-E-2026-10-07.md similarity index 97% rename from docs/tutor-quality-freetutor-E-2026-10-07.md rename to docs/archive/tutor-quality/tutor-quality-freetutor-E-2026-10-07.md index 342bf8305..537d6f992 100644 --- a/docs/tutor-quality-freetutor-E-2026-10-07.md +++ b/docs/archive/tutor-quality/tutor-quality-freetutor-E-2026-10-07.md @@ -4,7 +4,7 @@ Status: Auswertung, **nicht normativ**. Rohartefakte liegen nur lokal unter `.tutor-quality-output/freetutor-E-2026-10-07/`. Die verdeckte Referenzbewertung steht in `evals/tutor-quality/human-ratings/freetutor-E-2026-10-07.md`; sie stammt von einem LLM-Evaluator, nicht von einer Person. Vorläufer: -`docs/tutor-quality-freetutor-D-2026-10-07.md` (dort steht die vorab festgelegte +`docs/archive/tutor-quality/tutor-quality-freetutor-D-2026-10-07.md` (dort steht die vorab festgelegte Entscheidungsregel). ## 1. Lauf diff --git a/docs/tutor-quality-guidance-only-rerun-2026-10-03.md b/docs/archive/tutor-quality/tutor-quality-guidance-only-rerun-2026-10-03.md similarity index 98% rename from docs/tutor-quality-guidance-only-rerun-2026-10-03.md rename to docs/archive/tutor-quality/tutor-quality-guidance-only-rerun-2026-10-03.md index d72738773..db4e83a9b 100644 --- a/docs/tutor-quality-guidance-only-rerun-2026-10-03.md +++ b/docs/archive/tutor-quality/tutor-quality-guidance-only-rerun-2026-10-03.md @@ -3,7 +3,7 @@ Status: Auswertung, **nicht normativ**. Misst die Guidance-Änderung aus #141 getrennt (#146: Remediation/Klärung nennen ihren SSOT-Auslöser, sonst nichts) und grenzt sie gegen Prompt v1 und den Antwortbewertungsrahmen v3 ab -(`docs/tutor-quality-answer-frame-rerun-2026-10-03.md`). Rohartefakte liegen +(`docs/archive/tutor-quality/tutor-quality-answer-frame-rerun-2026-10-03.md`). Rohartefakte liegen nur lokal. ## 1. Lauf diff --git a/docs/tutor-quality-judge-smoke-2026-10-01.md b/docs/archive/tutor-quality/tutor-quality-judge-smoke-2026-10-01.md similarity index 100% rename from docs/tutor-quality-judge-smoke-2026-10-01.md rename to docs/archive/tutor-quality/tutor-quality-judge-smoke-2026-10-01.md diff --git a/docs/tutor-quality-next-architecture-2026-10-03.md b/docs/archive/tutor-quality/tutor-quality-next-architecture-2026-10-03.md similarity index 98% rename from docs/tutor-quality-next-architecture-2026-10-03.md rename to docs/archive/tutor-quality/tutor-quality-next-architecture-2026-10-03.md index 22613e0d8..388388278 100644 --- a/docs/tutor-quality-next-architecture-2026-10-03.md +++ b/docs/archive/tutor-quality/tutor-quality-next-architecture-2026-10-03.md @@ -1,8 +1,8 @@ # Tutor Quality – Neubewertung nach der Promptserie und nächster Architekturschritt Status: Analyse und Vorschlag, **nicht normativ**. Schließt die Promptserie -v1 / v3 / v4 ab (`docs/tutor-quality-answer-frame-rerun-2026-10-03.md`, -`docs/tutor-quality-guidance-only-rerun-2026-10-03.md`) und bewertet die +v1 / v3 / v4 ab (`docs/archive/tutor-quality/tutor-quality-answer-frame-rerun-2026-10-03.md`, +`docs/archive/tutor-quality/tutor-quality-guidance-only-rerun-2026-10-03.md`) und bewertet die verbleibenden Probleme neu. Alle Aussagen über den echten Course Content beziehen sich auf `MoDevIO/UnoSim-Examples` @ `2ac716f` (= `origin/main`) und wurden deterministisch mit dem echten `TutorService`, `CurriculumTutorAdapter`, diff --git a/docs/tutor-quality-remediation-rerun-2026-10-03.md b/docs/archive/tutor-quality/tutor-quality-remediation-rerun-2026-10-03.md similarity index 99% rename from docs/tutor-quality-remediation-rerun-2026-10-03.md rename to docs/archive/tutor-quality/tutor-quality-remediation-rerun-2026-10-03.md index bb9bf6e99..0532a723e 100644 --- a/docs/tutor-quality-remediation-rerun-2026-10-03.md +++ b/docs/archive/tutor-quality/tutor-quality-remediation-rerun-2026-10-03.md @@ -1,7 +1,7 @@ # Tutor Quality – Remediation-Nachlauf F1/F3 (L3) Status: Auswertung des Nachlaufs zu den Befunden F1 und F3 aus -`docs/tutor-quality-baseline-2026-10-03.md` (im Folgenden „Baseline“). Die +`docs/archive/tutor-quality/tutor-quality-baseline-2026-10-03.md` (im Folgenden „Baseline“). Die Baseline bleibt als historische Referenz unverändert. Normativ ist `ssot/ssot_function_definition_TutorQualityEvaluation.md` (im Folgenden „SSOT“, Regeln als `R-…`). Dieser Bericht legt keine Verdict-Regel fest; diff --git a/docs/audits/2026-09-25-tutor-strategy-conformance-audit.md b/docs/audits/2026-09-25-tutor-strategy-conformance-audit.md deleted file mode 100644 index 8773868d2..000000000 --- a/docs/audits/2026-09-25-tutor-strategy-conformance-audit.md +++ /dev/null @@ -1,147 +0,0 @@ -# Tutor Strategy Conformance Audit - -Date: 2026-09-25 -Baseline: `6f3676d40fb5ce5f8eab625d035b77f2f99b9016` -Stage-A SSOT commit: `cc516953` - -## Scope and conclusion - -This is a read-only audit of the merged Course Content Tutor implementation -against the strengthened Course Content and LearningQuestions SSOT. No -implementation or test file was modified for this audit. - -The implementation has one normalized strategy schema and resolves the -precedence metadata correctly. The conformance gap is behavioral: in the free -initial path `TutorService.generateQuestion` resolves a strategy but calls -`buildUserPrompt` without it (`server/services/tutor/tutor-service.ts:449-464`). -The free dialog path has the same problem: `generateDialogResponse` resolves a -strategy but calls `buildDialogPrompt` without it and uses hard-coded -remediation/progression/repetition behavior (`tutor-service.ts:472-503`). The -result is then decorated by `applyStrategyMetadata`, so a repository strategy -can be reported without having influenced the behavior-producing path. - -The deterministic planner consumes `questionKindWeights` only as a tie-break -in `selectQuestion` (`server/services/tutor/curriculum/learning-planner.ts:257-272`). -Its remediation, clarification, progression, scaffolding, and repetition -decisions are currently fixed policy. The remaining strategy fields never -reach planner or free-Tutor guidance. - -## Strategy conformance matrix - -`N/A` means the field has no operational decision before an initial question -exists under the SSOT. `PASS` means the current contract is deliberately -preserved, not that schema v1 supports an alternate behavior. - -| Strategy field | Planned initial | Planned dialog | Free initial | Free dialog | Existing tests | Status | -|---|---|---|---|---|---|---| -| `questionKindWeights` | PASS: `selectQuestion` ranks otherwise applicable candidates after difficulty | PARTIAL: fallback candidate selection uses weights, but scaffold/clarification/mastery branches bypass them | MISSING: `buildUserPrompt` receives no strategy | MISSING: `buildDialogPrompt` receives no strategy | `multi-topic.test.ts:55-61` tests only planner tie-break | PARTIAL | -| `sketchSpecificity` | MISSING: no planner read | MISSING: no planner read | MISSING: no free guidance | MISSING: no dialog guidance | schema-only coverage in `effective-tutor-strategy.test.ts` | MISSING | -| `repetition` | MISSING: `usedQuestionIds` is unconditional strict avoidance | MISSING: same unconditional ID policy; strategy is ignored | N/A | MISSING: `ensureDistinctDialogQuestion` always applies the same similarity threshold | repetition behavior is covered only as built-in hard-coded behavior in `tutor-service.test.ts` | MISSING | -| `remediation` | N/A | MISSING: `selectAfterRating` always calls `selectRemediation`; `chooseScaffold` always runs before question selection | N/A | MISSING: `getRemediationInstruction` is hard-coded scaffold-first | `curriculum.test.ts:96-110` tests only the default path | MISSING | -| `clarification` | N/A | MISSING: `selectClarification` always stays on the same indicator | N/A | MISSING: dialog prompt has no strategy-controlled aspect choice | no behavioral strategy test | MISSING | -| `progression` | N/A | MISSING: `selectAdvance` always tries a mastery probe before moving on | N/A | MISSING: `getProgressionInstruction` hard-codes mastery/advance behavior | `tutor-service.test.ts:278-302` tests only built-in behavior | MISSING | -| `scaffolding` | N/A | MISSING: `chooseScaffold` always prefers Topic content and has no generated-preference policy | N/A | MISSING: no strategy guidance or content/generated decision | `curriculum.test.ts:108-110` tests only content scaffold output | MISSING | -| `feedbackVerbosity` | N/A | MISSING: provider prompt does not receive the field | N/A | MISSING: provider prompt does not receive the field | no behavioral test | MISSING | -| `hintFirst` | N/A | MISSING: no strategy-controlled hint sequencing | N/A | MISSING: hard-coded prompt says to hint after weak answers | `effective-tutor-strategy.test.ts:9-30` is schema-only | MISSING | -| `adaptiveDifficulty` | PASS: `difficulty` is passed through the planner and the only schema-v1 profile is `current-contract` | PASS: same existing contract; no alternate profile is implemented | PASS: free prompt receives the existing difficulty value | PASS: same existing client/server current-contract behavior | `shared/tutor.ts:23-29,90-100` and existing difficulty tests | PASS by contract | - -## Exact path findings - -### Strategy resolution and metadata-only cases - -- `server/services/tutor/strategy/effective-tutor-strategy.ts:60-72` correctly - normalizes per-example, repository, and built-in candidates. -- `server/services/tutor/curriculum-tutor-adapter.ts:122-137` correctly - resolves repository/default and per-example strategy metadata when a valid - snapshot is present. -- `server/services/tutor/curriculum-tutor-adapter.ts:100-113` returns `null` - when no valid Topic plan exists. This is correct Topic behavior, but it - leaves TutorService on the free path. -- `server/services/tutor/tutor-service.ts:449-464` resolves the strategy and - then calls `buildUserPrompt(code, context, difficulty, planningResult)`. - The strategy does not enter the prompt. `applyStrategyMetadata` then adds - only ID/source to the result. -- `server/services/tutor/tutor-service.ts:476-503` repeats the same issue for - dialog. `buildDialogPrompt` and `ensureDistinctDialogQuestion` have no - strategy input; metadata is appended after provider/planned behavior. -- `tests/server/services/tutor/strategy-precedence.test.ts:68-99` asserts the - free path only by `strategyId` and `strategySource`; it would pass even if - two strategies produced identical prompts and dialog behavior. -- `tests/server/services/tutor/effective-tutor-strategy.test.ts:9-71` validates - schema values and resolution, not behavioral consumption. - -### Hard-coded free-Tutor policy - -- `getRemediationInstruction` (`tutor-service.ts:207-218`) always requests a - hint/scaffold after weak answers and escalates after fixed streak lengths. -- `getProgressionInstruction` (`tutor-service.ts:220-232`) always treats rating - 5 as mastery/advance, rating 4 as brief clarification then advance, and - rating 3 as same-path clarification. -- `ensureDistinctDialogQuestion` (`tutor-service.ts:290-307`) always applies - the same near-duplicate threshold and chooses hard-coded remediation or - concept-transition questions. -- `buildRemediationQuestion` and `buildConceptTransitionQuestion` - (`tutor-service.ts:238-287`) are application-owned safety helpers, but their - policy selection is not parameterized by the effective strategy. -- `buildUserPrompt` and `buildDialogPrompt` include difficulty and validated - didactic data, but no application-owned translation of strategy values. - -### Planner coverage - -- `DefaultLearningPlanner.selectQuestion` uses only `questionKindWeights` and - only after difficulty distance. This is a valid deterministic preference but - not a complete strategy implementation. -- `selectRemediation` always selects a content scaffold when one exists and - otherwise chooses a question; it does not distinguish scaffold-first from - question-first and does not apply `scaffolding` preference. -- `selectClarification` always searches the current indicator first; it does - not implement `new-indicator`. -- `selectAdvance` always calls `selectMasteryProbe`; it does not implement - `advance-immediately`. -- `collectUsedQuestionIds` and related selection always enforce strict - question-ID avoidance; `relaxed` is unused. -- `sketchSpecificity`, `feedbackVerbosity`, and `hintFirst` are not read by the - planner. They belong in application-owned provider/dialog guidance for - planned and free paths, while the deterministic planner handles only the - selection decisions that have structured Topic data. -- `adaptiveDifficulty` is correctly a literal schema value; the current - algorithm in `shared/tutor.ts` remains the single implementation with - deltas `-6/-3/0/+2/+4`, a four-rating window, clamping, and step limits. - -## Required scenario audit - -| Scenario | Resolution | Behavior result | Status | -|---|---|---|---| -| Arbitrary/local sketch + repository defaultStrategy | `resolveSnapshotStrategy` can select repository default | No Topic means `planInitial` returns null; free prompt omits strategy and only result metadata identifies it | NON-CONFORMANT, metadata-only | -| Repository strategy with zero Topics | Repository default resolves; `matchCourseContent` intentionally returns null for empty topics | Free Tutor remains available but does not consume repository strategy | NON-CONFORMANT | -| Valid Topics but no Topic matches | Strategy can resolve, matcher returns no plan | Free Tutor falls back without retaining strategy behavior | NON-CONFORMANT | -| Invalid Tutor capability | Adapter resolves built-in and disables Topic planning | Free behavior is hard-coded built-in policy, not the same EffectiveTutorStrategy translation path | PARTIAL; fallback works but shared path invariant fails | -| No repository | Built-in strategy resolves | Free behavior is hard-coded application policy and not consumed through the normalized strategy path | PARTIAL | -| Per-example strategy | Correctly wins when a valid binding and applicable Topic exist | Planner receives it only for implemented weights/branches; free/no-match prompt and dialog are metadata-only | PARTIAL | -| Examples-only repository | Built-in fallback resolves | Free Tutor remains functional; shared strategy behavior path is still missing | PARTIAL | - -## Smallest reasonable fix architecture - -1. Resolve one `StrategyResolution` in `TutorService` per request and pass the - normalized strategy into every behavior-producing call. Extend the planning - extension input with that already-resolved strategy so planner metadata and - free guidance cannot diverge. -2. Add one application-owned translation layer that maps only normalized enum - and numeric values to trusted guidance strings and dialog policy. Pass its - output to both initial and dialog prompts; never append repository text as - instructions. -3. Parameterize the existing safe dialog helpers with the effective strategy: - remediation (`scaffold-first`/`question-first`), clarification, progression, - repetition, scaffolding preference, feedback verbosity, and hint ordering. -4. Extend deterministic planner branches only where structured Topic data gives - a meaningful decision: weight ordering, strict/relaxed reuse, remediation - mode, clarification aspect, progression mode, and content/generated - scaffolding. Keep factual applicability, answerRating, difficulty, and - anti-loop safety application-owned. -5. Add behavioral tests that compare trusted prompt guidance or deterministic - planner output, not only strategy metadata, for repository-default, - strategy-only, no-match, built-in, invalid-capability, two-strategy, and - per-example cases. Add field-specific assertions for the policy branches. - -No external `ttbombadil/unosim-examples` checkout was present under the local -workspace search paths, so no external repository was read or modified. diff --git a/docs/superpowers/plans/2026-09-23-capacity-calibration.md b/docs/superpowers/plans/2026-09-23-capacity-calibration.md deleted file mode 100644 index bb18deb0c..000000000 --- a/docs/superpowers/plans/2026-09-23-capacity-calibration.md +++ /dev/null @@ -1,722 +0,0 @@ -# Host Capacity Calibration Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build a reproducible `npm run capacity:calibrate` command that measures the current UnoSim host, evaluates directly measured capacity, and emits review-only recommendations without changing production configuration. - -**Architecture:** Extract the existing real-Docker scenario measurement mechanics into a shared runner that returns raw observations without deciding pass/fail. Add a pure policy module for candidate planning and recommendations, a host/Docker probe module, a CLI orchestrator with bounded phases and cleanup, and report writers that serialize one versioned result object to JSON, Markdown, and an unapplied env fragment. - -**Tech Stack:** TypeScript, Node.js 24, Vitest, WebSocket client, Docker CLI, existing UnoSim status/receipt APIs, existing `tsx` scripts, and existing Capacity lifecycle tracking. - -**Spec:** `docs/superpowers/specs/2026-09-23-capacity-calibration-design.md` - -## Global Constraints - -- Run the implementation only on `feature/capacity-calibration`, based on merged `main`. -- Do not modify production Capacity defaults or application behavior. -- Do not modify the Tutor Registry SSOT. -- No recommendation may exceed a directly measured stable active or startup candidate. -- Classroom simulations use a fixed 60-second duration in version 1. -- `COMPILE_MAX_CONCURRENT` is reported as not calibrated unless a separate compile phase is actually run. -- Every non-dry-run load phase requires Docker health and usable CPU and memory safety signals. -- All owned processes and containers are cleaned in `finally`; unrelated containers are never stopped or removed. -- Calibration output is written below ignored `capacity-test-results/` and never applied automatically. - -## Review Focus - -1. **Direct-measurement boundary:** `capacity.env` must never contain an active or startup value that was not directly measured and stable; tests cover first-candidate-over-target downward refinement and no extrapolated recommendation. -2. **Validation independence:** the existing Capacity test retains its own assertions for status, lifecycle peak, polling peak, completion, and cleanup after using shared measurement helpers; a test mutating the shared aggregator cannot make assertions disappear. -3. **Timeout phase separation:** Docker-control, simulation queue, startup-slot wait, startup watchdog, and runtime timeout values remain separate in the effective output; tests assert each recommendation uses only its own measurements. -4. **Safety/deadline cleanup:** sustained CPU/memory/iowait violations, OOM, backend exit, Docker failure, and deadline truncation produce cleanup before returning partial output; tests use fake clocks/samples to exercise each branch. -5. **Reproducibility and environment gaps:** output includes Git SHA/dirty state, Docker/image fingerprints, effective settings, policy version, and explicit `null`/reduced-confidence values for optional host probes; tests assert missing optional probes do not become guessed values. - ---- - -### Task 1: Create the isolated branch baseline - -**Files:** -- Existing worktree: `/tmp/unosim-capacity-calibration` -- No product files changed in this task. - -**Interfaces:** -- Consumes: merged `main` at `bf68148ff71cc87743405148c3ff9a3a968794d3` and spec commit `c474a225`. -- Produces: clean `feature/capacity-calibration` worktree with dependencies available for focused tests. - -- [ ] **Step 1: Verify branch and clean state** - -```bash -git -C /tmp/unosim-capacity-calibration branch --show-current -git -C /tmp/unosim-capacity-calibration rev-parse HEAD -git -C /tmp/unosim-capacity-calibration status --short -``` - -Expected branch is `feature/capacity-calibration`, HEAD is `c474a225`, and status is clean. - -- [ ] **Step 2: Install dependencies without changing package manifests** - -```bash -npm ci -``` - -Run from `/tmp/unosim-capacity-calibration`; do not run `npm install` with manifest updates. - -- [ ] **Step 3: Run the existing unit baseline** - -```bash -npm run test:unit -``` - -Record the test count and confirm there are no pre-existing failures before adding implementation files. - ---- - -### Task 2: Add pure calibration policy types and tests - -**Files:** -- Create: `scripts/capacity-calibration-policy.ts` -- Test: `tests/server/capacity-calibration-policy.test.ts` - -**Interfaces:** - -```ts -export const CALIBRATION_POLICY_VERSION = "1" as const; - -export type CalibrationOptions = { - expectedUsers: number; - targetCpuPercent: number; - maxCpuPercent: number; - maxUserWaitSec: number; - maxDurationMin: number; - classroomDurationSec: 60; -}; - -export type ActiveMeasurement = { - requested: number; - stable: boolean; - cpuP95Percent: number | null; - cpuMaxPercent: number | null; - minAvailableMemoryBytes: number | null; - lifecycleDockerPeak: number; - pollingDockerPeak: number; - errors: string[]; - elapsedMs: number; -}; - -export type StartupMeasurement = { - requested: number; - stable: boolean; - startupSlotWaitP95Ms: number | null; - startupSlotWaitMaxMs: number | null; - startupDurationP95Ms: number | null; - cpuP95Percent: number | null; - iowaitP95Percent: number | null; - minAvailableMemoryBytes: number | null; - failures: number; - timeouts: number; -}; - -export type Recommendation = { - value: T | null; - status: "recommended" | "not-calibrated" | "infeasible" | "out-of-range"; - confidence: "HIGH" | "MEDIUM" | "LOW"; - measuredBasis: string[]; - warnings: string[]; -}; - -export function validateCalibrationOptions(input: Partial): CalibrationOptions; -export function planActiveCandidates(options: CalibrationOptions, logicalCpus: number): number[]; -export function planDownwardRefinement(firstExceeded: number, options: CalibrationOptions, logicalCpus: number): number[]; -export function selectActiveRecommendation(measurements: ActiveMeasurement[], options: CalibrationOptions): Recommendation; -export function selectStartupRecommendation(measurements: StartupMeasurement[], productionDefault: number, maxActive: number): Recommendation; -export function recommendDockerControlTimeout(p99Ms: number | null, productionDefault: number, range: { min: number; max: number }): Recommendation; -export function recommendSandboxStartSlotTimeout(p99Ms: number | null, maxObservedMs: number | null, productionDefault: number, range: { min: number; max: number }): Recommendation; -export function recommendQueueTimeout(queueP95Ms: number | null, queueP99Ms: number | null, queueMaxMs: number | null, maxUserWaitSec: number): Recommendation; -export function scoreCalibrationConfidence(measuredPhases: number, complete: boolean, safetyEvents: number): "HIGH" | "MEDIUM" | "LOW"; -``` - -- [ ] **Step 1: Write failing option-validation tests** - -Add tests for defaults (`expectedUsers=200`, CPU 75/85, wait 240 seconds, duration 30 minutes, classroom duration 60 seconds), rejection of zero/negative/non-finite values, rejection of `targetCpu >= maxCpu`, rejection of `maxCpu > 95`, and acceptance of decimal CPU thresholds. - -- [ ] **Step 2: Run the policy test file and verify the expected missing-export failures** - -```bash -npx vitest run --project=unit-node tests/server/capacity-calibration-policy.test.ts -``` - -The test must fail because the policy module and exports do not exist yet. - -- [ ] **Step 3: Add failing candidate-planning and active-selection tests** - -Cover host-sized bounded coarse candidates, expected-user and 128 caps, downward refinement when the first candidate exceeds target CPU, refinement only between measured bounds, stable lifecycle/polling requirements, target-CPU selection, and no recommendation when every candidate is unsafe. - -- [ ] **Step 4: Add failing startup and timeout-policy tests** - -Cover the smallest startup candidate within 10% of best p95, baseline-relative CPU/iowait/memory/error regressions, Docker-control recommendation below default, in-range recommendation, out-of-range recommendation without clamping, startup-slot formula, no-startup-data `not-calibrated`, queue technical minimum, and `technical > maxUserWait` producing `infeasible` with no value. - -- [ ] **Step 5: Implement the minimal pure policy functions** - -Implement validation, candidate planning, direct-measurement selection, downward refinement, startup baseline comparisons, timeout formulas, range status, and confidence without importing Docker, filesystem, process, or server modules. - -- [ ] **Step 6: Run policy tests to verify green** - -```bash -npx vitest run --project=unit-node tests/server/capacity-calibration-policy.test.ts -``` - -- [ ] **Step 7: Commit the policy unit** - -```bash -git add scripts/capacity-calibration-policy.ts tests/server/capacity-calibration-policy.test.ts -git commit -m "feat: add capacity calibration policy" -``` - ---- - -### Task 3: Add host and Docker probes - -**Files:** -- Create: `scripts/capacity-calibration-host.ts` -- Test: `tests/server/capacity-calibration-host.test.ts` - -**Interfaces:** - -```ts -export type HostProbe = { - required: { gitSha: string; gitDirty: boolean; nodeVersion: string; architecture: string; logicalCpus: number }; - optional: { physicalMemoryBytes: number | null; loadAverage: number[] | null; availableMemoryBytes: number | null; swapUsedBytes: number | null; iowaitPercent: number | null; thermalPressure: string | null }; - safetySignals: { cpuAvailable: boolean; memoryAvailable: boolean }; -}; - -export type DockerProbe = { - clientVersion: string; - serverVersion: string; - architecture: string; - cpus: number; - memoryBytes: number; - storageDriver: string; - daemonHealthy: boolean; - image: { reference: string; id: string; digest: string | null }; - runningContainers: Array<{ id: string; name: string; image: string; status: string; labels: Record }>; -}; - -export type ControlLatencySample = { command: string; condition: "idle" | "parallel"; durationsMs: number[]; error: string | null }; - -export async function collectHostProbe(deps?: HostProbeDependencies): Promise; -export async function collectDockerProbe(image: string, deps?: DockerProbeDependencies): Promise; -export async function measureDockerControlLatency(image: string, parallelism: number, samples: number, deps?: ControlDependencies): Promise; -``` - -- [ ] **Step 1: Write failing required/optional probe tests** - -Use injected command results to assert required Git/Node/architecture/CPU/Docker/image probes fail fast on missing data, while physical memory, load, swap, iowait, thermal, and process metrics become `null` without invented values. - -- [ ] **Step 2: Write failing Docker and control-latency tests** - -Cover Docker health failure, image ID/digest capture, unrelated container listing, idle versus parallel samples, per-command p50 input preservation, and command error capture. - -- [ ] **Step 3: Implement probe dependencies and platform adapters** - -Use `execFile`/`execFileSync` wrappers for Git and Docker. Use portable Node APIs first; use `sysctl`/`vm_stat` on macOS and `/proc`/`free`/`uptime` on Linux only for optional enrichment. Require at least one CPU and memory safety signal before non-dry-run load. - -- [ ] **Step 4: Run host-probe tests** - -```bash -npx vitest run --project=unit-node tests/server/capacity-calibration-host.test.ts -``` - -- [ ] **Step 5: Commit the probe unit** - -```bash -git add scripts/capacity-calibration-host.ts tests/server/capacity-calibration-host.test.ts -git commit -m "feat: collect reproducible calibration host probes" -``` - ---- - -### Task 4: Extract shared real-Docker measurement mechanics - -**Files:** -- Create: `scripts/capacity-scenario-runner.ts` -- Modify: `tests/server/capacity-validation.test.ts` -- Test: `tests/server/capacity-scenario-runner.test.ts` - -**Interfaces:** - -```ts -export type ScenarioKind = "burst" | "classroom"; - -export type CapacityScenarioOptions = { - baseUrl: string; - runId: string; - scenario: ScenarioKind; - clientCount: number; - holdDurationMs: number; - simulationTimeoutSec: number; - arrivalWindowMs?: number; - outputDir?: string; -}; - -export type ClientResult = { - clientId: number; - connected: boolean; - started: boolean; - requestedAtMs: number; - admittedAtMs: number | null; - queueEnteredAtMs: number | null; - simulationSlotAcquiredAtMs: number | null; - startupSlotWaitBeganAtMs: number | null; - startupSlotAcquiredAtMs: number | null; - startupBeganAtMs: number | null; - runtimeStartedAtMs: number | null; - completedAtMs: number | null; - disconnectedAtMs: number | null; - startLatencyMs: number | null; - queueWaitMs: number | null; - startupSlotWaitMs: number | null; - startupDurationMs: number | null; - runtimeDurationMs: number | null; - operationErrorCodes: string[]; - errors: string[]; -}; - -export type StatusSnapshot = { - capacityTestRunId?: string; - capacity?: { - simulation?: { maxConcurrent: number; active: number }; - sandboxStart?: { maxConcurrent: number; active: number; waiting: number; slotTimeoutMs: number }; - admission?: { max: number; current: number }; - queue?: { waiting: number; timeoutMs: number }; - compile?: { maxConcurrent: number; active: number }; - }; -}; - -export type EffectiveCapacityConfiguration = { - simulationMaxConcurrent: number; - sandboxStartMaxConcurrent: number; - simulationAdmissionMax: number; - simulationQueueTimeoutMs: number; - sandboxStartSlotTimeoutMs: number; - dockerControlTimeoutMs: number; - compileMaxConcurrent: number; -}; - -export type HostSample = { - atMs: number; - cpuPercent: number | null; - loadAverage: number | null; - availableMemoryBytes: number | null; - swapUsedBytes: number | null; - iowaitPercent: number | null; - runningDockerContainers: number; - capacityDockerContainers: number; -}; - -export type CleanupResult = { - backendExited: boolean; - remainingCapacityContainers: number; - activeSimulationCount: number | null; - queueWaiting: number | null; - admissionCurrent: number | null; - sandboxStartActive: number | null; - sandboxStartWaiting: number | null; -}; - -export type ScenarioRunnerDependencies = { - now?: () => number; - sleep?: (ms: number) => Promise; - getStatus?: (baseUrl: string) => Promise; - createSessionCookie?: (baseUrl: string) => Promise; - dockerContainerCount?: (runId: string, includeStopped?: boolean) => number; - sampleHost?: () => Promise; -}; - -export type CapacityScenarioMeasurement = { - clients: ClientResult[]; - statusHistory: StatusSnapshot[]; - runtimeConfiguration: EffectiveCapacityConfiguration; - lifecycleDockerPeak: number; - pollingDockerPeak: number; - activePeak: number; - queuePeak: number; - admissionPeak: number; - sandboxStartPeak: number; - sandboxStartWaitingPeak: number; - startupSlotWaitMs: number[]; - startupDurationMs: number[]; - queueWaitMs: number[]; - hostSamples: HostSample[]; - errors: string[]; - cleanup: CleanupResult; -}; - -export async function runCapacityScenario(options: CapacityScenarioOptions, deps?: ScenarioRunnerDependencies): Promise; -``` - -- [ ] **Step 1: Write failing aggregation tests** - -Feed deterministic client/status/event/sample fixtures and assert separate lifecycle and polling peaks, simulation queue versus startup-slot waits, fixed classroom duration metadata, percentile inputs, client phase timestamps, and raw observations. - -- [ ] **Step 2: Run the runner tests to verify the missing module failure** - -```bash -npx vitest run --project=unit-node tests/server/capacity-scenario-runner.test.ts -``` - -- [ ] **Step 3: Extract HTTP/WebSocket clients, status polling, lifecycle tracking, and cleanup** - -Move only reusable mechanics from `capacity-validation.test.ts` into the script module. Keep `capacity-validation.test.ts` responsible for `assertCapacityRuntimeMatches`, expected admission/rejection counts, exact success counts, peak assertions, timeout assertions, and leak assertions. - -- [ ] **Step 4: Add classroom arrival and fixed-duration handling** - -Use a 60-second simulation duration and spread client starts across the requested arrival window. Record admission, queue, slot, startup, runtime, completion, and error timestamps for every client. - -- [ ] **Step 5: Run shared-runner and existing Capacity validation tests** - -```bash -npx vitest run --project=unit-node tests/server/capacity-scenario-runner.test.ts tests/server/capacity-validation-config.test.ts -git diff --check -``` - -- [ ] **Step 6: Commit the measurement layer** - -```bash -git add scripts/capacity-scenario-runner.ts tests/server/capacity-scenario-runner.test.ts tests/server/capacity-validation.test.ts -git commit -m "refactor: share capacity scenario measurements" -``` - ---- - -### Task 5: Implement the calibration orchestration and owned backend lifecycle - -**Files:** -- Create: `scripts/calibrate-capacity.ts` -- Test: `tests/server/calibrate-capacity-cli.test.ts` - -**Interfaces:** - -```ts -export type CalibrationRunResult = { - schemaVersion: 1; - policyVersion: string; - policy: CalibrationOptions; - fingerprint: { gitSha: string; gitDirty: boolean; host: HostProbe; docker: DockerProbe; effectiveCapacity: EffectiveCapacityConfiguration }; - plan: { activeCandidates: number[]; startupCandidates: number[]; classroomDurationSec: 60 }; - phases: { dockerControl: ControlLatencySample[]; active: ActiveMeasurement[]; startup: StartupMeasurement[]; classroom: ClassroomMeasurement | null }; - recommendations: CalibrationRecommendations; - safetyEvents: SafetyEvent[]; - partial: boolean; - stopReason: string | null; - cleanup: CleanupResult; -}; - -export type SafetyEvent = { - atMs: number; - phase: "preflight" | "control" | "active" | "startup" | "classroom" | "cleanup"; - kind: "cpu" | "memory" | "iowait" | "oom" | "docker" | "backend" | "deadline" | "cleanup"; - message: string; - immediate: boolean; -}; - -export type CalibrationCliConfig = CalibrationOptions & { - outputDir: string; - skipClassroom: boolean; - skipStartupTuning: boolean; - dryRun: boolean; - verbose: boolean; -}; - -export type CalibrationRecommendations = { - simulationMaxConcurrent: Recommendation; - sandboxStartMaxConcurrent: Recommendation; - simulationAdmissionMax: Recommendation; - simulationQueueTimeoutMs: Recommendation; - sandboxStartSlotTimeoutMs: Recommendation; - dockerControlTimeoutMs: Recommendation; - compileMaxConcurrent: Recommendation; -}; - -export type ClassroomMeasurement = CapacityScenarioMeasurement & { - queueP50Ms: number | null; - queueP95Ms: number | null; - queueP99Ms: number | null; - queueMaxMs: number | null; - startupSlotWaitP50Ms: number | null; - startupSlotWaitP95Ms: number | null; - startupSlotWaitP99Ms: number | null; - startupSlotWaitMaxMs: number | null; - completed: number; - failed: number; - fairness: { starvation: boolean; reorderPercentage: number | null; outliers: number }; -}; - -export type CalibrationDependencies = { - now?: () => number; - collectHostProbe?: () => Promise; - collectDockerProbe?: (image: string) => Promise; - measureDockerControlLatency?: (image: string, parallelism: number, samples: number) => Promise; - runScenario?: (options: CapacityScenarioOptions) => Promise; - startBackend?: (capacity: EffectiveCapacityConfiguration, runId: string) => Promise<{ baseUrl: string; stop: () => Promise }>; -}; - -export async function parseCalibrationArgs(argv: string[]): Promise; -export async function runCalibration(config: CalibrationCliConfig, deps?: CalibrationDependencies): Promise; -``` - -- [ ] **Step 1: Write failing CLI parser tests** - -Cover defaults, every supported flag, unknown/missing values, invalid ranges, `--dry-run`, and printing the effective policy before load. - -- [ ] **Step 2: Write failing orchestration tests with fake probes and runner** - -Cover phase ordering, deadline truncation, first-candidate-over-target downward probing, safe escalation stop, startup skip, classroom skip, backend cleanup on thrown errors, and no `capacity.env` application. - -- [ ] **Step 3: Implement argument parsing and policy printing** - -Use a strict parser with status 2 for invalid input. Set fixed classroom duration to 60 seconds in the configuration object and include it in the printed policy. - -- [ ] **Step 4: Implement owned backend startup** - -Start `tsx server/index.ts` on an allocated loopback port with `NODE_ENV=test`, Docker mode, `CAPACITY_TEST_RUN_ID`, semantic environment overrides, and the measured sandbox image. Poll `/api/readiness`, fetch `/api/status`, and abort if runtime settings do not match the requested phase. - -- [ ] **Step 5: Implement phase orchestration and safety deadline** - -Run control probes, active candidates, startup candidates, and classroom phase in order. Use an absolute deadline, sustained host samples, `AbortController` cancellation, and `finally` cleanup that terminates only the owned backend and exact run-ID containers. - -- [ ] **Step 6: Run CLI/orchestration tests** - -```bash -npx vitest run --project=unit-node tests/server/calibrate-capacity-cli.test.ts -``` - -- [ ] **Step 7: Commit the orchestrator** - -```bash -git add scripts/calibrate-capacity.ts tests/server/calibrate-capacity-cli.test.ts -git commit -m "feat: add host capacity calibration command" -``` - ---- - -### Task 6: Add versioned report and env-fragment writers - -**Files:** -- Create: `scripts/capacity-calibration-report.ts` -- Test: `tests/server/capacity-calibration-report.test.ts` -- Modify: `scripts/calibrate-capacity.ts` - -**Interfaces:** - -```ts -export function writeCalibrationArtifacts(result: CalibrationRunResult, outputDir: string): Promise<{ jsonPath: string; markdownPath: string; envPath: string }>; -export function renderCalibrationMarkdown(result: CalibrationRunResult): string; -export function renderCapacityEnv(result: CalibrationRunResult): string; -``` - -- [ ] **Step 1: Write failing report tests** - -Assert schema/policy version, reproducibility fingerprints, raw phase measurements, confidence/reason fields, measured/technical/UX timeout columns, partial-run and safety sections, and omission of uncalibrated or out-of-range values from `capacity.env`. - -- [ ] **Step 2: Implement JSON, Markdown, and review-only env rendering** - -Serialize the exact result object without recomputing policy. Include comments in `capacity.env` explaining that it is unapplied and list only directly measured recommendations. - -- [ ] **Step 3: Integrate artifact writing in success, partial, and failure paths** - -Write reports after cleanup for normal completion, safety stop, deadline truncation, and pre-load dry runs. Preserve the original error in the report while returning a nonzero exit status for unsafe/preflight failures. - -- [ ] **Step 4: Run report tests** - -```bash -npx vitest run --project=unit-node tests/server/capacity-calibration-report.test.ts -``` - -- [ ] **Step 5: Commit report generation** - -```bash -git add scripts/capacity-calibration-report.ts scripts/calibrate-capacity.ts tests/server/capacity-calibration-report.test.ts -git commit -m "feat: generate reviewable capacity calibration reports" -``` - ---- - -### Task 7: Add the npm command and authoritative documentation - -**Files:** -- Modify: `package.json` -- Modify: `docs/CAPACITY_VALIDATION_PLAN.md` -- Test: `tests/server/calibrate-capacity-cli.test.ts` - -**Interfaces:** -- `npm run capacity:calibrate -- --expected-users 200` invokes `tsx scripts/calibrate-capacity.ts`. -- Documentation describes Calibrate → Review → Apply and never instructs automatic application. - -- [ ] **Step 1: Add the package script** - -Add exactly: - -```json -"capacity:calibrate": "tsx scripts/calibrate-capacity.ts" -``` - -Do not change production environment defaults or existing Capacity scripts. - -- [ ] **Step 2: Add documentation tests/fixtures for command and output names** - -Assert that the documented command, `capacity-calibration.json`, `capacity-calibration.md`, `capacity.env`, fixed 60-second classroom duration, and review-only workflow are present. - -- [ ] **Step 3: Extend the authoritative Capacity plan** - -Document measured versus inferred values, direct-measurement limits, policy versions, host/Docker probes, safety behavior, global duration, optional compile calibration, report review, and rerun triggers after hardware/runtime/resource/workload changes. - -- [ ] **Step 4: Run documentation checks** - -```bash -npm run check:docs -git diff --check -``` - -- [ ] **Step 5: Commit command and documentation** - -```bash -git add package.json docs/CAPACITY_VALIDATION_PLAN.md tests/server/calibrate-capacity-cli.test.ts -git commit -m "docs: document capacity calibration workflow" -``` - ---- - -### Task 8: Run focused tests and a dry-run smoke test - -**Files:** -- No new files; inspect all files created above. - -- [ ] **Step 1: Run all calibration-focused tests** - -```bash -npx vitest run --project=unit-node \ - tests/server/capacity-calibration-policy.test.ts \ - tests/server/capacity-calibration-host.test.ts \ - tests/server/capacity-scenario-runner.test.ts \ - tests/server/calibrate-capacity-cli.test.ts \ - tests/server/capacity-calibration-report.test.ts -``` - -- [ ] **Step 2: Run the dry-run CLI** - -```bash -npm run capacity:calibrate -- --expected-users 200 --dry-run --output-dir /tmp/unosim-capacity-calibration-dry-run -``` - -Verify it prints the policy, probes metadata, writes all three artifacts, starts no backend, creates no Docker containers, and does not modify any environment/configuration file. - -- [ ] **Step 3: Run the existing Capacity unit/configuration tests** - -```bash -npx vitest run --project=unit-node \ - tests/server/capacity-validation-config.test.ts \ - tests/server/capacity-docker-events.test.ts \ - tests/server/config.test.ts \ - tests/server/routes/server-status.test.ts -``` - -- [ ] **Step 4: Review generated output and cleanup** - -Inspect the JSON/Markdown/env files, verify the env fragment is unapplied, remove only the temporary `/tmp` dry-run directory, and confirm no Docker resources remain. - ---- - -### Task 9: Run the small real-Docker calibration smoke test - -**Files:** -- No source changes; generated output stays under ignored `capacity-test-results/`. - -- [ ] **Step 1: Verify Docker and image prerequisites** - -```bash -docker info -docker image inspect unosim-sandbox:latest -``` - -If either prerequisite is unavailable, report the smoke test as blocked without changing code. - -- [ ] **Step 2: Run a bounded calibration smoke** - -```bash -npm run capacity:calibrate -- \ - --expected-users 20 \ - --target-cpu 75 \ - --max-cpu 85 \ - --max-user-wait 120 \ - --max-duration 5 \ - --skip-classroom \ - --output-dir capacity-test-results/calibration-smoke -``` - -Use the same workload and run-ID cleanup as the production command; do not run a 30-minute Dell campaign during development. - -- [ ] **Step 3: Verify smoke output and cleanup** - -Confirm JSON/Markdown/env artifacts contain fingerprints and measured-only recommendations, then run: - -```bash -docker ps -a --filter label=unosim.capacity-test-run-id -git status --short -``` - -No owned containers or backend process may remain; generated results remain ignored. - ---- - -### Task 10: Full validation and final audit - -**Files:** -- All implementation files from Tasks 2–7. - -- [ ] **Step 1: Run required repository checks** - -```bash -npm run check -npm run test:unit -npm run build -npm run test:integration -npm run check:docs -git diff --check -``` - -- [ ] **Step 2: Run the calibration-focused tests again after the full suite** - -```bash -npx vitest run --project=unit-node \ - tests/server/capacity-calibration-policy.test.ts \ - tests/server/capacity-calibration-host.test.ts \ - tests/server/capacity-scenario-runner.test.ts \ - tests/server/calibrate-capacity-cli.test.ts \ - tests/server/capacity-calibration-report.test.ts -``` - -- [ ] **Step 3: Audit changed and untracked files** - -```bash -git status --short -git diff --stat bf68148ff71cc87743405148c3ff9a3a968794d3 -git diff --check -rg -n "DELL|UnoSim_Test|192\\.168\\.|capacity.env.*apply|SIMULATION_MAX_CONCURRENT=70|SANDBOX_START_MAX_CONCURRENT=20" scripts tests docs package.json -``` - -Remove only accidental debug output, hostnames/IPs, credentials, or generated artifacts introduced by this feature. Do not touch the Tutor Registry SSOT. - -- [ ] **Step 4: Verify production defaults remain unchanged** - -```bash -rg -n "SIMULATION_MAX_CONCURRENT|SANDBOX_START_MAX_CONCURRENT|SIMULATION_ADMISSION_MAX|SIMULATION_QUEUE_TIMEOUT_MS|SANDBOX_START_SLOT_TIMEOUT_MS|DOCKER_CONTROL_TIMEOUT_MS" server/config.ts docker-compose.yml tests/deployment/docker-compose.gateway.yml -``` - -Compare values with the merged-main baseline and confirm calibration code uses test-only overrides rather than editing these files. - -- [ ] **Step 5: Create the final implementation commits** - -Keep the spec commits and implementation commits separate. Use the logical implementation messages already defined in Tasks 2–7; do not commit calibration result artifacts. - -- [ ] **Step 6: Report readiness without merging** - -Report branch HEAD, commits, files, CLI usage, policy rules, validation results, smoke result, and any blocked Dell validation. Do not merge to `main`. diff --git a/docs/superpowers/plans/2026-09-24-course-content-tutor-control.md b/docs/superpowers/plans/2026-09-24-course-content-tutor-control.md deleted file mode 100644 index 198ba06ed..000000000 --- a/docs/superpowers/plans/2026-09-24-course-content-tutor-control.md +++ /dev/null @@ -1,260 +0,0 @@ -# Course Content Tutor Control Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Implement the Stage-A unified Course Content contract so one immutable repository/ref snapshot controls Examples and the optional repository Tutor capability, with capability-scoped validation, effective strategy selection, topic binding, and server-authoritative session pinning. - -**Architecture:** Extend the existing Examples revision loader and cache into the sole Course Content loader. The loader always validates the Examples core independently, then attempts the optional Tutor capability as an all-or-nothing bundle; Tutor planning consumes an immutable snapshot context and falls back to the existing free Tutor path plus the normative built-in strategy. Browser metadata is request context only; the server derives the active revision from the existing source-selection/cache path and issues the session pin needed to keep follow-up dialogs on that revision. - -**Tech Stack:** TypeScript, Zod, YAML, Express, React, Vitest, existing Examples source/revision cache, existing Tutor service/planner, npm scripts. - -**Spec:** `ssot/ssot_function_definition_CourseContent.md`, `ssot/ssot_function_definition_LearningQuestions.md`, `ssot/ssot_function_definition_ExternalExamples.md`, `docs/adr/0006-unified-course-content-and-tutor-strategy.md`, `docs/adr/0004-repository-based-tutor-curriculum.md`, `docs/ARCHITECTURE.md`, approved in Stage-A commit `526c97c967bcf7695197ba175a1d86927ff15726`. - -## Global Constraints - -- Keep one repository/ref selection and one Course Content fetch stack for Examples and Tutor. -- Schema-v1 manifests remain Examples-only compatible; schema-v2 adds only the optional Tutor descriptor. -- Validate Examples core and Tutor capability separately; a Tutor-only failure preserves valid Examples and activates no partial Tutor bundle. -- Use the exact normative built-in strategy `built-in-default` with weights `recall 10`, `concept 25`, `application 35`, `prediction 15`, `transfer 15`. -- Preserve current LearningQuestions adaptive semantics: ratings `1:-6`, `2:-3`, `3:0`, `4:+2`, `5:+4`, current window/clamp/step limits. -- Strategy precedence is per-example, then repository default, then built-in; the built-in must use the same effective-strategy path. -- Topic precedence is applicable active-example primary topic, other bound topics, fact-matched repository topics, then free Tutor. -- Repository/ref/revision/example values from the browser are untrusted metadata; the server validates or derives immutable Course Content context. -- Tutor content is usable only from the active Course Content revision; no stale cross-revision substitution. -- No repository-controlled system prompt, no lecturer lock, no second Tutor source selector, no Capacity changes, and no Tutor Registry SSOT changes. -- Preserve API-key privacy, difficulty, dialog, and existing free-Tutor behavior. - -## Review Focus - -- A malformed Tutor descriptor, manifest, topic, strategy, hash, cross-reference, or per-example Tutor reference must not discard valid Examples or partially activate Tutor files. Test in Tasks 1, 4, and 5. -- A schema-v1 root with no Tutor descriptor must load unchanged and use free Tutor/built-in behavior. Test in Tasks 1 and 5. -- A browser-supplied revision that is valid-looking but not the server-resolved active revision must not select content or cross-pin a dialog. Test in Task 8. -- An external example moved from revision A to B must reset the dialog and never use A's Tutor content for B. Test in Tasks 7 and 8. -- The no-repository and no-topic paths must remain fully functional and must not leak credentials into Course Content metadata or prompts. Test in Tasks 5, 8, and 11. - ---- - -### Task 1: Add capability-scoped Course Content schemas - -**Files:** -- Create: `server/services/course-content/course-content-schema.ts` -- Modify: `server/services/examples/examples-schema.ts` -- Test: `tests/server/services/course-content/course-content-schema.test.ts` -- Test: `tests/server/services/examples/examples-schema.test.ts` - -**Interfaces:** -- Consumes: existing `ManifestExample`, `ExamplesManifest`, `validateManifestReferences`. -- Produces: `courseContentManifestSchema`, `tutorDescriptorSchema`, `exampleTutorBindingSchema`, `courseContentTopicEntrySchema`, `courseContentStrategyEntrySchema`, and typed validation results that distinguish `examples` from `tutor` capability errors. - -- [ ] **Step 1: Write the failing tests** for v1 compatibility, v2 optional Tutor descriptor, closed Tutor descriptor fields, per-example binding shape, and independent capability result. -- [ ] **Step 2: Run** `npx vitest run tests/server/services/course-content/course-content-schema.test.ts tests/server/services/examples/examples-schema.test.ts`; verify the new exports/tests fail before implementation. -- [ ] **Step 3: Implement** the strict v1/v2 root parsing and capability-scoped result. Keep existing Example field validation and reference validation unchanged for the Examples result. Permit only `tutor.manifest` at root and only `topics`, `primaryTopic`, and `strategy` in an Example Tutor binding. -- [ ] **Step 4: Run** the focused tests and the existing Examples service tests; verify v1 parses without Tutor and malformed Tutor data reports only Tutor invalidity. -- [ ] **Step 5: Commit** `feat: add course content manifest schemas`. - -### Task 2: Add EffectiveTutorStrategy and the normative built-in - -**Files:** -- Create: `server/services/tutor/strategy/effective-tutor-strategy.ts` -- Modify: `server/services/tutor/curriculum/curriculum-schema.ts` -- Modify: `shared/tutor.ts` -- Test: `tests/server/services/tutor/effective-tutor-strategy.test.ts` -- Test: `tests/shared/tutor.test.ts` - -**Interfaces:** -- Consumes: Stage-A strategy contract and existing `TutorDifficulty`/`calculateNextTutorDifficulty` behavior. -- Produces: `effectiveTutorStrategySchema`, `BUILT_IN_TUTOR_STRATEGY`, `resolveEffectiveTutorStrategy(input)`, and typed strategy metadata for planner/service responses. - -- [ ] **Step 1: Write failing tests** for the exact built-in values, strict strategy schema rejection of prompts/URLs/executable fields, precedence resolution, and unchanged adaptive deltas. -- [ ] **Step 2: Run** the focused tests and verify missing strategy exports fail. -- [ ] **Step 3: Implement** the closed strategy schema and resolver. Set `BUILT_IN_TUTOR_STRATEGY.id` to `built-in-default`, use weights 10/25/35/15/15, and encode the approved Socratic/scaffolding/progression values. Represent adaptive difficulty as `current-contract`; do not introduce a second delta implementation. -- [ ] **Step 4: Run** focused Tutor/shared tests and verify all current difficulty contract tests remain green. -- [ ] **Step 5: Commit** `feat: add effective tutor strategy policy`. - -### Task 3: Generalize the Tutor curriculum to multiple topics - -**Files:** -- Modify: `server/services/tutor/curriculum/curriculum-schema.ts` -- Modify: `server/services/tutor/curriculum/topic-matcher.ts` -- Modify: `server/services/tutor/curriculum/learning-planner.ts` -- Modify: `server/services/tutor/curriculum/sketch-facts.ts` only where shared fact types need to be exported without semantic changes -- Test: `tests/server/services/tutor/curriculum.test.ts` -- Test: `tests/server/services/tutor/learning-planner.test.ts` - -**Interfaces:** -- Consumes: `EffectiveTutorStrategy` from Task 2 and current topic/planner contracts. -- Produces: generic topic IDs in manifest/topic schemas, validated multi-topic collections, deterministic topic matching, and planner selection that accepts a selected topic plus effective strategy without changing existing LearningQuestions difficulty semantics. - -- [ ] **Step 1: Add failing tests** for two valid topic entries, duplicate topic IDs, unknown topic references, per-topic cross-reference errors, fact matching across multiple topics, and deterministic strategy-weighted question selection. -- [ ] **Step 2: Run** the focused curriculum tests and verify the current literal/single-topic implementation fails the new cases. -- [ ] **Step 3: Implement** generic topic IDs, manifest uniqueness, per-topic validation, and deterministic multi-topic selection. Apply strategy question-kind weights only as a tie-break/selection policy and preserve the existing difficulty distance and progression/remediation rules. -- [ ] **Step 4: Run** curriculum and planner tests, including existing pilot fixtures, and verify the existing `onRating` contract still maps 1–2/3/4/5 as documented. -- [ ] **Step 5: Commit** `feat: generalize tutor curriculum topics`. - -### Task 4: Extend immutable Examples snapshots with optional Tutor capability - -**Files:** -- Modify: `server/services/examples/http-provider.ts` -- Modify: `server/services/examples/examples-cache.ts` -- Modify: `server/services/examples/source-provider.ts` -- Create: `server/services/course-content/course-content-loader.ts` -- Test: `tests/server/services/examples/http-provider.test.ts` -- Test: `tests/server/services/examples/source-provider.test.ts` -- Test: `tests/server/services/course-content/course-content-loader.test.ts` - -**Interfaces:** -- Consumes: Tasks 1–3 schemas and existing `RevisionProvider`, `ExamplesCache`, and source/ref resolution. -- Produces: `CourseContentSnapshot` carrying repository, immutable revision, Examples, optional validated Tutor capability/status, and content size; the existing Examples revision loader remains the only outbound fetch path. - -- [ ] **Step 1: Write failing loader tests** for schema-v1, valid schema-v2, missing optional Tutor manifest, all Tutor files loaded from the same revision, and no extra source/fetch repository. -- [ ] **Step 2: Run** the focused loader/provider tests and verify the snapshot type/loader behavior is absent. -- [ ] **Step 3: Implement** the optional Tutor load under the immutable Examples base. Fetch the descriptor-referenced `tutor/manifest.yaml`, then only enumerated strategy/topic paths, verify SHA-256, enforce existing bounded fetch/concurrency/total limits, and return an all-or-nothing Tutor capability status while retaining valid Examples. -- [ ] **Step 4: Run** the loader/cache/source tests and verify revision cache entries carry one combined snapshot and retain source pinning/eviction behavior. -- [ ] **Step 5: Commit** `feat: load tutor capability with course snapshots`. - -### Task 5: Implement capability-scoped Tutor validation and complete fallback - -**Files:** -- Modify: `server/services/tutor/curriculum/content-repository.ts` or retire it from the production composition root after migrating its tests to the Course Content snapshot -- Modify: `server/services/tutor/curriculum-tutor-adapter.ts` -- Modify: `server/services/tutor/tutor-service-factory.ts` -- Modify: `server/services/tutor/tutor-service.ts` -- Test: `tests/server/services/tutor/curriculum.test.ts` -- Test: `tests/server/services/tutor/tutor-service.test.ts` -- Test: `tests/server/services/tutor/fallback-matrix.test.ts` - -**Interfaces:** -- Consumes: `CourseContentSnapshot` from Task 4, `BUILT_IN_TUTOR_STRATEGY`, and the existing app-owned `TUTOR_SYSTEM_PROMPT`. -- Produces: adapter/service behavior implementing fallback matrix A–G: valid Examples survive Tutor-only invalidity; invalid/missing Tutor uses built-in plus free Tutor; no partial Tutor bundle activates; no repository content is required for free Tutor. - -- [ ] **Step 1: Write failing matrix tests** for absent repository, schema-v1, valid Tutor, invalid descriptor, invalid manifest, invalid topic/strategy/reference/hash, no topic match, and valid Examples with Tutor capability rejected. -- [ ] **Step 2: Run** the fallback matrix tests and verify the current separate repository path cannot satisfy the matrix. -- [ ] **Step 3: Implement** the adapter against a supplied Course Content snapshot/context, returning `null` for unavailable/invalid Tutor capability and leaving the service’s free Tutor generation path intact. Keep system instructions app-owned; pass repository data as validated context only. -- [ ] **Step 4: Run** matrix, service, route, and existing offline/free Tutor tests; verify no credential or arbitrary browser revision is accepted by this layer. -- [ ] **Step 5: Commit** `feat: isolate tutor capability fallback`. - -### Task 6: Apply strategy precedence through one effective-strategy path - -**Files:** -- Modify: `server/services/tutor/curriculum-tutor-adapter.ts` -- Modify: `server/services/tutor/tutor-planning.ts` -- Modify: `server/services/tutor/curriculum/learning-planner.ts` -- Modify: `server/services/tutor/tutor-service.ts` -- Test: `tests/server/services/tutor/strategy-precedence.test.ts` -- Test: `tests/server/services/tutor/tutor-planning.test.ts` - -**Interfaces:** -- Consumes: per-example binding, repository default strategy, built-in resolver, and existing plan metadata. -- Produces: plans with `strategyId`/strategy source from the resolved effective strategy; precedence exactly per-example → repository default → built-in for both repository and free Tutor paths. - -- [ ] **Step 1: Write failing tests** for each precedence branch, invalid per-example reference fallback, repository strategy selection, and built-in free-Tutor strategy metadata. -- [ ] **Step 2: Run** the focused tests and verify current scaffold IDs are incorrectly reported as strategy IDs or no strategy is attached. -- [ ] **Step 3: Implement** strategy resolution before planning; preserve scaffold IDs separately from strategy IDs and use the same normalized object for planning and response metadata. -- [ ] **Step 4: Run** focused planning/service tests and verify adaptive difficulty is still calculated by the existing shared function. -- [ ] **Step 5: Commit** `feat: apply tutor strategy precedence`. - -### Task 7: Implement example binding and topic-selection precedence - -**Files:** -- Modify: `server/services/tutor/curriculum-tutor-adapter.ts` -- Modify: `server/services/tutor/curriculum/topic-matcher.ts` -- Modify: `server/services/examples/examples-repository.ts` -- Modify: `shared/examples.ts` -- Test: `tests/server/services/tutor/topic-precedence.test.ts` -- Test: `tests/server/services/examples/examples-repository.test.ts` - -**Interfaces:** -- Consumes: current external example identity and validated snapshot metadata. -- Produces: topic selection in the exact order active example primary topic, other bound topics, fact matches, free path; catalog/detail metadata that identifies the same Course Content revision without exposing credentials. - -- [ ] **Step 1: Write failing tests** for primary topic, bound-topic ordering, fact match fallback, no-match free Tutor, missing/invalid binding, and example detail/catalog revision consistency. -- [ ] **Step 2: Run** focused Examples/Tutor tests and verify current adapter always chooses the first topic. -- [ ] **Step 3: Implement** explicit binding selection and retain `DefaultTopicMatcher` only for the fact-match phase. Reject binding to absent topics as Tutor capability failure rather than activating a partial bundle. -- [ ] **Step 4: Run** focused tests and verify schema-v1 examples still select free Tutor. -- [ ] **Step 5: Commit** `feat: bind tutor topics to examples`. - -### Task 8: Make revision and dialog context server-authoritative - -**Files:** -- Modify: `shared/tutor.ts` -- Modify: `server/routes/tutor.routes.ts` -- Modify: `server/services/tutor/tutor-service.ts` -- Modify: `server/services/tutor/tutor-service-factory.ts` -- Modify: `server/services/examples/source-provider.ts` -- Modify: `client/src/hooks/use-tutor.ts` -- Modify: `client/src/lib/external-examples.ts` -- Modify: `client/src/components/simulator/ArduinoSimulatorPageLayout.tsx` -- Test: `tests/server/routes/tutor.routes.test.ts` -- Test: `tests/server/services/tutor/revision-pinning.test.ts` -- Test: `tests/client/tutor.test.tsx` -- Test: `tests/client/external-examples.test.tsx` - -**Interfaces:** -- Consumes: source/ref selection and combined snapshots from Tasks 4 and 7. -- Produces: validated Course Content context/session handle for question and dialog requests; active dialog pinning; context change reset; server rejection/ignoring of arbitrary client SHA/repository metadata. - -- [ ] **Step 1: Write failing tests** for A→A follow-up, A→B reset, client SHA mismatch, unknown revision, and browser override affecting both Examples and Tutor through the existing one selector. -- [ ] **Step 2: Run** focused route/service/client tests and verify current Tutor routes accept no authoritative context contract. -- [ ] **Step 3: Implement** a server-derived context token/handle or equivalent validated request context. Bind initial external-example context to the resolved revision, require the same context for dialog follow-ups, and reset client dialog state when the Course Content source/ref/example context changes. -- [ ] **Step 4: Run** focused revision/dialog tests and verify credential fields remain request-only and no revision from the browser is trusted. -- [ ] **Step 5: Commit** `feat: pin tutor dialogs to course content revisions`. - -### Task 9: Remove active separate Tutor source and add migration tombstones - -**Files:** -- Modify: `server/config.ts` -- Modify: `server/services/tutor/curriculum/content-repository.ts` -- Modify: `server/services/tutor/tutor-service-factory.ts` -- Modify: `.env.example` or the repository’s existing environment documentation if it lists obsolete Tutor variables -- Test: `tests/server/config.test.ts` -- Test: `tests/server/services/tutor/curriculum.test.ts` - -**Interfaces:** -- Consumes: the unified Course Content composition root from Tasks 4–8. -- Produces: no active `UNOSIM_TUTOR_CURRICULUM_*` source/refresh/fetch configuration; every known obsolete variable fails fast with a migration message, while the free Tutor and Course Content path remain available. - -- [ ] **Step 1: Write failing tests** for each exact old variable tombstone and for production composition not constructing `GitHubDidacticContentRepository`. -- [ ] **Step 2: Run** config/composition tests and verify old configuration is still accepted or active. -- [ ] **Step 3: Implement** explicit startup tombstones for the eight Stage-A-listed old variables and remove the old fetch stack from active composition; retain only test doubles/adapters needed by focused tests with names that cannot be mistaken for the production source. -- [ ] **Step 4: Run** config, Tutor, and startup tests and verify migration errors are actionable and no Capacity code is changed. -- [ ] **Step 5: Commit** `refactor: retire separate tutor repository source`. - -### Task 10: Update Course Content settings and status metadata - -**Files:** -- Modify: `shared/examples.ts` -- Modify: `server/routes/examples.routes.ts` or the existing catalog/status route -- Modify: `client/src/lib/external-examples.ts` -- Modify: `client/src/components/features/external-examples-settings.tsx` -- Modify: `client/src/components/features/examples-menu.tsx` -- Test: `tests/server/routes/examples.routes.test.ts` -- Test: `tests/client/external-examples.test.tsx` -- Test: `tests/client/settings-dialog.test.tsx` -- Test: `tests/client/examples-menu.behavior.test.tsx` - -**Interfaces:** -- Consumes: combined snapshot/status metadata and the existing `unoExternalExamplesSelection` browser preference. -- Produces: Course Content status that reports Examples and Tutor capability independently, clearly shows the one shared source, and preserves the existing browser-scoped override semantics. - -- [ ] **Step 1: Write failing tests** for shared source wording/status, Tutor valid/invalid metadata, one-selector persistence, and no lecturer-lock behavior. -- [ ] **Step 2: Run** focused Settings/Examples tests and verify the UI lacks Tutor capability status. -- [ ] **Step 3: Implement** status plumbing and UI copy without adding another source selector or changing the localStorage key. -- [ ] **Step 4: Run** focused Settings/Examples tests and verify browser override is personal and does not mutate server/operator defaults. -- [ ] **Step 5: Commit** `feat: expose course content status`. - -### Task 11: Add authoring fixtures, final documentation adjustments, and full verification - -**Files:** -- Create or modify: existing test fixture files under `tests/fixtures/` for schema-v1, schema-v2, valid multi-topic Tutor, invalid Tutor capability, and revision A/B snapshots -- Modify: `docs/ARCHITECTURE.md` only for implementation facts that remain consistent with the Stage-A SSOT -- Test: all relevant Tutor, Examples, and Settings suites - -**Interfaces:** -- Consumes: all implementation contracts from Tasks 1–10. -- Produces: maintainable authoring fixtures and a verified branch without changing normative Stage-A decisions. - -- [ ] **Step 1: Add failing integration/fixture coverage** for built-in free Tutor, matrix A–G, schema-v1/v2, multi-topic validation, strategy/topic precedence, revision pinning, reset, tombstones, and settings metadata. -- [ ] **Step 2: Run** the focused suites and fix only implementation defects exposed by the tests; record any contract conflict as a ruling before changing code. -- [ ] **Step 3: Update** factual architecture documentation only where the final implementation differs in mechanics from the Stage-A descriptions; do not weaken or rewrite the normative contract. -- [ ] **Step 4: Run the required final commands:** `npm run check`, `npm run test:unit`, `npm run build`, `npm run test:integration`, `npm run check:docs`, `git diff --check`, plus explicit relevant Tutor/Examples/Settings tests. -- [ ] **Step 5: Commit** `test: cover course content tutor control` and record test counts, remaining limitations, PR number if available, and final HEAD SHA. Do not merge or push. diff --git a/docs/superpowers/plans/2026-09-25-embedded-tutor-annotation.md b/docs/superpowers/plans/2026-09-25-embedded-tutor-annotation.md deleted file mode 100644 index 0128733d4..000000000 --- a/docs/superpowers/plans/2026-09-25-embedded-tutor-annotation.md +++ /dev/null @@ -1,115 +0,0 @@ -# Embedded Tutor Annotations and Learning Objectives Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Move example-specific Tutor metadata from Course manifest bindings into a validated, server-stripped annotation at the end of the example main `.ino`, and make its learning objectives affect both planned and free Tutor behavior. - -**Architecture:** The Course Content loader will extract exactly one terminal `/* @unosim-tutor ... @end-unosim-tutor */` block from each declared main `.ino`. It will return cleaned Example source plus a normalized `ExampleTutorAnnotation` inside the immutable server snapshot; public Example responses expose only cleaned source. Tutor planning receives the annotation separately and resolves embedded strategy, repository default, or built-in fallback without allowing annotation text to become instructions. - -**Tech Stack:** TypeScript, Zod, YAML core parser, Vitest, existing CourseContentLoader, TutorService, CurriculumTutorAdapter, and existing client Example/source contracts. - -**Spec:** Stage-A SSOT/ADR changes in this branch, based on the user-provided embedded Tutor annotation contract. - -## Global Constraints - -- Validation and implementation base is exactly `3ec19f7f0ea11071834be00977b10f93365e486e`. -- `/tmp/unosim-examples-v2-pilot` and the original UnoSim-Examples worktree are read-only for this task. -- `ssot/ssot_function_tutor_model_registration.md` remains untouched. -- One optional terminal annotation may exist only in the declared main `.ino`. -- Annotation data is strict, bounded, safe YAML data; it cannot provide prompts, roles, URLs, regexes, scripts, templates, or provider controls. -- Invalid Tutor annotation data strips from Example source, preserves valid Examples, and invalidates Tutor capability without partial activation. -- `learningObjectives` describe what to learn; `EffectiveTutorStrategy` controls how to teach. -- Existing revision/session pinning, privacy, API-key, difficulty, response, and source-authority contracts remain unchanged. - -## Review Focus - -- Marker placement and malformed/unterminated blocks: parser tests cover EOF, duplicate, middle-of-source, header-file, and hidden-source behavior. -- Invalid annotations must not leak teacher text: loader/API/client/compile tests assert cleaned source only. -- Objective persistence across topic/free paths and pinned dialogs: Tutor tests cover planned, free, no-match, and continuation paths. -- Removing manifest bindings must not break schema-v1 or unannotated schema-v2 Examples: loader/schema fixtures cover both. -- Embedded strategy precedence must remain distinct from repository/default fallback: strategy matrix tests cover embedded, repository, built-in, and arbitrary sketches. - -### Task 1: Normative contract - -**Files:** -- Modify: `ssot/ssot_function_definition_CourseContent.md` -- Modify: `ssot/ssot_function_definition_LearningQuestions.md` -- Modify if needed: `ssot/ssot_function_definition_ExternalExamples.md`, `docs/adr/0006-unified-course-content-and-tutor-strategy.md`, `docs/ARCHITECTURE.md` - -- [ ] Specify the terminal annotation grammar, ownership, safe stripping, strict schema, bounds, objective semantics, and strategy/topic precedence. -- [ ] Record that manifest example Tutor bindings are retired as the canonical mechanism because the only existing external pilot is local and unpushed. -- [ ] Run `npm run check:docs` and `git diff --check`. -- [ ] Commit only documentation with `docs: define embedded tutor annotations and learning objectives`. - -### Task 2: Annotation parser and normalized data - -**Files:** -- Create: `server/services/course-content/embedded-tutor-annotation.ts` -- Test: `tests/server/services/course-content/embedded-tutor-annotation.test.ts` - -- [ ] Add a strict Zod schema for `schemaVersion`, optional safe-ID arrays/IDs, and bounded control-character-free objectives. -- [ ] Add `extractEmbeddedTutorAnnotation(source: string, fileName: string)` that returns cleaned source plus optional normalized annotation, rejects non-main files, and recognizes only the exact terminal marker. -- [ ] Use YAML core parsing with duplicate-key and custom-tag rejection, and enforce a byte limit before parsing. -- [ ] Write failing tests for valid, absent, duplicate, middle, unterminated, invalid field, primary-topic, objective-bound, tag, and header-file cases. -- [ ] Run the focused test red before implementation and green after implementation. - -### Task 3: Loader boundary and hidden-source model - -**Files:** -- Modify: `server/services/examples/examples-schema.ts` -- Modify: `server/services/course-content/course-content-loader.ts` -- Modify: `server/services/examples/examples-repository.ts` -- Modify: `server/services/examples/examples-cache.ts` or related snapshot types -- Test: `tests/server/services/course-content/course-content-loader.test.ts`, `tests/server/services/examples/examples-repository.test.ts` - -- [ ] Process only each example’s declared main `.ino` after fetching files; attach annotation by Example ID and replace the main file content with the cleaned source. -- [ ] Make malformed annotation data invalidate Tutor capability while retaining cleaned valid Examples; ensure annotation is never included in Example API details or cache metadata. -- [ ] Keep ordinary source bytes unchanged except permitted trailing newline normalization. -- [ ] Add loader tests for valid extraction, missing annotation, malformed annotation isolation, secondary `.ino`/`.h` rejection, and all-reference behavior. -- [ ] Run Course Content and Examples tests. - -### Task 4: Tutor context and objective behavior - -**Files:** -- Modify: `server/services/tutor/tutor-planning.ts` -- Modify: `server/services/tutor/curriculum-tutor-adapter.ts` -- Modify: `server/services/tutor/tutor-service.ts` -- Modify: `server/services/tutor/strategy/tutor-strategy-guidance.ts` or add an app-owned objective guidance helper -- Test: `tests/server/services/tutor/strategy-conformance.test.ts`, `tests/server/services/tutor/tutor-service.test.ts`, relevant route tests - -- [ ] Carry annotation data in `TutorPlanningContentContext` and the pinned session context, never from mutable client metadata. -- [ ] Resolve embedded strategy first, then repository default, then `built-in-default`; preserve repository source metadata only for actual behavior. -- [ ] Add application-owned bounded guidance for validated learning objectives to free and planned prompts, explicitly labeling them as data rather than instructions. -- [ ] Preserve current-sketch factual authority, one-question, no-solution, provider, privacy, and response constraints. -- [ ] Add behavioral tests for objective-only free Tutor, objective-plus-topic, nonmatching topic with objectives retained, no annotation, arbitrary sketches, long objective acceptance, and strategy matrix A-E. - -### Task 5: Retire manifest Example bindings - -**Files:** -- Modify: `server/services/course-content/course-content-schema.ts` -- Modify: `server/services/course-content/course-content-loader.ts` -- Modify: `server/services/tutor/curriculum-tutor-adapter.ts` -- Modify: affected fixtures and tests under `tests/fixtures/course-content` and `tests/server/services` - -- [ ] Remove manifest-level per-Example Tutor binding fields and their validation because no published external dependency exists. -- [ ] Make unannotated schema-v2 manifests valid and preserve schema-v1 behavior. -- [ ] Convert existing strategy/topic tests to embedded annotations or direct normalized snapshot data without retaining a second authoring authority. -- [ ] Add explicit regression tests proving no ambiguous manifest-versus-embedded precedence remains. - -### Task 6: Client/API/compile hidden-source regression coverage - -**Files:** -- Inspect and modify only if required: `server/services/examples/examples-repository.ts`, `client/src/lib/external-examples.ts`, `client/src/components/features/examples-menu.tsx`, `client/src/hooks/use-tutor.ts`, compile/source-project paths -- Test: relevant client Example loading, Tutor, route, compile, and Playwright tests - -- [ ] Assert Example API, editor, compiler, and simulator receive cleaned source only. -- [ ] Assert the annotation never appears in client-visible file content or generated source payloads. -- [ ] Verify normal external Examples and unannotated sketches are unchanged. - -### Task 7: Final verification, review, and PR - -- [ ] Run focused Course Content, Examples, Tutor, strategy, revision/session, API, and client tests. -- [ ] Run `npm run check`, `npm run test:unit`, `npm run test:integration`, `npm run build`, `npm run check:docs`, `npm run test:coverage`, `npm run sonar`, `git diff --check`, and Playwright. -- [ ] Confirm Capacity code, Tutor Registry SSOT, external pilot, original worktrees, and unrelated deployment code are unchanged. -- [ ] Request a code review against the exact base SHA before pushing. -- [ ] Push `feature/embedded-tutor-annotation` and open the PR; do not merge. diff --git a/docs/superpowers/plans/2026-09-25-tutor-strategy-conformance.md b/docs/superpowers/plans/2026-09-25-tutor-strategy-conformance.md deleted file mode 100644 index a7f9c79ff..000000000 --- a/docs/superpowers/plans/2026-09-25-tutor-strategy-conformance.md +++ /dev/null @@ -1,215 +0,0 @@ -# Tutor Strategy Conformance Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make every normal Tutor request consume exactly one EffectiveTutorStrategy, including free Tutor requests for arbitrary sketches, while preserving the Course Content SSOT and all app-owned Tutor safety rules. - -**Architecture:** First strengthen the Course Content and LearningQuestions SSOT so strategy behavior is normative for planned and free Tutor paths. Then audit the merged implementation and record a field-by-field matrix before writing code. Finally add failing behavioral tests and introduce the smallest app-owned translation from EffectiveTutorStrategy to planner and free-Tutor behavior; repository data remains normalized closed-schema data and never becomes prompt instructions. - -**Tech Stack:** TypeScript, Vitest, existing TutorService/planner/provider abstractions, Markdown SSOT/ADR documentation, npm scripts, Git worktrees. - -**Spec:** `ssot/ssot_function_definition_CourseContent.md`, `ssot/ssot_function_definition_LearningQuestions.md`, and the user-provided three-stage Tutor Strategy Conformance requirements. - -## Global Constraints - -- `origin/main` must remain the baseline `6f3676d40fb5ce5f8eab625d035b77f2f99b9016` before work begins. -- Stage A modifies only the requested normative documentation and the execution-plan artifact; no implementation or test files change before the audit is complete. -- Every normal Tutor request uses exactly one EffectiveTutorStrategy. -- Topic planning answers what should be learned; strategy answers how the Tutor teaches; absence of a Topic never disables the selected strategy. -- Strategy precedence is per-example, repository defaultStrategy, built-in-default. -- Topic precedence is active-example primaryTopic, other applicable bound topics, fact-matched repository topics, then free Tutor with the same EffectiveTutorStrategy. -- Invalid Tutor capability disables repository Topics and Strategies together; valid Examples remain usable. -- Repository content cannot provide system prompts, raw prompts, roles, URLs, regexes, scripts, template code, or arbitrary instructions. -- Preserve the current sketch as factual authority, one-primary-question behavior, answerRating 1..5, provider isolation, privacy/API-key behavior, response limits, Mermaid safety, and persistent-storage rules. -- Do not modify Capacity code/defaults, Tutor Registry SSOT, unrelated deployment code, or `ttbombadil/unosim-examples`. -- Do not invent adaptive-difficulty profiles; schema version 1 uses `current-contract` and the existing LearningQuestions algorithm. - -## Review Focus - -- An arbitrary local sketch with a repository defaultStrategy must show trusted strategy guidance in the provider prompt, not only strategy metadata; Task 4 adds this regression test. -- A valid repository strategy with zero Topics must control free Tutor behavior; Task 4 covers this separately from the no-match case. -- A no-match sketch must retain the repository strategy while omitting only the Topic plan; Task 4 asserts the effective strategy identity and behavior together. -- Invalid Tutor capability and no repository must select built-in-default through the same path; Task 4 covers both fallback cases. -- Strategy values must remain closed normalized data and cannot inject prompt text or executable/template content; Task 4 adds the security regression assertion. - ---- - -### Task 1: Strengthen Course Content and LearningQuestions SSOT - -**Files:** -- Modify: `ssot/ssot_function_definition_CourseContent.md` -- Modify: `ssot/ssot_function_definition_LearningQuestions.md` -- Modify: `docs/adr/0006-unified-course-content-and-tutor-strategy.md` only where the clarified invariant is needed -- Modify: `docs/ARCHITECTURE.md` only where the clarified invariant is needed -- Modify: `docs/superpowers/plans/2026-09-25-tutor-strategy-conformance.md` -- Test: no implementation/test files in this task - -**Interfaces:** -- Consumes: Stage-A SSOT/ADR from PR #108 and the explicit field semantics in the user request. -- Produces: normative definitions of EffectiveTutorStrategy application, topic/strategy independence, all ten strategy fields, and app-owned non-configurable rules for Stage B and Stage C. - -- [ ] **Step 1: Edit the Course Content SSOT** - - Add the exact normal-request invariant, the planned/free path cases, the independent Topic/strategy model, precedence rules, behavioral-not-metadata conformance rule, and closed-schema security constraints. Define operational semantics for `questionKindWeights`, `sketchSpecificity`, `repetition`, `remediation`, `clarification`, `progression`, `scaffolding`, `feedbackVerbosity`, `hintFirst`, and schema-v1 `adaptiveDifficulty`. - -- [ ] **Step 2: Edit the LearningQuestions SSOT** - - Define how each strategy field affects deterministic planning and free-Tutor guidance without changing answerRating, one-question, factual-authority, difficulty, privacy, response-limit, or provider contracts. - -- [ ] **Step 3: Make only necessary ADR/architecture wording corrections** - - Update the unified Course Content ADR and architecture overview to state that a no-Topic free Tutor still consumes the repository/default EffectiveTutorStrategy, and that metadata without behavioral consumption is non-conformant. - -- [ ] **Step 4: Run documentation validation** - - Run `npm run check:docs` and `git diff --check`. Expected: both exit successfully and only the requested Markdown/plan files are changed. - -- [ ] **Step 5: Commit Stage A** - - ```bash - git add ssot/ssot_function_definition_CourseContent.md ssot/ssot_function_definition_LearningQuestions.md docs/adr/0006-unified-course-content-and-tutor-strategy.md docs/ARCHITECTURE.md docs/superpowers/plans/2026-09-25-tutor-strategy-conformance.md - git commit -m "docs: clarify tutor strategy behavior across all tutor paths" - ``` - -### Task 2: Read-only implementation conformance audit - -**Files:** -- Inspect: `server/services/tutor/tutor-service.ts` -- Inspect: `server/services/tutor/tutor-planning.ts` -- Inspect: `server/services/tutor/curriculum-tutor-adapter.ts` -- Inspect: `server/services/tutor/curriculum/learning-planner.ts` -- Inspect: `server/services/tutor/strategy/effective-tutor-strategy.ts` -- Inspect: `server/services/course-content/*` -- Inspect: `shared/tutor.ts` -- Inspect: `client/src/hooks/use-tutor.ts` -- Inspect: relevant Tutor, Examples, Course Content, Settings, and route tests -- Create: `docs/audits/2026-09-25-tutor-strategy-conformance-audit.md` - -**Interfaces:** -- Consumes: the Stage-A SSOT commit from Task 1. -- Produces: a complete matrix with rows for every strategy field and columns Planned initial, Planned dialog, Free initial, Free dialog, Tests, and Status; explicit metadata-only findings; fallback/precedence findings; exact file/function evidence; smallest fix architecture. - -- [ ] **Step 1: Trace strategy resolution and behavior consumers** - - Search for every `EffectiveTutorStrategy` resolution and record whether the resolved object reaches prompt construction, deterministic planning, remediation, progression, repetition, clarification, scaffolding, feedback, hint ordering, and difficulty behavior before response generation. - -- [ ] **Step 2: Trace free-Tutor hard-coded policy** - - Inspect `buildUserPrompt`, `buildDialogPrompt`, `getRemediationInstruction`, `getProgressionInstruction`, `ensureDistinctDialogQuestion`, `buildRemediationQuestion`, and `buildConceptTransitionQuestion`; record each hard-coded policy and whether the SSOT makes it configurable. - -- [ ] **Step 3: Trace planner field coverage** - - Verify actual use of all ten strategy fields in the deterministic curriculum path and distinguish meaningful fallback semantics from fields that are only parsed or returned as metadata. - -- [ ] **Step 4: Audit required scenarios and tests** - - Inspect arbitrary local sketch/default strategy, no Topic match/default strategy, repository strategy without Topics, invalid Tutor capability, no repository, per-example override, and the existing tests that assert only `strategyId`/`strategySource`. - -- [ ] **Step 5: Write the audit report before any Stage-C code** - - Write the matrix and exact gaps to `docs/audits/2026-09-25-tutor-strategy-conformance-audit.md`. The report must label each row `PASS`, `PARTIAL`, `MISSING`, or `NOT-APPLICABLE-BY-CONTRACT`, name the exact functions/files, and describe the smallest fix architecture without implementing it. - -### Task 3: Add failing behavioral conformance tests - -**Files:** -- Modify: existing focused Tutor/Course Content test files identified in Task 2 -- Create: a focused conformance test file only if existing test ownership cannot express the behavior without duplication - -**Interfaces:** -- Consumes: the audit matrix and current public/internal Tutor test seams. -- Produces: failing tests that observe trusted strategy guidance/selection and behavior, not merely metadata, for the exact gaps identified in the audit. - -- [ ] **Step 1: Add the arbitrary/no-Topic regression** - - Assert that a repository default strategy changes provider-owned strategy guidance for an arbitrary sketch when no Topic plan exists. - -- [ ] **Step 2: Add zero-Topics and no-match regressions** - - Assert that a repository strategy with zero Topics and a repository with valid Topics that do not match both retain the repository strategy on the free path. - -- [ ] **Step 3: Add built-in and invalid-capability regressions** - - Assert that no repository and invalid Tutor capability both use built-in-default through the same behavior-producing path. - -- [ ] **Step 4: Add two-strategy behavioral distinction and per-example precedence** - - Assert that two valid strategies produce different trusted guidance or deterministic policy, and that a valid per-example strategy overrides the repository default. - -- [ ] **Step 5: Add field-specific behavioral tests for every demonstrated gap** - - Cover weights, specificity, repetition, remediation, clarification, progression, scaffolding, feedback verbosity, hintFirst, and current-contract adaptive difficulty only where Task 2 shows the current path needs correction. - -- [ ] **Step 6: Run the focused tests and record RED** - - Run the exact focused Vitest files. Expected: each newly added test fails for the audited reason before implementation changes; unrelated baseline tests remain green. - -- [ ] **Step 7: Commit the regression tests** - - ```bash - git add tests - git commit -m "test: expose tutor strategy conformance gaps" - ``` - -### Task 4: Fix only demonstrated conformance gaps - -**Files:** -- Modify: only the Tutor service/planner/strategy files named by the audit -- Modify: only the focused tests from Task 3 as required for final assertions -- Do not modify: Course Content architecture, Capacity, Tutor Registry SSOT, external examples repository - -**Interfaces:** -- Consumes: `EffectiveTutorStrategy` and the closed strategy schema defined by Stage A; existing Tutor provider, dialog, difficulty, privacy, and answerRating contracts. -- Produces: one application-owned translation layer from EffectiveTutorStrategy to deterministic planner policy, free-Tutor guidance, and dialog/remediation policy without passing raw repository text to prompts. - -- [ ] **Step 1: Introduce the smallest trusted strategy-to-behavior mapping** - - Map closed enum/number values to application-owned guidance or deterministic choices. Ensure the built-in strategy uses this exact path and repository strategies cannot inject arbitrary prompt text. - -- [ ] **Step 2: Apply the strategy to free initial questions** - - Pass the effective strategy into the free initial prompt path and ensure arbitrary sketches, zero-Topic repositories, and no-match repositories retain repository defaultStrategy behavior. - -- [ ] **Step 3: Parameterize free dialog behavior** - - Replace only the audited hard-coded configurable remediation, repetition, clarification, progression, scaffolding, feedback, and hint-first decisions with the trusted mapping while preserving core Tutor safety rules. - -- [ ] **Step 4: Complete planner coverage where the SSOT requires it** - - Consume every audited applicable field in deterministic planning, using generated/app-owned fallbacks where the SSOT defines them and retaining current-contract adaptive difficulty without a second delta implementation. - -- [ ] **Step 5: Run the focused tests GREEN** - - Run the focused Tutor/Course Content tests from Task 3. Expected: all new regression tests pass and metadata matches the strategy that actually produced the behavior. - -- [ ] **Step 6: Commit the implementation fix** - - ```bash - git add server client shared tests - git commit -m "fix: apply effective tutor strategy to free tutor behavior" - ``` - -### Task 5: Final verification and review handoff - -**Files:** -- Inspect: complete branch diff and final audit/SSOT changes -- Modify: no new functionality; only test/doc adjustments required by verification evidence - -**Interfaces:** -- Consumes: all Stage-A, test, and implementation commits. -- Produces: a reviewable branch with exact test evidence, no external-repository changes, and a PR-ready report; it is not merged automatically. - -- [ ] **Step 1: Run relevant focused Tutor/Examples/Settings tests** - - Run all focused tests named by the audit plus the existing Tutor, Course Content, Examples, and Settings suites. Expected: zero failures and no test-only metadata assertions left for corrected behavior. - -- [ ] **Step 2: Run the required release gates** - - Run `npm run check`, `npm run test:unit`, `npm run test:integration`, `npm run build`, `npm run check:docs`, `npm run test:coverage`, `npm run sonar`, and `git diff --check`. Run Playwright if the changed behavior reaches browser Tutor behavior. - -- [ ] **Step 3: Verify scope and security** - - Confirm no Capacity files/defaults, Tutor Registry SSOT, or external examples repository files changed; search the final diff for raw repository prompt fields, roles, URLs, regexes, scripts, and template content. - -- [ ] **Step 4: Perform final review before push/PR** - - Review the complete `origin/main...HEAD` diff against the strengthened SSOT, record remaining limitations, and only then push `feature/tutor-strategy-conformance` and open a PR. Do not merge automatically. diff --git a/docs/superpowers/plans/2026-09-27-tutor-mastery-progression-stage-a.md b/docs/superpowers/plans/2026-09-27-tutor-mastery-progression-stage-a.md deleted file mode 100644 index 4867535a9..000000000 --- a/docs/superpowers/plans/2026-09-27-tutor-mastery-progression-stage-a.md +++ /dev/null @@ -1,46 +0,0 @@ -# Implement mastery-driven Tutor progression Stage A - -## Scope - -Implement the approved mastery-progression contract from SSOT commit -`fdd641bbae30ceae8347f9835620c3d1274462d9` in the isolated worktree only. -Preserve schema-v1 compatibility, capability-scoped fallback, the existing -free-Tutor safety contract, Course revision/session pinning, and all protected -unrelated areas. - -## Tasks - -1. Audit the approved SSOT and current implementation; record gaps and - unresolved implementation rulings in the execution ledger. -2. Add failing schema/data-model tests for Tutor manifest v2, Topic v2, - extensions, phase strategies, and v1 compatibility; implement strict - normalization and capability-scoped loading. -3. Add failing planner tests for Topic mastery-domain classification, - probeable/unresolved/content-exhausted outcomes, multi-topic precedence, - LEARN/DEEPEN/EXPAND transitions, deepening evidence, and retention; then - implement the deterministic state machine. -4. Add phase-strategy and extension-flow tests; implement phase-specific - effective strategy selection and bounded extension guidance. -5. Thread server-owned progression state through Tutor planning/session - boundaries and add the minimum additive API/client state needed for the - visible Tutor phase and blocked diagnostics, without exposing evidence or - teacher-controlled instructions. -6. Add regression coverage for free Tutor behavior, revision/context reset, - invalid capability isolation, strategy precedence, hidden-source and - security invariants; run all focused suites. -7. Run the complete release gates, review the diff and protected worktrees, - then create focused logical commits and open (but do not merge) a PR only - if every required gate passes. - -## Verification gates - -- `npm run check` -- `npm run test:unit` -- `npm run test:integration` -- `npm run test:coverage` -- `npm run build` -- `npm run check:docs` -- `npm run sonar` -- `git diff --check` -- focused Tutor/Course Content/strategy/session/API/client suites -- Playwright; Docker/Arduino only when affected by the implementation diff --git a/docs/superpowers/plans/2026-09-27-tutor-quality-stage-1.md b/docs/superpowers/plans/2026-09-27-tutor-quality-stage-1.md deleted file mode 100644 index d7c137353..000000000 --- a/docs/superpowers/plans/2026-09-27-tutor-quality-stage-1.md +++ /dev/null @@ -1,98 +0,0 @@ -# Tutor Quality Stage 1 Implementation Plan - -> **HISTORICAL — completed plan (2026-10-02).** Not current worker guidance. The Stage-1 runtime gate stays normative in `ssot/ssot_function_definition_TutorQuality.md`; current plan: `docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`. - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Establish deterministic Tutor quality hard gates, a reusable scenario runner, the PWM regression case, and compatible Course Content validation in UnoSim and UnoSim-Examples. - -**Architecture:** Keep runtime ownership in `TutorService`, `CurriculumTutorAdapter`, and existing Course Content schemas. Add one pure authoring validator that reuses production fact matching, one filesystem CLI adapter, and test-only scenario orchestration around the real Tutor service with only the provider faked. Pin the Examples CI consumer to the resulting UnoSim commit. - -**Tech Stack:** TypeScript, Vitest, Zod/YAML, GitHub Actions, existing Course Content loader and Tutor planner. - ---- - -Specification: `ssot/ssot_function_definition_TutorQuality.md` - -### Task 1: Establish the normative quality contract - -**Files:** -- Create: `ssot/ssot_function_definition_TutorQuality.md` -- Modify: `docs/ARCHITECTURE.md` -- Test: `scripts/check-docs.mjs` - -1. Add the deterministic learning-support proxy, trust boundaries, runtime/content invariants, scenario contract, CI contract, and explicit Stage 2 exclusions. -2. Link the SSOT from the architecture documentation. -3. Run `npm run check:docs` and commit the documentation. - -### Task 2: Build the reusable scenario runner and adversarial service cases - -**Files:** -- Create: `tests/server/services/tutor/support/tutor-quality-scenario-runner.ts` -- Create: `tests/server/services/tutor/tutor-quality-scenarios.test.ts` -- Modify: `server/services/tutor/tutor-service.ts` - -1. Write failing scenarios for provider errors before commit, invalid output, complete solutions, multiple primary questions, contradictory provider metadata, exact/near repetition, transition/blocked repair retention, and TutorPlan question authority. -2. Run the focused tests and confirm each new invariant fails for the intended reason. -3. Implement the minimal validation/metadata-boundary changes in `TutorService`. -4. Run the focused tests and existing Tutor service/planning tests; commit. - -### Task 3: Preserve TQ-REG-001 with real PWM inputs - -**Files:** -- Create: `tests/fixtures/tutor-quality/TQ-REG-001-pwm.ino` -- Create: `tests/server/services/tutor/tq-reg-001-pwm.test.ts` - -1. Write a failing scenario using the real PWM sketch and the narrowed Topic activation from the reviewed Course Content. -2. Assert that `variables-and-serial` is not selected, a strong-answer repetition is repaired, and phase/revision/state metadata remains consistent. -3. Make only runner/fixture corrections if needed; production changes require a separate failing invariant test. -4. Run the focused regression test and commit. - -### Task 4: Add deterministic Course Content quality validation - -**Files:** -- Create: `server/services/course-content/tutor-quality-schema.ts` -- Create: `server/services/course-content/tutor-quality-validator.ts` -- Create: `server/services/course-content/tutor-quality-directory-validator.ts` -- Create: `tests/server/services/course-content/tutor-quality-validator.test.ts` -- Create: `scripts/validate-tutor-course-content.mjs` -- Create: `tests/server/services/course-content/tutor-quality-directory-validator.test.ts` -- Modify: `package.json` - -1. Write failing unit tests for activation cases, missing coverage, unreachable Concepts/Indicators, mastery probe/kind shortages, prerequisite reachability, DEEPEN exhaustion/kind shortages, and a valid bundle. -2. Implement a pure validator using existing facts/matcher/question applicability/deepening defaults. -3. Write a failing CLI test for local bundle loading, hashes, and quality-case parsing. -4. Implement the filesystem fetch adapter and machine-readable/nonzero CLI result. -5. Run focused tests, typecheck, and commit. - -### Task 5: Make UnoSim-Examples pass the authoring hard gate - -**Files (UnoSim-Examples worktree):** -- Create: `tutor/quality-cases.yaml` -- Modify: `tutor/topics/variables-and-serial.yaml` -- Modify: `tutor/topics/long-values.yaml` -- Modify: `tutor/manifest.yaml` - -1. Add minimal positive/negative activation cases, including PWM as a negative case for `variables-and-serial`. -2. Run the validator and observe the expected mastery/DEEPEN exhaustion failures. -3. Add the smallest structurally distinct application/transfer probes needed for reachable mastery and default DEEPEN; update hashes. -4. Re-run the pinned local validator and commit. - -### Task 6: Add explicit CI hard gates and compatible-version pinning - -**Files:** -- Modify (UnoSim): `.github/workflows/ci.yml`, `package.json` -- Create (UnoSim-Examples): `.github/workflows/tutor-quality.yml`, `.unosim-compatible-commit` - -1. Add `test:tutor-quality` as a deterministic, credential-free UnoSim PR step. -2. Pin the Examples workflow to the final UnoSim Stage 1 commit and invoke its Course Content validator against the Examples checkout. -3. Validate workflow syntax structurally and run both local commands. -4. Commit CI changes. - -### Task 7: Verify and review the complete change - -1. Run `npm run check`, `npm run check:docs`, `npm run test:tutor-quality`, focused Tutor/Course Content tests, and `npm run test:unit` in UnoSim. -2. Run the exact pinned validator against the UnoSim-Examples worktree and verify hashes/status are clean. -3. Inspect both diffs and commits for accidental changes, secrets, scope creep, or reliance on a real provider. -4. Perform a fresh self-review because parallel review agents are disabled for this task; fix verified issues and repeat affected gates. -5. Report both branch heads without pushing or merging. diff --git a/docs/superpowers/plans/2026-10-07-optional-gateway-auth-ubuntu-vbox.md b/docs/superpowers/plans/2026-10-07-optional-gateway-auth-ubuntu-vbox.md deleted file mode 100644 index e005fe8c3..000000000 --- a/docs/superpowers/plans/2026-10-07-optional-gateway-auth-ubuntu-vbox.md +++ /dev/null @@ -1,92 +0,0 @@ -# Optional Gateway Authentication and Ubuntu VBox Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Align UnoSim's deployment documentation and Ubuntu VBox installer with the policy that gateway trust is mandatory while user authentication is optional. - -**Architecture:** Keep the application gateway contract unchanged. Update the current deployment summaries and add ADR 0008 to supersede only ADR 0001's user-authentication requirement. Add an Ubuntu guide and guarded bootstrap script that configure Nginx as a TLS and identity-header proxy, with IP-derived identity by default. - -**Tech Stack:** Markdown, Bash, Docker Compose v2, Nginx, OpenSSL, Node.js project checks. - -**Spec:** `docs/superpowers/specs/2026-10-07-optional-gateway-auth-ubuntu-vbox-design.md` - -## Global Constraints - -- Docker mode requires `UNOSIM_GATEWAY_SECRET`, `UNOSIM_TRUSTED_PROXY`, `UNOSIM_ALLOWED_WS_ORIGINS`, a subject, and the `user` role. -- User authentication before the gateway is optional; IP-derived identity uses the directly observed `$remote_addr` and is not authentication. -- The Ubuntu script requires an explicit LAN IPv4, validates it is assigned locally, binds the backend to `127.0.0.1:3000`, and does not modify firewall, router, client DNS, client trust stores, or SSH configuration. -- Do not print the generated gateway secret or overwrite an existing `.env` or UnoSim Nginx site. -- Preserve ADR 0001 as historical and leave unrelated working-tree changes untouched. -- Do not change application authorization, gateway validation, isolation, rate limits, or origin checks. - -## Review Focus - -- Same-IP users share the IP-derived subject and its admission/rate limits; document NAT, VPN and upstream-proxy implications. -- `UNOSIM_TRUSTED_PROXY` configures Express proxy trust but current authorization does not compare the socket peer to this CIDR; document the loopback backend binding as the actual Ubuntu topology boundary. -- Browser origins omit the default HTTPS port and include `:8443` explicitly for both hostname and address forms. -- Nginx must set/overwrite the gateway secret, subject and role for both HTTP and WebSocket requests. -- Existing `.env`, UnoSim site, secret snippet or TLS assets must cause a safe abort before replacing operator state. - ---- - -### Task 1: Update gateway policy records and current summaries - -**Files:** -- Modify: `README.md` -- Modify: `docs/SECURITY.md` -- Modify: `docs/INSTALL_SERVER.md` -- Modify: `docs/ARCHITECTURE.md` -- Create: `docs/adr/0008-optional-gateway-authentication.md` -- Modify: `docker-compose.yml` comments if they still imply authentication is mandatory - -- [x] Update Docker deployment wording to state that the trusted gateway contract is mandatory while authenticating a person is optional. -- [x] Document authenticated and IP-identified gateway modes, including that IP mode grants access to every reachable client and shares identity for clients behind the same observed IP. -- [x] State precisely that `UNOSIM_TRUSTED_PROXY` configures Express proxy trust and is not itself an authorization check of the peer address. -- [x] Document exact WebSocket origin matching and proxy header overwrite requirements for both HTTP and WebSocket traffic. -- [x] Add ADR 0008 with status Accepted; explain that it supersedes ADR 0001 only on the user-authentication requirement and preserves the mandatory gateway trust contract. -- [x] Describe the Docker installation entry consistently in README, security, installation and architecture summaries. -- [x] Avoid hard-coding an admission count in deployment summaries; describe the effective limit as source-version/configuration dependent. -- [x] Review changed paragraphs against the code in `server/security/access-control.ts`, `server/index.ts`, and `server/services/simulation-admission-controller.ts`. - -### Task 2: Add and index the Ubuntu VBox installation guide - -**Files:** -- Create: `README_UbuntuServer.md` -- Modify: `docs/README.md` - -- [x] Write the clean Ubuntu Desktop prerequisites and step-by-step invocation of `scripts/install-ubuntu-vbox.sh --lan-ip ` from the intended checkout. -- [x] Explain the resulting URLs `https://unosim.vbox/` and `https://unosim.vbox:8443/`, the generated certificate location, and Docker readiness/simulation verification. -- [x] Cover DHCP reservation and hostname resolution using router-local DNS or a single-client hosts entry. -- [x] Explain client trust of the self-signed certificate; provide optional temporary SSH transfer instructions and the command to disable `ssh.socket` afterward. -- [x] Explain that TCP 443 and optional 8443 may need an operator firewall rule, without changing UFW in the script. -- [x] Clearly distinguish gateway secret/header trust from optional user authentication, and explain shared-IP identity consequences. -- [x] Explain that the effective per-identity simulation admission limit comes from the installed source/configuration, not a fixed README promise. - -### Task 3: Add the guarded Ubuntu bootstrap script - -**Files:** -- Create: `scripts/install-ubuntu-vbox.sh` -- Create: `scripts/test-install-ubuntu-vbox.sh` - -- [x] Write a shell regression test that passes malformed `--lan-ip` input and asserts the installer rejects it before invoking system changes; run it to observe the expected failure before adding the installer. -- [x] Implement Bash argument parsing for required `--lan-ip IPv4`; reject non-Ubuntu hosts, malformed IPv4, and addresses not assigned to the host before making changes. -- [x] Require execution from the UnoSim checkout and verify required Dockerfile, Compose and Nginx paths exist. -- [x] Preflight `.env`, UnoSim Nginx site, gateway-secret snippet and TLS target paths; abort rather than overwrite existing operator files. -- [x] Install `docker.io`, `docker-compose-v2`, `docker-buildx`, `nginx`, `openssl` and `curl`; add the invoking user to the `docker` group. -- [x] Create runtime directories, then write `.env` with mode 0600, socket GID, a 64-hex-character generated secret, `UNOSIM_TRUSTED_PROXY=127.0.0.1/32`, and exact origins for hostname, LAN IPv4, localhost and loopback at default HTTPS and explicit 8443 ports. Do not echo the secret. -- [x] Build `unosim-sandbox:latest` from `Dockerfile.sandbox`, then build/start only `unosim-backend` with `UNOSIM_BIND_ADDRESS=127.0.0.1`. -- [x] Generate a self-signed certificate with SANs for `unosim.vbox`, `localhost`, `127.0.0.1`, and the supplied LAN IPv4. -- [x] Install a root-owned 0600 Nginx secret snippet and a site listening on loopback and LAN IPv4 at 443 and 8443, with no Basic Auth, `ip-$remote_addr` subject, `user` role, HTTP forwarding, and WebSocket upgrade/timeout headers. -- [x] Preserve the package default site file, disable only its standard enabled symlink, and abort if that link is custom or a listener already occupies ports 443 or 8443. Enable the new site, run `nginx -t`, and reload only after validation passes; remove the new enabled symlink if validation fails. -- [x] Print service URLs, certificate path and remaining client-side setup steps; never print secrets. - -### Task 4: Verify documentation, script and repository checks - -**Files:** -- Verify: all files above - -- [x] Run `bash scripts/test-install-ubuntu-vbox.sh`, `bash -n scripts/install-ubuntu-vbox.sh` and `shellcheck scripts/install-ubuntu-vbox.sh` when ShellCheck is available. -- [x] Run `git diff --check`, `npm run check`, `npm run check:docs`, and `npm run test:deployment` if Docker is available. -- [x] Run relevant safe docs and shell checks, then inspect `git status --short` and ensure unrelated pre-existing changes are untouched. -- [x] Inspect `run-tests.sh`; run `./run-tests.sh` only if its cleanup and process/port effects cannot touch existing user state. -- [x] Do not push or merge. diff --git a/docs/superpowers/specs/2026-09-23-capacity-calibration-design.md b/docs/superpowers/specs/2026-09-23-capacity-calibration-design.md deleted file mode 100644 index 75f1258be..000000000 --- a/docs/superpowers/specs/2026-09-23-capacity-calibration-design.md +++ /dev/null @@ -1,284 +0,0 @@ -# UnoSim Host-Capacity Calibration Design - -## Goal - -Provide a reproducible `npm run capacity:calibrate -- --expected-users 200` -command that measures the current host and Docker-backed UnoSim workload, -selects directly measured capacity settings, and emits reviewable -recommendations without changing production configuration. - -## Scope and constraints - -- The calibration tool is branch-independent. It may run from any clean - checkout; the report records the exact Git SHA and dirty-state indicator used - for the measurement. -- Production defaults and application behavior remain unchanged. -- The existing Capacity semantics are authoritative: - `SIMULATION_MAX_CONCURRENT`, `SANDBOX_START_MAX_CONCURRENT`, - `SIMULATION_ADMISSION_MAX`, `SIMULATION_QUEUE_TIMEOUT_MS`, - `SANDBOX_START_SLOT_TIMEOUT_MS`, `DOCKER_CONTROL_TIMEOUT_MS`, and - `COMPILE_MAX_CONCURRENT`. -- Existing run-ID isolation, Docker lifecycle events, polling, status snapshots, - receipts, cleanup, and CPU-intensive workload remain the measurement basis. -- No recommendation may exceed a directly measured stable simulation level or - startup-concurrency candidate. Interpolation is allowed only inside measured - refinement bounds; extrapolated values are report-only and never written to - `capacity.env`. -- `COMPILE_MAX_CONCURRENT` is optional in version 1. If it is not measured, the - output records `not calibrated` and preserves the existing derived default. -- The Tutor Registry SSOT is outside this branch and is not touched. - -## Architecture - -The feature has four separable layers: - -1. **Measurement collection** starts an owned backend with test-only environment - overrides, drives the existing CPU-intensive client workload, collects raw - host/Docker/status/lifecycle data, and performs label-scoped cleanup. -2. **Policy evaluation** is a pure module. It accepts recorded measurements and - calibration policy, then returns recommendations, confidence, and reasons. -3. **Orchestration** validates CLI input, plans bounded phases, enforces the - global deadline and safety guards, and coordinates collection/evaluation. -4. **Output formatting** writes versioned JSON, Markdown, and a review-only env - fragment from the same result object. - -The validation test keeps its own assertions. Shared measurement helpers expose -observations but do not decide whether a validation scenario passes, so a bug in -measurement cannot make both the test and calibration agree accidentally. - -## CLI contract - -Command: - -```text -npm run capacity:calibrate -- [options] -``` - -Supported options and defaults: - -| Option | Default | Meaning | -| --- | ---: | --- | -| `--expected-users ` | `200` | Classroom/admission envelope | -| `--target-cpu ` | `75` | Desired steady-state CPU p95 | -| `--max-cpu ` | `85` | Hard active-cap CPU p95 ceiling | -| `--max-user-wait ` | `240` | UX policy ceiling for queue waits | -| `--max-duration ` | `30` | End-to-end calibration budget | -| `--output-dir ` | `capacity-test-results/calibration-` | Artifact directory | -| `--skip-classroom` | false | Skip the expected-users arrival phase | -| `--skip-startup-tuning` | false | Keep the existing startup candidate/default | -| `--dry-run` | false | Validate, probe metadata, print the plan, run no load | -| `--verbose` | false | Print phase and sample detail | - -Numeric values must be finite integers or decimals where applicable, positive, -and within safe bounds. `target-cpu < max-cpu <= 95`; expected users and duration -must be positive; user wait must be positive. Invalid or unknown arguments exit -with status 2 before any Docker workload starts. The effective policy is printed -before the first measurement. - -Version 1 uses a fixed 60-second active classroom simulation duration for every -client. It is recorded in the policy and every classroom run; it is not a CLI -override, so classroom results remain comparable across hosts. - -## Measurement phases - -### Pre-flight - -Record: - -- Git SHA and dirty-state indicator -- host architecture, logical CPUs, physical memory, load average, available - memory, and swap where available -- Docker client/server versions, architecture, visible CPUs/RAM, storage driver, - daemon health, and sandbox image ID/digest -- effective sandbox CPU/memory limits and relevant Capacity defaults -- unrelated running containers and their names/statuses - -The tool aborts before load if Docker is unhealthy or the configured sandbox -image cannot be inspected. Existing workloads are never stopped automatically. - -### Phase A: Docker control latency - -Measure `docker --version`, `docker info`, and `docker image inspect` during an -idle sample and a bounded parallel burst. Store every duration and summarize -p50/p95/p99/max per command and condition. - -The pure policy recommends: - -```text -max(productionDefault, ceil(p99 * 1.5 / 100ms) * 100ms) -``` - -The output always shows measured p99/max, the technical recommendation, the -production default, and the configured range. If the computed value is below -the production default, the recommendation is to keep the production default. -If it exceeds the validated maximum, the result is marked `out-of-range`, the -technical value is reported for review, and no Docker-control value is written -to `capacity.env`; the tool never silently clamps it. This phase never changes -Docker configuration. - -### Phase B: active simulation capacity - -Plan a bounded ladder from host size and expected users. The initial maximum is -`min(expectedUsers, max(20, logicalCPUs * 4))`, and candidates are rounded to -host-sized increments. The plan cannot exceed 128 or any explicit safety bound. - -Each candidate uses the existing CPU-intensive workload, requires all clients to -reach `RUNNING`, requires lifecycle-derived physical overlap equal to the -requested level, corroborates with polling, holds a stable all-running plateau, -and records host/Docker/status/error/cleanup data. - -A run is stable only when it has no functional errors, no OOM/Docker/backend -failure, no cleanup leak, lifecycle peak equal to the request, and polling peak -consistent with that peak. A sustained safety violation ends escalation and -cleanup immediately. - -Coarse candidates use 20–30 second plateaus. If the coarse results straddle the -target CPU region, refinement tests only measured intermediate values between the -last passing and first exceeding candidate. If the first candidate already -exceeds the target CPU, refinement proceeds downward after cleanup using the same -bounded step, testing `candidate - step`, then lower values until a candidate -meets the target or the lower bound is reached. If a candidate reaches the hard -CPU ceiling, the high run stops immediately; a lower probe is allowed only after -the host is healthy and cleanup is complete. Refinement uses -`max(5, ceil(logicalCPUs / 2))`, never extrapolates, and stops when the target -region is directly measured or the time budget is exhausted. - -The recommended `SIMULATION_MAX_CONCURRENT` is the highest directly measured -stable candidate with CPU p95 at or below `target-cpu`. If no candidate meets -the target but a stable candidate is below `max-cpu`, the highest such candidate -is reported with reduced confidence and an explicit warning. No candidate above -`max-cpu`, with unsafe memory/iowait, or with functional failure is recommended. - -### Phase C: sandbox-start concurrency - -At or below the selected active cap, test a bounded startup candidate ladder -adapted to the host and existing default, such as 8, 12, 16, 20, 24, and 32, -without exceeding the measured active cap or safety budget. - -For every candidate record startup-slot wait and slot-acquisition-to-runtime -latencies, CPU/load/iowait/memory, failures/timeouts, time to the active plateau, -and cleanup. Resource comparisons use the smallest safe directly measured -startup candidate as the baseline. A candidate has a material regression when -CPU p95 rises by more than 5 percentage points or 10% relative, iowait rises by -more than 5 percentage points, available memory falls by more than 10%, or any -failure/timeout occurs. - -The policy chooses the smallest safe candidate whose startup p95 is within 10% -of the best observed safe startup p95, while also requiring no material -regression against that baseline. The output lists all candidates and explains -why larger candidates were not selected. No startup candidate is recommended -unless it was directly measured. - -### Phase D: expected-users classroom phase - -Unless skipped, run `expected-users` clients over a controlled 5–10 second -arrival window using the selected measured active and startup values. Every -client uses the fixed 60-second simulation duration defined above. Set -admission max to the requested expected-user count. Record simulation queue wait -separately from sandbox-start-slot wait, per-client phase timestamps, active and -waiting peaks, lifecycle/polling peaks, completion/failure counts, fairness, and -cleanup. - -The output reports technical queue timing separately from UX policy: - -```text -technical minimum = rounded max(queue p95 * 1.5, queue p99 * 1.25, - queue max * 1.10) -UX policy ceiling = max-user-wait -``` - -When the technical minimum exceeds the UX ceiling, the classroom policy is -marked `infeasible`. The report includes the technical minimum and the UX -ceiling, but emits no insufficient queue-timeout value in `capacity.env`; it -does not hide the conflict by lowering admission or active capacity. When the -technical minimum fits within the UX ceiling, that technical value becomes the -review-only queue-timeout recommendation. - -### Optional compilation phase - -Version 1 does not run a separate compile-concurrency campaign unless a -representative compile workload is already available within the deadline. When -not run, `COMPILE_MAX_CONCURRENT` remains `not calibrated`, and the output -records the current derived/default value without recommending a new one. - -## Safety and deadline handling - -Safety observations use sustained windows rather than single samples: - -- CPU p95 at or above `max-cpu` for a sustained phase window stops escalation. -- CPU at or above 95%, severe iowait, or available memory below - `max(2 GiB, 10% of host RAM)` stops the current phase after the sample window. -- OOM, Docker daemon failure, backend exit, host probe failure, and cleanup leak - stop immediately. - -Every phase checks the global deadline. On deadline, the current run is cleaned -up, remaining phases are skipped, and the result is marked partial with reduced -confidence. Owned containers and backend processes are cleaned in `finally`; -unrelated containers are never removed. - -## Result schema and artifacts - -The root JSON object contains: - -- `schemaVersion` and `policyVersion` -- `startedAt`, `completedAt`, elapsed duration, and `partial`/`stopReason` -- CLI policy and phase plan -- reproducibility fingerprint: Git SHA/dirty state, host architecture/CPU/RAM, - Docker client/server/storage details, sandbox image ID/digest, sandbox CPU/RAM - limits, and every effective Capacity value -- raw phase measurements and samples -- safety events and cleanup verification -- recommendations with value, confidence (`HIGH`, `MEDIUM`, `LOW`), measured - basis, and warnings - -The `SANDBOX_START_SLOT_TIMEOUT_MS` recommendation is explicit: after a -classroom or startup-tuning phase has directly measured slot waits, calculate -`max(productionDefault, p99 * 2, maxObserved * 1.5)`, round up to the nearest -second, and include the measured p99/max, technical value, production default, -and selected value. If no slot wait was measured, emit `not calibrated` and do -not add the variable to `capacity.env`. Values outside the configuration range -are reported as `out-of-range` and are omitted from `capacity.env` rather than -silently clamped. - -The Markdown report summarizes the same result for review. `capacity.env` is a -commented, unapplied proposal containing only directly measured recommendations; -`COMPILE_MAX_CONCURRENT` is included only when calibrated. All artifacts remain -under the ignored `capacity-test-results/` tree. - -## Host probes - -Required before any non-dry-run load: - -- Git SHA/dirty state and Node version -- process architecture and logical CPU count -- Docker client/server health, version, architecture, visible CPU/RAM, storage - driver, and sandbox image ID/digest -- effective sandbox CPU/memory limits and relevant Capacity defaults -- at least one usable CPU signal and one usable memory-availability signal for - sustained safety checks - -Physical RAM, load average, swap, iowait, thermal pressure, memory-pressure -detail, and per-process backend/Docker metrics are optional enrichment probes. -Missing optional values are recorded as `null` and lower confidence. If no -usable memory or CPU safety signal exists, `--dry-run` may still inspect and -plan, but load phases abort before starting. - -## Testing - -- Pure policy tests cover input validation, bounded candidate planning, active - selection, refinement, startup diminishing returns, timeout formulas, safety - margins, confidence, and deadline truncation. -- Measurement tests use deterministic fake status/event/sample sources and verify - phase separation, run-ID filtering, cleanup, and raw-data preservation. -- The existing Capacity validation test retains independent assertions for - runtime profile, lifecycle peak, polling peak, completion, and cleanup. -- CLI smoke tests cover `--dry-run`, invalid arguments, output generation, and - non-application of `capacity.env`. - -## Documentation - -Extend the authoritative Capacity documentation with the calibration command, -phase meanings, safety behavior, direct-measurement restriction, report schema, -review/apply workflow, expected runtime, and rerun triggers: -CPU/RAM allocation, Docker platform/runtime, sandbox resource limits, or major -workload changes. diff --git a/docs/superpowers/specs/2026-10-07-optional-gateway-auth-ubuntu-vbox-design.md b/docs/superpowers/specs/2026-10-07-optional-gateway-auth-ubuntu-vbox-design.md deleted file mode 100644 index 360c31559..000000000 --- a/docs/superpowers/specs/2026-10-07-optional-gateway-auth-ubuntu-vbox-design.md +++ /dev/null @@ -1,143 +0,0 @@ -# Optional Gateway Authentication and Ubuntu VBox Setup - -- Status: Approved -- Date: 2026-10-07 -- Scope: deployment documentation and a commented Ubuntu bootstrap script - -## Goal - -Document and automate the tested UnoSim Docker setup on an Ubuntu Desktop VM: -Nginx provides TLS and the UnoSim gateway headers, user login is optional, and -the IP-identified gateway mode identifies clients by source IP. The browser -uses `https://unosim.vbox/` on port 443, with port 8443 retained as an alternate. - -## Current contract - -Docker mode continues to require a trusted reverse proxy. The backend still -requires the gateway secret, a trusted proxy address, a subject and the `user` -role for requests. The proxy MAY authenticate users, or it may grant access to -every client that can reach it. - -Use two distinct names for the gateway configurations: - -- **Authenticated gateway mode:** the proxy authenticates a user and derives - the UnoSim subject from that account. -- **IP-identified gateway mode:** the proxy performs no user authentication and - derives the subject from the directly observed source IP, for example - `ip-$remote_addr`, rather than the client-supplied `X-Forwarded-For` header. - This is not user authentication. - -In IP-identified mode, clients reaching the proxy from the same source IP share -an UnoSim identity and identity-based limits. NAT, VPNs or another proxy may -cause different people to share that identity. The current per-identity -admission limit is defined by the checked-out source version. Network -reachability is the access boundary in this mode: any client that can reach the -proxy can use UnoSim. - -This policy does not make origin validation or the gateway secret optional. -Nginx sets or overwrites `X-UnoSim-Gateway-Secret`, `X-UnoSim-Subject` and -`X-UnoSim-Roles` for both HTTP and WebSocket traffic. The Ubuntu setup binds the -backend to loopback. IP-identified gateway mode is intended for a trusted -private network. Authenticated gateway mode remains available for deployments -that need individual accounts or a stronger identity boundary. - -`UNOSIM_TRUSTED_PROXY` is required and configures Express's proxy trust behavior. -The current authorization middleware does not compare the actual connection -source address with that CIDR. The Ubuntu setup therefore keeps the published -backend port bound to `127.0.0.1:3000`, leaving Nginx as the intended backend -entry point. - -## Decision record approach - -Add a new ADR that supersedes the authentication-required statement in ADR -0001 while retaining ADR 0001 as the historical gateway contract. Update the -current summary documents to say that the reverse proxy and gateway headers -remain required in Docker mode while user authentication is optional. - -## Documentation changes - -Update these files consistently: - -- `README.md` — explain the required gateway variables and distinguish gateway - identity from optional user authentication. -- `docs/SECURITY.md` — describe authenticated and IP-identified gateway - configurations, their identity boundaries and access implications; do not - claim that the backend enforces the `UNOSIM_TRUSTED_PROXY` source address. -- `docs/INSTALL_SERVER.md` — document the proxy contract, optional login, and - exact-origin allowlist without presenting the IP mode as an authentication - mechanism. Explain that the Nginx proxy overwrites the gateway secret, subject - and roles for both HTTP and WebSocket traffic. -- `docs/ARCHITECTURE.md` — align the runtime and WebSocket summaries. -- `docs/README.md` — index the Ubuntu guide and clarify the Docker installation - entry. -- Add `docs/adr/0008-optional-gateway-authentication.md` to amend the relevant - decision in ADR 0001 while preserving its history. -- Add root-level `README_UbuntuServer.md` for a clean Ubuntu Desktop install. -- Add `scripts/install-ubuntu-vbox.sh`, the commented Bash bootstrap script. - -## Ubuntu bootstrap script - -Add a commented Bash script under `scripts/` that runs from an existing UnoSim -checkout. It must: - -1. Require Ubuntu and an explicit `--lan-ip` IPv4 argument so it does not - accidentally select a NAT or Docker interface; verify the address is - assigned to the host. -2. Install Ubuntu packages for Docker Engine, Compose v2, Buildx, Nginx and - OpenSSL; add the invoking user to the Docker group. -3. Create the runtime directories and a mode-0600 `.env` with a generated - gateway secret, Docker socket group ID, loopback trusted proxy, and exact - WebSocket origins for `unosim.vbox`, the LAN IP, localhost and - `127.0.0.1`, with the default HTTPS port omitted and port 8443 explicit. -4. Build the sandbox image, build/start the backend Compose service, and keep - the backend bound to `127.0.0.1:3000`. -5. Generate a self-signed TLS certificate whose SANs include `unosim.vbox`, - localhost, `127.0.0.1`, and the supplied LAN IPv4. -6. Configure Nginx on loopback and the supplied LAN IPv4 at ports 443 and 8443. - Nginx must pass the gateway secret and role, derive the subject from - `$remote_addr`, forward HTTP and WebSocket traffic, and omit HTTP Basic Auth. -7. Validate Nginx configuration before reload and report the URLs, certificate - path, and client-side steps still required. - -The script must not change UFW policy, router settings, client DNS, client trust -stores, hosts files, or install/enable an SSH server. It must avoid printing -generated secrets and abort without overwriting an existing `.env` or UnoSim -Nginx site. - -## Client and network steps in the README - -The README will cover: - -- reserving the VM's LAN IPv4 in the router; -- resolving `unosim.vbox` through router-local DNS for the whole LAN or a hosts - entry on a single client; -- trusting the generated self-signed certificate on each client; -- optional temporary certificate transfer over SSH and disabling the SSH - socket afterward; -- allowing TCP 443 (and optionally 8443) if a host firewall is active; -- verifying Docker readiness and testing a simulation from the client. - -The guide will use the UnoSim checkout containing these files. It will describe -the per-identity admission limit as a property of the checked-out source version -rather than assume a value that may not yet be merged. - -## Out of scope - -- Changing application authorization code or removing the gateway secret, - proxy-header validation, Docker isolation, rate limits, or origin checks. -- Automatically configuring a router, DHCP reservation, or client operating - system. -- Recommending IP-identified gateway mode for an untrusted or public network. - -## Acceptance criteria - -- No current summary document says that user authentication is mandatory for - every Docker deployment. -- The documentation clearly distinguishes the mandatory gateway trust contract - from optional user authentication. -- The IP-identified configuration accurately explains shared-IP identity and - its access consequences, with the effective admission limit tied to the - checked-out source version. -- The Ubuntu README and script describe the same hostname, ports, certificate, - identity headers and WebSocket origins as the tested VBox setup. -- Existing unrelated working-tree changes remain untouched. diff --git a/scripts/ubuntu-vbox-render.sh b/scripts/ubuntu-vbox-render.sh index 6136f8b8c..84c9fc155 100755 --- a/scripts/ubuntu-vbox-render.sh +++ b/scripts/ubuntu-vbox-render.sh @@ -59,7 +59,7 @@ render_env() { printf 'UNOSIM_ALLOWED_WS_ORIGINS=%s\n' "$origins" printf 'UNOSIM_BIND_ADDRESS=127.0.0.1\n' # Measured on the reference VBox VM (8 vCPU, 11 GiB): 30 learners plus reserve for - # extra tabs and restarts; see docs/CAPACITY_VALIDATION_PLAN.md before changing it. + # extra tabs and restarts; see docs/CAPACITY_MODEL.md before changing it. printf 'SIMULATION_ADMISSION_MAX=%s\n' "$UNOSIM_VBOX_ADMISSION_MAX" } diff --git a/server/routes/tutor.routes.ts b/server/routes/tutor.routes.ts index e1f39cb0d..a31ae483f 100644 --- a/server/routes/tutor.routes.ts +++ b/server/routes/tutor.routes.ts @@ -144,7 +144,7 @@ export function registerTutorRoutes(app: Express, deps: TutorRouteDeps = {}): vo try { const content = await resolveTutorContent(req, res, parsed.data.courseContent, parsed.data.courseContentSession, deps.courseContent, sessionStore, signal); if (signal.aborted) return; - if ((parsed.data.courseContent !== undefined || parsed.data.courseContentSession !== undefined) && !content) return; + if (res.headersSent) return; const generated = await service.generateQuestion( parsed.data.code, parsed.data.credential, @@ -212,7 +212,7 @@ export function registerTutorRoutes(app: Express, deps: TutorRouteDeps = {}): vo try { const content = await resolveTutorContent(req, res, parsed.data.courseContent, parsed.data.courseContentSession, deps.courseContent, sessionStore, signal); if (signal.aborted) return; - if ((parsed.data.courseContent !== undefined || parsed.data.courseContentSession !== undefined) && !content) return; + if (res.headersSent) return; const generated = await service.generateDialogResponse( parsed.data.code, parsed.data.history, @@ -262,6 +262,10 @@ function commitTutorSession(content: TutorRequestContent | undefined, sessions: return content.session ?? sessions.commit(content.identity, content.content); } +/** + * Returns undefined for the free Tutor, or after an error response was sent; + * callers tell both apart by `res.headersSent`. + */ async function resolveTutorContent( req: Request, res: Response, @@ -301,19 +305,30 @@ async function resolveTutorContent( try { resolved = await resolver.resolveTutorContent(request, context); } catch (error) { - if (signal.aborted) return undefined; - if (isStaleCourseContent(error)) { - // The example's revision is neither a validated snapshot of its selection nor the current one. - responseError(res, 409, "COURSE_CONTENT_STALE", "Der Kursinhalt wurde aktualisiert. Bitte das Beispiel neu laden."); - return undefined; - } - responseError(res, 400, "INVALID_REQUEST", "Der Course-Content-Kontext ist ungültig oder nicht mehr aktiv."); + if (!signal.aborted) respondToUnresolvedContent(res, error); return undefined; } if (signal.aborted) return undefined; return { identity, content: prepareTutorCourseContentSession(resolved) }; } +/** Sends the error response for unresolved Course Content; the free Tutor needs none. */ +function respondToUnresolvedContent(res: Response, error: unknown): void { + // Course Content SSOT 11, case G: without the current revision the free Tutor answers. + if (isSnapshotUnavailable(error)) return; + if (isStaleCourseContent(error)) { + // The example's revision is neither a validated snapshot of its selection nor the current one. + responseError(res, 409, "COURSE_CONTENT_STALE", "Der Kursinhalt wurde aktualisiert. Bitte das Beispiel neu laden."); + return; + } + responseError(res, 400, "INVALID_REQUEST", "Der Course-Content-Kontext ist ungültig oder nicht mehr aktiv."); +} + +function isSnapshotUnavailable(error: unknown): boolean { + return error instanceof ExamplesError + && (error.code === "SOURCE_UNAVAILABLE" || error.code === "RATE_LIMITED" || error.code === "LOAD_CAPACITY_EXCEEDED"); +} + function isStaleCourseContent(error: unknown): boolean { return error instanceof ExamplesError && (error.code === "INVALID_REVISION" || error.code === "EXAMPLE_NOT_FOUND"); } diff --git a/ssot/ssot_function_definition_TutorQualityEvaluation.md b/ssot/ssot_function_definition_TutorQualityEvaluation.md index b6ebc67b4..eb28ba9e9 100644 --- a/ssot/ssot_function_definition_TutorQualityEvaluation.md +++ b/ssot/ssot_function_definition_TutorQualityEvaluation.md @@ -3,7 +3,7 @@ Status: normative model for Tutor quality evaluation. Supersedes: the earlier execution-runner specification and the earlier semantic-layer design (see the historical glossary entries in §2). Implementation -plan: `docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md` (not normative). +plan: `docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md` (not normative). > **Conformance.** `main` implements this document; Appendix A records the > conformance status and lists no known deviation. A deviation found later is @@ -62,7 +62,7 @@ key under §6.4. | *Historical:* Stage 1 | Name of the deterministic runtime gate; still the title of `ssot_function_definition_TutorQuality.md`. | | *Historical:* Stage 2A, "Stage A" | Earlier name of the execution run. Its former SSOT and runbook were removed; the runbook moved to Appendix B. Survives only in artifact identifiers (§11.4) and internal type names; not used in the current model. | | *Historical:* Stage 2B, Protocol A | Removed semantic layer and its non-reproducible judge protocol. Not normative. | -| *Historical:* Protocol B, G1 | The minimal-Judge protocol and its one-time acceptance measurement (schema ≥ 95 %, evidence ≥ 90 %, repeatability ≥ 80 %), documented in `docs/tutor-quality-judge-smoke-2026-10-01.md`. G1 is bound to the Judge prompt revision it measured; it is not a runtime gate. | +| *Historical:* Protocol B, G1 | The minimal-Judge protocol and its one-time acceptance measurement (schema ≥ 95 %, evidence ≥ 90 %, repeatability ≥ 80 %), documented in `docs/archive/tutor-quality/tutor-quality-judge-smoke-2026-10-01.md`. G1 is bound to the Judge prompt revision it measured; it is not a runtime gate. | ## 3. Architecture @@ -709,8 +709,8 @@ fields requires a new schema identifier. ## 12. Quality verdict Evidence for this rule: the real-provider baseline -(`docs/tutor-quality-baseline-2026-10-03.md`) and the remediation rerun of the -repaired strategy cases (`docs/tutor-quality-remediation-rerun-2026-10-03.md`). +(`docs/archive/tutor-quality/tutor-quality-baseline-2026-10-03.md`) and the remediation rerun of the +repaired strategy cases (`docs/archive/tutor-quality/tutor-quality-remediation-rerun-2026-10-03.md`). In those runs, deterministic application violations never occurred; stable LLM-dependent findings recurred in 3 of 3 samples across runs; and the same case with byte-identical prompts showed a finding once in 1 of 3 and once in @@ -822,7 +822,7 @@ dashboard, workflow engine, or statistics platform. ## Appendix A – Conformance status (non-normative) Verified against `main` @ `425bf362` plus the conformance fix that follows it -(PRs #128–#132 of `docs/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`). Closed since +(PRs #128–#132 of `docs/archive/tutor-quality/UNOSIM_TUTOR_QUALITY_AUTOMATION_PLAN.md`). Closed since the first version of this document: the EXPAND continuation (R-EXP-1..4, PR A), the corpus, rating-band, and binding contract (R-TURN-4, R-RAT-1..6, R-AUT-1..3, R-EXPD-1, R-BLK-1/2, PR B), the Judge quote, provenance, and report diagnostics diff --git a/tests/server/calibrate-capacity-cli.test.ts b/tests/server/calibrate-capacity-cli.test.ts index c3cf5c0e2..22558c195 100644 --- a/tests/server/calibrate-capacity-cli.test.ts +++ b/tests/server/calibrate-capacity-cli.test.ts @@ -36,7 +36,7 @@ describe("capacity calibration CLI", () => { it("is documented as a review-only workflow with stable artifact names", () => { const packageJson = JSON.parse(fs.readFileSync(path.resolve(process.cwd(), "package.json"), "utf8")) as { scripts?: Record }; - const documentation = fs.readFileSync(path.resolve(process.cwd(), "docs/CAPACITY_VALIDATION_PLAN.md"), "utf8"); + const documentation = fs.readFileSync(path.resolve(process.cwd(), "docs/CAPACITY_MODEL.md"), "utf8"); expect(packageJson.scripts?.["capacity:calibrate"]).toBe("tsx scripts/calibrate-capacity.ts"); expect(documentation).toContain("npm run capacity:calibrate -- --expected-users 200"); expect(documentation).toContain("capacity-calibration.json"); diff --git a/tests/server/routes/tutor.routes.test.ts b/tests/server/routes/tutor.routes.test.ts index 3128c9880..5513d4d54 100644 --- a/tests/server/routes/tutor.routes.test.ts +++ b/tests/server/routes/tutor.routes.test.ts @@ -353,6 +353,25 @@ describe("Tutor HTTP route", () => { expect(service.generateQuestion).not.toHaveBeenCalled(); }); + it.each(["SOURCE_UNAVAILABLE", "RATE_LIMITED", "LOAD_CAPACITY_EXCEEDED"] as const)( + "answers as the free Tutor without a session when the Course Content source is unavailable (%s)", + async (code) => { + const resolver = { resolveTutorContent: vi.fn().mockRejectedValue(new ExamplesError(code, "GitHub is unavailable")) }; + const service = { generateQuestion: vi.fn().mockResolvedValue({ result: { question: "Was macht pinMode?" }, model: "test-model" }) }; + const listening = await start(service, false, resolver); + server = listening.server; + const response = await post(listening.url, "/api/tutor/question", { + code: "void setup(){}", + credential: "request-only-secret", + courseContent: { repository: "owner/repo", ref: "main", revision: "a".repeat(40) }, + }); + expect(response.status).toBe(200); + expect(response.body).not.toHaveProperty("courseContentSession"); + expect(service.generateQuestion).toHaveBeenCalledOnce(); + expect(service.generateQuestion.mock.calls[0]?.[4]).toBeUndefined(); + }, + ); + it("aborts the shared route and Course Content signal when the client disconnects", async () => { const content = { repository: "owner/repo" as const, diff --git a/tests/server/services/tutor/evaluation/quality-verdict.test.ts b/tests/server/services/tutor/evaluation/quality-verdict.test.ts index b75e0a2fb..fbe4a2469 100644 --- a/tests/server/services/tutor/evaluation/quality-verdict.test.ts +++ b/tests/server/services/tutor/evaluation/quality-verdict.test.ts @@ -9,7 +9,7 @@ import { } from "../../../../../server/services/tutor/evaluation/quality-verdict"; // Synthetic, anonymized fixtures shaped after the baseline and rerun data -// (docs/tutor-quality-baseline-2026-10-03.md, docs/tutor-quality-remediation-rerun-2026-10-03.md). +// (docs/archive/tutor-quality/tutor-quality-baseline-2026-10-03.md, docs/archive/tutor-quality/tutor-quality-remediation-rerun-2026-10-03.md). function sample(scenarioId: string, sampleIndex: number, overrides: Partial = {}): TutorQualityVerdictSample { return { scenarioId, sampleIndex, executionStatus: "completed", violations: [], ...overrides };