Record FluentAI using real OpenAI responses

README.md

@@ -4,11 +4,11 @@ FluentAI is an OpenAI-powered agentic language-learning demo. It feels like a co
 
 FluentAI now requires `OPENAI_API_KEY` for real lesson and conversation runs. Tests use mocked OpenAI responses so CI never needs secrets.
 
-[![Current FluentAI lesson screen — mocked model responses](docs/media/current-lesson.png)](docs/media/workflow-demo.mp4)
+[![FluentAI lesson generated through OpenAI](docs/media/current-lesson.png)](docs/media/workflow-demo.mp4)
 
-**[Watch the narrated 83-second workflow demo](docs/media/workflow-demo.mp4)** · [Download MP4](https://raw.githubusercontent.com/Jo2234/FluentAI/main/docs/media/workflow-demo.mp4) · [Reproduce the recording](docs/DEMO.md)
+**[Watch the narrated 93-second walkthrough](docs/media/workflow-demo.mp4)** · [Download MP4](https://raw.githubusercontent.com/Jo2234/FluentAI/main/docs/media/workflow-demo.mp4) · [Reproduce the recording](docs/DEMO.md)
 
-This is the current browser app with **mocked model responses and temporary learner memory**, labelled throughout the recording. It shows a lesson, a deliberate quiz mistake, saved XP and spaced review, text conversation, and progress surviving reload. It demonstrates the interface and state flow; live model quality, in-app voice, and camera behavior are not shown. Explanatory English narration was added after recording. [Captions](docs/media/narration.vtt) · [Narration script](docs/media/narration.json).
+This recording runs the real browser app against OpenAI. It shows a generated Spanish lesson, specific quiz feedback, an adaptive text conversation, and learner memory surviving reload. Five Responses API calls completed during capture. English narration explains the visible actions; optional captions and the [narration script](docs/media/narration.json) accompany the video. This recording uses text conversation; microphone, Realtime voice, and camera are outside its scope. [Captions](docs/media/narration.vtt).
 
 ## What works now
 

docs/DEMO.md

@@ -1,47 +1,35 @@
-# Recorded workflow demo
+# Live API walkthrough
 
-[Watch the MP4](media/workflow-demo.mp4) or inspect the [current lesson screenshot](media/current-lesson.png). Both are captured from the current shared browser/Electron renderer served through the real Python HTTP bridge, not from the historical mockups.
+[Watch the narrated recording](media/workflow-demo.mp4) · [Captions](media/narration.vtt) · [Capture metadata](media/recording.json)
 
-## What the recording proves
+The 93-second recording runs the production Python bridge and shared browser renderer against OpenAI's Responses API. Five requests completed using `gpt-5.5-2026-04-23`, consuming 2,136 input and 623 output tokens. Responses were observed without modification; there was no fixture provider or response substitution.
 
-| Approximate time | Workflow |
+| Time | Visible behavior |
 | --- | --- |
-| 0–6 s | Home loads an isolated learner profile with no prior practice |
-| 6–26 s | Start a lesson from weak-topic memory; answer six quiz questions |
-| 26–41 s | Submit one deliberate mistake; see feedback, XP, and scheduled review |
-| 41–61 s | Practice two text conversation replies using the same learner memory |
-| 61–83 s | Inspect memory, reload the app, and see persisted progress |
+| 2–13 s | Load a fresh Spanish learner and start a lesson |
+| 13–34 s | Read generated vocabulary and answer six questions, including one deliberate mistake |
+| 35–46 s | See specific feedback, a 5/6 result, 55 XP, and scheduled review |
+| 47–77 s | Practice with the text tutor; it corrects the tense and asks a follow-up question |
+| 78–93 s | Inspect saved memory and reload; 55 XP and three reviews persist |
 
-The yellow banner remains visible throughout: **mocked model responses, temporary learner profile, no API calls**. The Python harness replaces the provider with deterministic curriculum/tutor scaffolding. The renderer, HTTP bridge, local grading, transaction logic, review scheduling, and persistence remain real. It changes provider-success log wording so the recording does not claim a live OpenAI call.
+Capture used a disposable default A1 profile with onboarding and placement marked complete before recording. It did not read existing personal learner data. The top 48 pixels of a recording-only status banner were cropped from the final 1440×952 video; the timeline is continuous, with no speed changes or generated UI. Real calls, replies, timings, and initial/final state were retained privately for verification. Credentials and request headers are excluded from published artifacts.
 
-This is workflow evidence, not a demonstration of OpenAI output quality, language-course quality, Realtime voice, camera accuracy, or desktop signing/distribution. The screenshot includes the same disclosure. No microphone/camera permission is requested. No personal profile, `.env`, or API credential is needed.
+English narration was added after capture using the generic Kokoro `af_heart` voice. Its [timed script and hashes](media/narration.json) describe the actual visible results. This walkthrough uses text conversation; Realtime voice, microphone, camera, and desktop packaging are outside its scope.
 
-## Reproduce
+## Run the real app
 
-Use Python 3.10+, Node.js, FFmpeg (`ffmpeg` and `ffprobe` on PATH), and a Playwright Chromium installation. From the repository root:
+Install the project as described in the root README, configure `OPENAI_API_KEY` in the server environment, and start:
 
 ```bash
-python3 -m venv .venv
-source .venv/bin/activate
-python -m pip install -e '.[dev]'
-npm ci
-npm install --no-save --package-lock=false playwright@1.62.1
-npx playwright install chromium
-node scripts/record_portfolio_demo.cjs
+python -m fluent_ai.web --port 7860
 ```
 
-The recording script starts `scripts/portfolio_demo.py` on localhost, creates a disposable profile, drives visible controls, blocks external browser requests, and shuts down the server afterward. It checks for browser/bridge errors, quiz XP, scheduled review, and persistence after reload. It writes `docs/media/current-lesson.png`, `workflow-demo.mp4`, and `recording.json`; FFmpeg converts the captured browser video to a silent H.264 MP4. This recreates the silent base recording with on-screen chapter labels; the published MP4 additionally contains the post-production narration described below.
+Open `http://127.0.0.1:7860`, use a separate learner profile, complete placement, then try a lesson and text conversation. The recording used `OPENAI_MODEL=gpt-5.5`, reasoning effort `low`, and verbosity `low`. Model-generated lessons, replies, and timings vary; inspect the actual results when narrating a new capture. Paid calls are required for ordinary app use.
 
-Optional environment overrides: `PYTHON_EXECUTABLE` for the interpreter, `CHROMIUM_EXECUTABLE` for an already installed browser, and `DEMO_PORT` for an available localhost port (default 7862). An existing Playwright installation can be resolved through `NODE_PATH`. The checked recording reused an installed Chromium binary; the commands above install Playwright's own compatible browser for a fresh reproduction. Timing, session IDs, and timestamps may vary; `recording.json` records the actual duration and source commit.
+## Offline development harness
 
-To inspect the workflow manually, run `python scripts/portfolio_demo.py` and open `http://127.0.0.1:7861`. Ctrl-C deletes the temporary state. This explicit development harness is not a production offline-tutor mode; ordinary app entrypoints continue to require OpenAI.
+`scripts/portfolio_demo.py` and `scripts/record_portfolio_demo.cjs` remain available as explicit mock-provider development tools. They write the older labelled workflow capture under ignored `cache/offline-recording/`, keeping it separate from the published live API assets. Tests continue to mock providers so CI requires no credentials or paid requests.
 
-## Checks and provenance
+## Validation
 
-The screenshot and video are generated by the two scripts above. `recording.json` identifies the pre-recording source commit and automated assertions. The Git commit containing the artifacts also contains the exact harness and driver. Original design mockups and their notices remain in `mockups/`. The repository's MIT license covers Johan Vaz's original work; third-party dependencies retain their own terms.
-
-## Explanatory narration
-
-English voiceover and optional embedded captions explain the committed recording. [Timed cue text and provenance](media/narration.json) and [WebVTT captions](media/narration.vtt) accompany the MP4. The generic synthetic Kokoro `af_heart` voice was generated locally after capture; it is not FluentAI's Realtime voice feature.
-
-The original H.264 video stream and its timing are unchanged. For another recording, re-time the cues to the captured actions, generate narration within those windows, and mux audio/captions while copying the video stream. The capture command does not generate speech. The narration manifest preserves source/output hashes, and Git history retains the silent original. Validation on 13 September 2026 confirmed unchanged video packets, AAC narration, ten captions, and all speech ending inside its allotted window.
+The live capture verified five completed Responses requests, specific wrong-answer feedback, three scheduled reviews, and saved progress after reload, with no browser errors or failed bridge responses. Separately, 133 Python tests passed without skips and all 11 JavaScript renderer scenarios passed. The lesson screenshot comes from the new API-backed recording. Original mockups, earlier captures, and historical validation remain in Git history.

docs/media/narration.json

@@ -1,87 +1,77 @@
 {
   "generated_at": "2026-09-13",
-  "engine": "kokoro-onnx 0.6.1 / Kokoro v1.0",
+  "source_video_sha256": "1e09bebc39ea6a6488e86d78d8f512c65af28f2db27a98e520fd26b2b6f39f4f",
+  "output_video_sha256": "725b096f558da1ff4a15e94e86659abc8f06df6b9a2b558a7ad9e954c92bfb8e",
+  "duration_seconds": 93.4,
+  "engine": "Kokoro v1.0 / kokoro-onnx 0.6.1",
   "voice": "af_heart",
-  "language": "en-US",
-  "purpose": "Synthetic explanatory voiceover added in post-production; not app-generated speech.",
-  "source_video_sha256": "0507e4e7f6e495552f4c7d7c065b642256236b7af1ef1668476f3d02231595ba",
-  "output_video_sha256": "4cf66e258ca8003fb10239d44e62486a27065febecdaa2e46816d9c887585d73",
-  "source_duration_seconds": 82.7,
-  "video_stream": "Copied unchanged; no cuts or speed changes.",
-  "audio_codec": "AAC",
-  "audio_sample_rate": 24000,
+  "purpose": "Explanatory voiceover over a recording of real API-backed app execution.",
+  "video_stream": "Copied unchanged from the supplied recording; see capture provenance for edits.",
+  "app_audio": "No source app audio",
   "model_source": "https://github.com/thewh1teagle/kokoro-onnx/releases/tag/model-files-v1.0",
-  "model_license": "Apache-2.0",
   "cues": [
     {
-      "start": 0.5,
-      "end": 7.2,
-      "text": "Fluent A I links lessons with learner memory. This demo uses scripted model responses.",
-      "actual_end": 6.047,
-      "speed": 0.95
+      "start": 2.5,
+      "end": 9.0,
+      "text": "FluentAI remembers your level and weak topics. This lesson focuses on the Spanish past tense.",
+      "actual_end": 8.601,
+      "speed": 0.97
     },
     {
-      "start": 7.8,
-      "end": 14.2,
-      "text": "A temporary learner profile opens a past-tense lesson, with a visible selection reason.",
-      "actual_end": 13.176,
-      "speed": 0.95
+      "start": 13.5,
+      "end": 22.5,
+      "text": "OpenAI builds a lesson around yesterday, with useful vocabulary, examples, and six practice questions.",
+      "actual_end": 21.009,
+      "speed": 0.97
     },
     {
-      "start": 14.8,
-      "end": 25.8,
-      "text": "The learner answers six questions, mixing vocabulary, sentences, and translation. One answer is deliberately wrong so we can follow the feedback.",
-      "actual_end": 24.464,
-      "speed": 0.95
+      "start": 23.0,
+      "end": 31.0,
+      "text": "Here, I answer the quiz and deliberately write ma\u00f1ana where the question asks for yesterday.",
+      "actual_end": 28.525,
+      "speed": 0.97
     },
     {
-      "start": 26.5,
-      "end": 33.3,
-      "text": "Five answers are correct. The mistake gets feedback and becomes a scheduled review item.",
-      "actual_end": 32.175,
-      "speed": 0.95
+      "start": 35.0,
+      "end": 44.0,
+      "text": "The app explains the mistake, awards fifty-five experience points, and adds the weak areas to the review queue.",
+      "actual_end": 41.677,
+      "speed": 0.97
     },
     {
-      "start": 34.0,
-      "end": 40.8,
-      "text": "Back home, the score, experience points, and review queue reflect the completed lesson.",
-      "actual_end": 39.227,
-      "speed": 0.95
+      "start": 47.5,
+      "end": 56.5,
+      "text": "The text tutor picks up the same goal, asking what I did yesterday to reinforce the lesson.",
+      "actual_end": 52.94,
+      "speed": 0.97
     },
     {
-      "start": 41.8,
-      "end": 47.3,
-      "text": "Now text practice picks up the same past-tense topic.",
-      "actual_end": 45.021,
-      "speed": 0.95
+      "start": 57.0,
+      "end": 64.0,
+      "text": "I reply with another deliberate error: using the present tense, voy, after ayer.",
+      "actual_end": 62.013,
+      "speed": 0.97
     },
     {
-      "start": 48.0,
-      "end": 53.4,
-      "text": "A short Spanish reply receives feedback and a follow-up prompt.",
-      "actual_end": 51.947,
-      "speed": 0.95
+      "start": 65.0,
+      "end": 74.0,
+      "text": "The tutor corrects voy to fui, preserves the meaning, and asks what I bought. It saves that correction for review.",
+      "actual_end": 71.997,
+      "speed": 0.97
     },
     {
-      "start": 54.0,
-      "end": 60.8,
-      "text": "A second reply continues the exchange. These are scripted responses, exercising the app's practice workflow.",
-      "actual_end": 60.613,
-      "speed": 1.083
+      "start": 78.0,
+      "end": 85.0,
+      "text": "The memory inspector shows the saved learner state, including the correction and review queue.",
+      "actual_end": 83.141,
+      "speed": 0.97
     },
     {
-      "start": 62.0,
-      "end": 68.8,
-      "text": "The memory inspector exposes the saved profile, skills, topic scores, and mistakes.",
-      "actual_end": 67.675,
-      "speed": 0.95
-    },
-    {
-      "start": 70.0,
-      "end": 81.8,
-      "text": "Reloading keeps the lesson result and review queue. This demonstrates saved progress across the workflow; live voice and model quality are outside this recording.",
-      "actual_end": 80.176,
-      "speed": 0.95
+      "start": 85.5,
+      "end": 92.5,
+      "text": "Reloading keeps the experience points and scheduled reviews, ready for the next session.",
+      "actual_end": 90.428,
+      "speed": 0.97
     }
   ]
 }

docs/media/narration.vtt

@@ -1,56 +1,50 @@
 WEBVTT
 
 1
-00:00:00.500 --> 00:00:06.047
-Fluent A I links lessons with learner memory.
-This demo uses scripted model responses.
+00:00:02.500 --> 00:00:08.601
+FluentAI remembers your level and weak topics.
+This lesson focuses on the Spanish past tense.
 
 2
-00:00:07.800 --> 00:00:13.176
-A temporary learner profile opens a past-tense
-lesson, with a visible selection reason.
+00:00:13.500 --> 00:00:21.009
+OpenAI builds a lesson around yesterday, with
+useful vocabulary, examples, and six practice
+questions.
 
 3
-00:00:14.800 --> 00:00:24.464
-The learner answers six questions, mixing
-vocabulary, sentences, and translation. One
-answer is deliberately wrong so we can follow
-the feedback.
+00:00:23.000 --> 00:00:28.525
+Here, I answer the quiz and deliberately write
+mañana where the question asks for yesterday.
 
 4
-00:00:26.500 --> 00:00:32.175
-Five answers are correct. The mistake gets
-feedback and becomes a scheduled review item.
+00:00:35.000 --> 00:00:41.677
+The app explains the mistake, awards fifty-five
+experience points, and adds the weak areas to
+the review queue.
 
 5
-00:00:34.000 --> 00:00:39.227
-Back home, the score, experience points, and
-review queue reflect the completed lesson.
+00:00:47.500 --> 00:00:52.940
+The text tutor picks up the same goal, asking
+what I did yesterday to reinforce the lesson.
 
 6
-00:00:41.800 --> 00:00:45.021
-Now text practice picks up the same past-tense
-topic.
+00:00:57.000 --> 00:01:02.013
+I reply with another deliberate error: using the
+present tense, voy, after ayer.
 
 7
-00:00:48.000 --> 00:00:51.947
-A short Spanish reply receives feedback and a
-follow-up prompt.
+00:01:05.000 --> 00:01:11.997
+The tutor corrects voy to fui, preserves the
+meaning, and asks what I bought. It saves that
+correction for review.
 
 8
-00:00:54.000 --> 00:01:00.613
-A second reply continues the exchange. These are
-scripted responses, exercising the app's
-practice workflow.
+00:01:18.000 --> 00:01:23.141
+The memory inspector shows the saved learner
+state, including the correction and review
+queue.
 
 9
-00:01:02.000 --> 00:01:07.675
-The memory inspector exposes the saved profile,
-skills, topic scores, and mistakes.
-
-10
-00:01:10.000 --> 00:01:20.176
-Reloading keeps the lesson result and review
-queue. This demonstrates saved progress across
-the workflow; live voice and model quality are
-outside this recording.
+00:01:25.500 --> 00:01:30.428
+Reloading keeps the experience points and
+scheduled reviews, ready for the next session.

docs/media/recording.json

@@ -1,23 +1,35 @@
 {
-  "source_commit": "85241bc625a2435324fffeca0bda2b0847d3f143",
-  "duration_seconds": 82.7,
-  "size_bytes": 2226409,
-  "viewport": "1440x1000",
-  "model_boundary": "Mocked Python provider; actual renderer, bridge, grading, state persistence, and review scheduling",
-  "external_browser_requests": "Blocked",
-  "checks": [
-    "Lesson awarded XP",
-    "Review queue populated",
-    "XP and review queue survive page reload",
-    "No browser errors or failed bridge responses"
-  ],
+  "source_commit": "3aff411514bd90f0fe887d02c66714b10b08bb30",
+  "api_calls": 5,
+  "totals": {
+    "input_tokens": 2136,
+    "output_tokens": 623,
+    "total_tokens": 2759
+  },
+  "capture": "Continuous Chromium browser renderer; actual production bridge and OpenAIProvider; SDK response creation wrapped only to observe unchanged requests/responses. No response substitution or fixtures.",
+  "state": "Disposable default Spanish learner; onboarding and placement marked complete before capture. Initial-state.json retains exact defaults. No private existing learner state used.",
+  "editing": "Cropped top 48px recording-only banner; no timeline cuts, no synthesized UI, no speed changes.",
   "excluded": [
-    "Live model quality",
     "Realtime voice",
+    "Microphone",
     "Camera",
-    "Signed desktop distribution"
+    "Desktop packaging"
   ],
-  "original_capture_size_bytes": 1312712,
-  "sha256": "4cf66e258ca8003fb10239d44e62486a27065febecdaa2e46816d9c887585d73",
-  "post_production": "Synthetic explanatory narration and captions added; see narration.json. In-app realtime voice remains excluded."
+  "checks": [
+    "All five real Responses calls completed",
+    "Lesson score 5/6",
+    "55 XP persisted after reload",
+    "Review queue grew from 0 to 2 after lesson to 3 after tutor correction",
+    "No browser errors or failed bridge responses",
+    "133 Python tests without skips",
+    "11 JavaScript renderer scenarios and syntax/redaction checks passed"
+  ],
+  "video_duration_seconds": 93.4,
+  "viewport": "1440x1000",
+  "output_dimensions": "1440x952",
+  "recorded_at": "2026-09-13",
+  "provider": "OpenAI Responses API",
+  "model": "gpt-5.5-2026-04-23",
+  "fixture_provider": false,
+  "narrated_video_sha256": "725b096f558da1ff4a15e94e86659abc8f06df6b9a2b558a7ad9e954c92bfb8e"
 }