diff --git a/AGENTS.md b/AGENTS.md index f23dd2c..a445d62 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,8 +20,8 @@ Conversation Mode: ## Goal Mode Orchestration - Treat Goal mode as a persistent completion contract: state the outcome, evidence of success, constraints, and stopping conditions before long-running work. - Follow the Codex Goals pattern from https://developers.openai.com/cookbook/examples/codex/using_goals_in_codex: keep the objective visible, verify progress from artifacts or tests, and choose the next useful action from evidence. -- Keep `plan.md` as the current plan and `notes.md` as the compact memory of actual progress, decisions, user preferences, and important facts to preserve across turns. -- The main goal orchestrator must write actual progress and important things to remember in `notes.md` whenever meaningful work is completed or priorities change. +- Keep `plan.md` as the current plan and `notes.md` as a concise record of durable decisions and dated validation evidence. Current architecture, privacy, curriculum, packaging, limitations, and roadmap are linked from `plan.md`; update those documents when behavior changes. +- Record meaningful validation results and unresolved acceptance limits in `notes.md`; avoid accumulating task-by-task construction logs or completed roadmap items. - If another user or agent modifies `plan.md` midway, stay calm, reread it, adapt Goal mode to the updated plan, and continue from the newest valid intent instead of fighting the change. - Keep all project documents concise. Prefer short bullets with concrete facts over long narrative. diff --git a/README.md b/README.md index 23f559e..2093613 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,7 @@ Build a macOS `.app` bundle: ```bash npm run build:app -open dist/FluentAI.app +open dist/mac-arm64/FluentAI.app # Apple Silicon; use the path reported by the build ``` The desktop app calls the same Python bridge/API as the CLI and web UI, so demo behavior stays consistent. @@ -169,9 +169,12 @@ tests/ unittest suite for core, bridge, and web endpoints data/progress.json demo learner memory ``` +## Design and limitations + +See [architecture and memory](docs/GAP_AND_DESIGN.md), [privacy controls](docs/ONBOARDING_HOME_DESIGN.md), [curriculum](docs/CURRICULUM_DESIGN.md), and [packaging](docs/PACKAGING_DESIGN.md). The [product direction](docs/FLUENTAI_LIMITLESS_IDEAL.md) separates implemented behavior from limitations and remaining work; [notes](notes.md) preserves historical validation and provenance. + ## Roadmap -- Add spaced repetition scheduling. - Add richer lesson packs for Hindi and French. - Add stable browser-based voice regression tests with mocked Realtime events. - Add camera-frame fixtures for vision-context tests. diff --git a/docs/CURRICULUM_DESIGN.md b/docs/CURRICULUM_DESIGN.md index 587d94d..8315b7e 100644 --- a/docs/CURRICULUM_DESIGN.md +++ b/docs/CURRICULUM_DESIGN.md @@ -1,368 +1,28 @@ -# FluentAI Curriculum Depth Design +# Curriculum structure and authoring decisions -## Scope +## Implemented content path -This document designs Phase 5 curriculum depth without changing runtime code yet. It is grounded in commit `5a43138`, because other agents may be editing `fluent_ai/agent.py` and `fluent_ai/conversation.py` concurrently. +`fluent_ai/curriculum.py` loads packaged JSON assets for Spanish, French, and Hindi. `pyproject.toml` includes `curriculum/*.json` in the Python package. The loader caches parsed assets and returns copies so callers cannot mutate shared content. -North-star references: +The public helpers are `lesson_bank(language)`, `topics_by_level(language)`, `conversation_ladder(language)`, and `topic_lesson(language, level, topic)`. Lessons and conversations consume these helpers instead of requiring content to live in Python literals. -- Lesson Mode should generate level-aware lessons with useful phrases, grammar, examples, pronunciation notes, cultural context, micro-tasks, quizzes, and a speaking bridge. -- Multilingual support should prioritize quality: CEFR mapping, topic ladders, grammar progression, pronunciation guidance, cultural context, script support, romanization, native examples, and conversation scenarios. -- Personalization should adapt topic choice to goals, weak topics, conversation mistakes, and prior outcomes. -- Phase 5 explicitly calls for richer Spanish, French, and Hindi lessons, CEFR progression, custom goals, scenarios, debate, and work modules. +## Asset contract -Current code references at `5a43138`: +Each language asset has `language`, `schema_version`, `levels`, and `generic`. A level declares `topic_order`, `topics`, and `default_topic`; it may also declare conversation entries directly. Topic records contain: -- `fluent_ai/agent.py:TOPICS_BY_LEVEL` defines Spanish-centric topic names for A1-C2. -- `fluent_ai/agent.py:LESSON_BANK` contains six hand-written Spanish topics, mostly A1. -- `fluent_ai/agent.py:generate_lesson` uses `LESSON_BANK` only for Spanish and `_generic_lesson_bank` for non-Spanish languages. -- `fluent_ai/agent.py:generate_quiz` uses static `answers` for Spanish bank lessons and `_lesson_driven_quiz` for OpenAI or non-Spanish lessons. -- `fluent_ai/agent.py:_lesson_driven_quiz` needs at least two vocabulary pairs and two example pairs to create stable questions. -- `fluent_ai/agent.py:_generic_lesson_bank` provides one generic Hindi lesson, one generic French lesson, and one generic other-language fallback. -- `fluent_ai/conversation.py:TOPIC_LADDER` contains Spanish A1-C2 conversation topics. -- `fluent_ai/conversation.py:LANGUAGE_TOPIC_OVERRIDES` contains French and Hindi A1-A2 overrides only. -- `fluent_ai/conversation.py:conversation_topics_for` chooses language override by level, otherwise Spanish `TOPIC_LADDER`. -- `fluent_ai/openai_provider.py:OpenAIProvider.enhance_lesson` rewrites lesson vocabulary, grammar, examples, and micro-task at runtime. +- `title` and `focus_skill`; +- target/English `vocabulary` and `examples`, with optional romanization and pronunciation hints; +- a grammar explanation and exact quiz `answers`; +- optional `pronunciation_hints`, `cultural_note`, and conversation prompts with topic, complexity, opening, support, and keywords. -## Part 1: Content Architecture +Stable topic keys support persisted mastery/review records. Aliases bridge older names such as `vocabulary` to newer content keys. The loader converts rich vocabulary/examples into the pair format used by lesson generation while retaining rich records for script and pronunciation display. -Move curriculum data out of Python literals and into JSON files: +## Lookup and quality boundaries -```text -fluent_ai/ - curriculum.py - curriculum/ - spanish.json - french.json - hindi.json -``` +`topic_lesson` tries the language bank's exact topic, its alias, the level default, then generic content. If generic content is absent it can try the Spanish bank for legacy compatibility, otherwise returns `None`. Unknown language files resolve to Spanish. These are compatibility rules, not evidence of broad multilingual course coverage. -`fluent_ai/curriculum.py` should be the only runtime loader. JSON files should be static content assets with no code execution. +Preserve accents and Devanagari in authored/displayed content; write JSON as UTF-8. Hindi material should keep romanization aligned with native text. Matching normalization must not erase meaningful script characters. Provide enough vocabulary/examples and correct answer keys for deterministic quiz generation; test every declared topic and level after content edits. -The first implementation should preserve the current dict shapes so `fluent_ai/agent.py:generate_lesson`, `fluent_ai/agent.py:generate_quiz`, and `fluent_ai/conversation.py:conversation_topics_for` can be migrated with minimal logic changes. +OpenAI enhances the selected lesson at runtime, including available pronunciation/culture context. Static banks remain useful scaffolding and test fixtures, while real lessons still require OpenAI. Native-language content review and deeper Hindi/French/advanced coverage remain in the [roadmap](FLUENTAI_LIMITLESS_IDEAL.md). -### JSON Schema - -Each language file should use this top-level shape: - -```json -{ - "language": "Spanish", - "schema_version": 1, - "levels": { - "A1": { - "topic_order": ["introductions"], - "topics": { - "introductions": { - "title": "Greetings and introductions", - "focus_skill": "vocabulary", - "vocabulary": [{"target": "Me llamo", "english": "My name is", "romanization": null, "pronunciation_hint": "meh YAH-moh"}], - "grammar": "Use ser for identity and origin.", - "examples": [{"target": "Me llamo Ana.", "english": "My name is Ana.", "romanization": null}], - "answers": {"mc": "My name is", "fill_prompt": "___ Ana.", "fill": "Me llamo", "open": "Me llamo Ana.", "translation": "Soy de Singapur."}, - "pronunciation_hints": ["ll in llamo is usually like y in many dialects."], - "cultural_note": "Use mucho gusto in many first-meeting situations.", - "conversation": [{"topic": "introductions", "complexity": "beginner", "opening": "Hola, yo empiezo. ¿Cómo te llamas?", "support": "Model answer: Me llamo Ana.", "keywords": ["me", "llamo", "soy"]}] - } - }, - "default_topic": "introductions" - } - }, - "generic": {"topic": "practice", "focus_skill": "vocabulary", "vocabulary": [], "grammar": "", "examples": [], "answers": {}, "conversation": []} -} -``` - -Required topic fields: - -- `title`: display label, not used as the stable key. -- `focus_skill`: one of the current skill names such as `vocabulary`, `grammar`, `conjugations`, `listening`, `speaking`, `reading`, or `translation`. -- `vocabulary`: at least five target-English pairs; Hindi items should include `romanization`. -- `grammar`: short level-appropriate explanation, compatible with `grammar_explanation` in generated lessons. -- `examples`: at least three target-English pairs; Hindi examples should include `romanization`. -- `answers`: exact answer keys for the static quiz path. -- `pronunciation_hints`: optional list, but strongly recommended for every topic. -- `cultural_note`: optional string. -- `conversation`: one or more entries shaped exactly like current conversation ladder entries: `topic`, `complexity`, `opening`, `support`, `keywords`. - -Compatibility transforms in `fluent_ai/curriculum.py`: - -- Convert JSON `vocabulary` objects into current `[(target, english), ...]` tuples or tuple-compatible lists. -- Convert JSON `examples` objects into current `[(target, english), ...]` tuples or tuple-compatible lists. -- Preserve `answers` as the same dict shape used by `fluent_ai/agent.py:generate_quiz`. -- Attach richer optional fields to lesson banks as `pronunciation_hints`, `cultural_note`, and `romanization`, without requiring existing code to consume them immediately. -- Keep `topic_order` as the source for `topics_by_level(language)`. -- Keep conversation entries shaped for `fluent_ai/conversation.py:conversation_topics_for`. - -### Loader API - -Add `fluent_ai/curriculum.py` with: - -```python -def lesson_bank(language: str) -> dict[str, dict[str, Any]]: - ... - -def topics_by_level(language: str) -> dict[str, list[str]]: - ... - -def conversation_ladder(language: str) -> dict[str, list[dict[str, Any]]]: - ... - -def topic_lesson(language: str, level: str, topic: str) -> dict[str, Any]: - ... -``` - -The fallback chain should be: - -1. Exact language + exact topic. -2. Exact language + level `default_topic`. -3. Exact language + `generic`. -4. Spanish exact topic, for legacy topic-score compatibility when the user has Spanish-coded weak topics. -5. Existing `_generic_lesson_bank(language, topic)` until the generic literals are fully deleted. - -Spanish zero-behavior-change migration: - -- `TOPICS_BY_LEVEL` maps to `topics_by_level("Spanish")`. -- `LESSON_BANK` maps to `lesson_bank("Spanish")`. -- `TOPIC_LADDER` maps to `conversation_ladder("Spanish")`. -- `LANGUAGE_TOPIC_OVERRIDES["French"]` and `LANGUAGE_TOPIC_OVERRIDES["Hindi"]` map to `conversation_ladder("French")` and `conversation_ladder("Hindi")`. -- `_generic_lesson_bank` remains as the last fallback during migration, then becomes content in `generic`. - -The initial Spanish extraction should preserve current strings exactly, including current missing accents, so existing tests can prove zero behavior change. A later content-depth package should replace user-facing Spanish with correct accents and diacritics. - -### OpenAI Enhancement Interaction - -`fluent_ai/openai_provider.py:OpenAIProvider.enhance_lesson` currently receives topic, focus skill, duration, reason, weak topics, goals, and returns richer vocabulary, grammar, examples, and micro-task. - -With richer base curriculum: - -- The lesson passed to `enhance_lesson` should include `pronunciation_hints` and `cultural_note` when present. -- The prompt should ask the model to preserve those hints unless it can make them more level-appropriate. -- For Hindi, the prompt should include Devanagari plus `romanization` and ask for both to remain aligned. -- Better base lessons give the model stronger grounding and reduce generic rewrites for French and Hindi. - -Suggested prompt extension: - -```text -Base pronunciation hints: -- ... - -Base cultural note: -- ... - -For Hindi, preserve Devanagari and romanization alignment. -Return pronunciation_hints and cultural_note when useful. -``` - -The JSON response can later add `pronunciation_hints` and `cultural_note`, but the first loader implementation can simply pass these through from the base lesson if the provider does not return replacements. - -### Accent and Unicode Policy - -New content should use proper accents and diacritics: `café`, `mañana`, `¿Cómo?`, `j'étudie`, `बाज़ार`. - -Verified current behavior: - -- `fluent_ai/state.py:save_state` writes JSON with `ensure_ascii=False`, so progress and stored evidence can preserve non-ASCII text. -- `fluent_ai/conversation.py:normalize` strips combining marks through Unicode NFKD normalization, so conversation keyword matching is accent-insensitive. -- `fluent_ai/agent.py:normalize` lowercases and strips punctuation, but does not strip accents. Therefore quiz grading is not fully accent-insensitive today. - -Implementation should not claim the evaluator already normalizes accents everywhere. It should add an accent-normalization round-trip test and either update `fluent_ai/agent.py:normalize` to match `conversation.normalize` or keep quiz answer keys and acceptable answers explicitly accent-aware. - -## Part 2: Actual Content Plan - -Authoring should happen in JSON implementation files, not inline in this design doc. Content must be native-quality, accent-correct, and level-appropriate. - -Use stable topic slugs across languages wherever possible so weak-topic mapping, mistake-topic mapping, reviews, and progress history can transfer across Spanish, French, and Hindi. - -### Topic Slug Set - -Core A1-A2 slugs for parity: - -- `introductions` -- `cafe_orders` -- `daily_routines` -- `numbers_time` -- `directions_travel` -- `shopping` -- `weather` -- `family` -- `work_basics` -- `likes_food` -- `past_tense` -- `conjugations` -- `core_vocabulary` - -B1 slugs: - -- `opinions` -- `storytelling` -- `future_plans` -- `work_and_goals` -- `news_summaries` - -Legacy alias handling: - -- Map current `cafe orders` to `cafe_orders`. -- Map current `travel plans` to `directions_travel` or `future_plans` depending on level. -- Map current `vocabulary` to `core_vocabulary`. -- Map current `likes and food` to `likes_food`. -- Map current `workplace situations` to `work_and_goals`. -- Keep aliases readable in the loader so old `data/progress.json` topic scores still resolve. - -### Topic Tables - -Spanish should reach full A1-B1 depth first. French and Hindi should get full A1-A2 lesson parity with Spanish A1-A2, plus A1-B1 conversation ladders. - -| Language | Level | Topic slugs with focus_skill | -|---|---:|---| -| Spanish | A1 | `introductions`/vocabulary, `cafe_orders`/speaking, `likes_food`/vocabulary, `numbers_time`/vocabulary, `weather`/listening, `family`/vocabulary | -| Spanish | A2 | `daily_routines`/grammar, `conjugations`/conjugations, `past_tense`/conjugations, `directions_travel`/speaking, `shopping`/speaking, `work_basics`/vocabulary, `core_vocabulary`/vocabulary | -| Spanish | B1 | `opinions`/speaking, `storytelling`/grammar, `future_plans`/grammar, `work_and_goals`/speaking, `news_summaries`/reading | -| French | A1 | `introductions`/vocabulary, `cafe_orders`/speaking, `likes_food`/vocabulary, `numbers_time`/vocabulary, `weather`/listening, `family`/vocabulary | -| French | A2 | `daily_routines`/grammar, `conjugations`/conjugations, `past_tense`/conjugations, `directions_travel`/speaking, `shopping`/speaking, `work_basics`/vocabulary, `core_vocabulary`/vocabulary | -| French | B1 conversation | `opinions`/speaking, `storytelling`/speaking, `future_plans`/speaking | -| Hindi | A1 | `introductions`/vocabulary, `cafe_orders`/speaking, `likes_food`/vocabulary, `numbers_time`/vocabulary, `weather`/listening, `family`/vocabulary | -| Hindi | A2 | `daily_routines`/grammar, `conjugations`/conjugations, `past_tense`/conjugations, `directions_travel`/speaking, `shopping`/speaking, `work_basics`/vocabulary, `core_vocabulary`/vocabulary | -| Hindi | B1 conversation | `opinions`/speaking, `storytelling`/speaking, `future_plans`/speaking | - -French B1 and Hindi B1 can start as conversation-only entries in the first parity package, then receive full lesson banks after Spanish B1 and A1-A2 parity are stable. - -### Worked Example: Spanish B1 Topic - -```json -{ - "title": "Giving opinions with reasons", - "focus_skill": "speaking", - "vocabulary": [ - {"target": "creo que", "english": "I think that", "pronunciation_hint": "KREH-oh keh"}, - {"target": "desde mi punto de vista", "english": "from my point of view"}, - {"target": "por un lado", "english": "on one hand"}, - {"target": "sin embargo", "english": "however"}, - {"target": "vale la pena", "english": "it is worth it"} - ], - "grammar": "Use creo que plus an indicative verb for a clear opinion, then add porque plus one concrete reason.", - "examples": [ - {"target": "Creo que las conversaciones ayudan mucho porque son reales.", "english": "I think conversations help a lot because they are real."}, - {"target": "Desde mi punto de vista, vale la pena practicar todos los días.", "english": "From my point of view, it is worth practicing every day."}, - {"target": "Por un lado es difícil; sin embargo, aprendo más rápido.", "english": "On one hand it is difficult; however, I learn faster."} - ], - "answers": { - "mc": "I think that", - "fill_prompt": "___ las conversaciones ayudan mucho.", - "fill": "Creo que", - "open": "Creo que practicar todos los días ayuda mucho.", - "translation": "Desde mi punto de vista, vale la pena practicar." - }, - "pronunciation_hints": [ - "Keep que short: keh, not kway.", - "In rápido, stress the first syllable: RÁ-pi-do." - ], - "cultural_note": "In many Spanish-speaking contexts, softening an opinion with creo que can sound more conversational than a blunt statement.", - "conversation": [ - { - "topic": "opinions", - "complexity": "intermediate", - "opening": "Empecemos con una opinión. ¿Prefieres aprender con música, videos o conversaciones? ¿Por qué?", - "support": "Give one opinion and one reason with porque.", - "keywords": ["creo", "prefiero", "porque", "sin embargo", "vale"] - } - ] -} -``` - -### Worked Example: Hindi A1 Topic - -```json -{ - "title": "Greetings and introductions", - "focus_skill": "vocabulary", - "vocabulary": [ - {"target": "नमस्ते", "english": "hello", "romanization": "namaste", "pronunciation_hint": "nuh-muh-STAY"}, - {"target": "मेरा नाम", "english": "my name", "romanization": "mera naam"}, - {"target": "आपका नाम क्या है?", "english": "What is your name?", "romanization": "aapka naam kya hai?"}, - {"target": "मैं ... से हूँ", "english": "I am from ...", "romanization": "main ... se hoon"}, - {"target": "मुझे खुशी हुई", "english": "Nice to meet you", "romanization": "mujhe khushi hui"} - ], - "grammar": "Use मेरा नाम ... है for 'my name is ...'. Use आपका for a polite 'your'.", - "examples": [ - {"target": "नमस्ते, मेरा नाम आना है।", "english": "Hello, my name is Ana.", "romanization": "namaste, mera naam Ana hai."}, - {"target": "आपका नाम क्या है?", "english": "What is your name?", "romanization": "aapka naam kya hai?"}, - {"target": "मैं सिंगापुर से हूँ।", "english": "I am from Singapore.", "romanization": "main Singapore se hoon."} - ], - "answers": { - "mc": "hello", - "fill_prompt": "___, मेरा नाम आना है।", - "fill": "नमस्ते", - "open": "मेरा नाम आना है।", - "translation": "मैं सिंगापुर से हूँ।" - }, - "pronunciation_hints": [ - "हूँ is nasalized: hoon.", - "Keep long आ in नाम longer than a short अ." - ], - "cultural_note": "नमस्ते is widely understood and polite; with strangers, prefer आप over तुम.", - "conversation": [ - { - "topic": "introductions", - "complexity": "beginner", - "opening": "नमस्ते, मैं शुरू करता हूँ। आपका नाम क्या है?", - "support": "Model answer: मेरा नाम आना है।", - "keywords": ["मेरा", "नाम", "है", "नमस्ते"] - } - ] -} -``` - -### Quiz Answer-Key Requirements - -Every lesson topic must include: - -- `answers.mc`: English meaning for the first or most important vocabulary item. -- `answers.fill_prompt`: target-language prompt containing `___`. -- `answers.fill`: exact target-language fill answer. -- `answers.open`: one short target-language model answer. -- `answers.translation`: one target-language sentence for translation/reuse prompts. - -For `_lesson_driven_quiz` compatibility: - -- At least five vocabulary pairs. -- At least three examples. -- First vocabulary pair must be a stable, beginner-readable pair. -- First example must be a natural sentence with an English meaning. -- Open-ended answers should include clear keywords, especially in Hindi where tokenization may be less forgiving. - -## Part 3: Sequenced Implementation Plan - -### Work Package A: Loader and Spanish Extraction - -Goal: move current Spanish literals into data files with zero behavior change. - -- Changes: add `fluent_ai/curriculum.py`; add `fluent_ai/curriculum/spanish.json` from current `TOPICS_BY_LEVEL`, `LESSON_BANK`, and `TOPIC_LADDER`; wire `fluent_ai/agent.py:TOPICS_BY_LEVEL`, `LESSON_BANK`, and `fluent_ai/conversation.py:TOPIC_LADDER` through loader outputs; leave `_generic_lesson_bank` as last fallback. -- Tests: Spanish schema validation; snapshot/equality test against commit `5a43138`; every Spanish topic loadable by level; `generate_quiz` works for every extracted topic; existing tests still pass. -- Acceptance: `npm run check` green, with manual diff confirming no user-facing Spanish behavior changed. - -### Work Package B: Spanish A2/B1 Depth and French A1-A2 - -Goal: add real depth without changing Lesson Mode orchestration. - -- Changes: add accent-correct Spanish A1 replacements after zero-diff tests; add Spanish A2/B1 banks; add `fluent_ai/curriculum/french.json` with A1-A2 lesson parity and A1-B1 conversation ladders; extend aliases; update `OpenAIProvider.enhance_lesson` prompt with pronunciation hints and cultural note. -- Tests: Spanish/French schema validation; every declared topic loadable; quiz generatable for every topic/level/language; French A1-B1 ladder loadable; accent normalization round-trip for Spanish and French. -- Acceptance: `npm run check` green; demo rotates Spanish B1 opinions/storytelling/future plans; French A1-A2 no longer falls back to one generic lesson for every topic. - -### Work Package C: Hindi A1-A2 and Script Support - -Goal: make Hindi first-class for beginner lessons while preserving Devanagari. - -- Changes: add `fluent_ai/curriculum/hindi.json` with Devanagari, romanization, pronunciation hints, and A1-A2 parity; add Hindi A1-B1 conversation ladders; preserve `romanization` metadata while returning tuple-compatible lesson fields; file a sequenced renderer follow-up for an optional transliteration toggle. -- Tests: Hindi schema validation requiring `romanization`; every Hindi A1-A2 topic loadable; quiz generatable for every Hindi topic; Devanagari keyword matching works; diacritic test covers Devanagari preservation, not ASCII-only conversion. -- Acceptance: `npm run check` green; Hindi no longer uses the same generic bank for every topic; demo can expose Devanagari plus romanization data to the renderer. - -## Stopping Conditions - -Stop implementation if: - -- Loader migration changes Spanish behavior before a zero-diff test exists. -- JSON content cannot generate quizzes for every declared topic. -- Hindi romanization is missing or mismatched with Devanagari examples. -- A package would require editing concurrent Python changes without rereading the latest files. - -The correct path is additive curriculum data first, runtime wiring second, UI rendering last. +`tests/test_curriculum.py` covers asset/loader behavior. Also run the full suite and mocked smoke, and verify installed-package loading when changing package data. diff --git a/docs/FLUENTAI_LIMITLESS_IDEAL.md b/docs/FLUENTAI_LIMITLESS_IDEAL.md index b3f3884..3d2016e 100644 --- a/docs/FLUENTAI_LIMITLESS_IDEAL.md +++ b/docs/FLUENTAI_LIMITLESS_IDEAL.md @@ -1,1387 +1,34 @@ -# FluentAI Limitless Ideal +# FluentAI product direction and limitations -## Purpose Of This Document +FluentAI connects adaptive lessons and tutor conversations through inspectable local learner memory. It is a language-learning demo, with OpenAI required for real lessons, conversations, voice, vision, and phrase audio. Mocked providers support repeatable tests. -This document describes the ideal version of FluentAI: the version that would exist if there were no practical bounds on model capability, engineering time, infrastructure, budget, device access, curriculum design, speech quality, personalization depth, or product polish. +## Current experience -It is not a narrow MVP spec. It is the product north star. +- Onboarding records goals, preferences, and a self-reported level; optional placement seeds the first practice plan. +- Home recommends practice from review due dates, mistakes, weak topics, and conversation goals. +- Lessons explain their selection, generate quizzes, grade answers, and update XP, skill evidence, review scheduling, and conversation goals. +- Text and Realtime conversations use the selected language and feed corrections back into lessons. Voice supports English help, adaptive pauses, and optional camera context. +- Desktop and browser share a renderer and Python bridge. Memory inspection, export, reset, and session recovery are implemented. -The current FluentAI already has the right spine: +## Product decisions -- Lesson Mode reads persistent learner memory, generates lessons and quizzes, evaluates answers, updates progress, and adapts future practice. -- Conversation Mode behaves like a live tutor session, initiates the conversation, adapts to level, supports voice/text/video context, and updates the same learner memory. -- Learner state is stored locally under `data/`. -- Real app usage is API-key-first, with mocked OpenAI providers used for tests. -- Agent decisions are visible in logs so a judge can see memory loading, topic choice, video-context handling, feedback, and adaptation. +Memory should explain why the tutor chose an activity. Lesson and conversation progress must reinforce each other. Keep generated teaching content separate from deterministic progress accounting, and make model failures visible. Use camera context only when enabled; express uncertainty rather than inventing objects. -The limitless ideal keeps that spine and expands it into a complete adaptive language-learning operating system. +## Known limitations -## One Sentence Essence +- CEFR labels and skill scores are app estimates, not validated proficiency assessments. Pronunciation practice notes and phrase playback are not a phonetic scoring system. +- Curriculum depth is uneven across levels and languages; advanced topic availability does not establish full course coverage. +- Local learner storage does not mean offline inference. Relevant prompts, learner text, voice audio, and enabled camera frames go to OpenAI. See [privacy boundaries](ONBOARDING_HOME_DESIGN.md). +- Automated tests use mocked providers and simulated renderer events. They do not establish microphone/camera permission behavior, natural live turn-taking, or successful launch on a clean macOS account. +- The packaged app uses local/ad-hoc signing and is not a notarized public release. See [packaging](PACKAGING_DESIGN.md). -FluentAI is a living AI language tutor that knows the learner deeply, starts the next most useful interaction, listens and watches like a human tutor, teaches through lessons and conversation, remembers every meaningful signal, and continuously adapts until the learner can confidently use the language in real life. +## Remaining roadmap -## The True Overarching Goal +1. Add browser-level voice regressions with mocked Realtime events, including interruption, reconnect, and finalization. +2. Add camera-frame fixtures and complete physical camera/window-lifecycle acceptance. +3. Deepen and review Hindi/French lessons and advanced content, preserving native script, romanization, and answer-key quality. +4. Complete clean-account packaged-app acceptance; add Developer ID signing and notarization before public distribution. -The goal of FluentAI is not to be a flashcard app, a quiz app, a chatbot, a classroom replacement, a video-call gimmick, or a generic AI wrapper. +Spaced repetition, state-v2 migration, home/privacy controls, and durable session completion are implemented; they are not pending roadmap items. -The true goal is to create the feeling and effectiveness of having a personal language tutor who is always available, always prepared, always aware of the learner's level, and always improving the learner's next step. - -The app should feel like this: - -- It knows what the learner already knows. -- It knows what the learner keeps missing. -- It knows when the learner is nervous, stuck, bored, rushing, guessing, or ready for harder material. -- It starts lessons without requiring the learner to plan them. -- It starts conversations instead of waiting passively. -- It corrects gently, specifically, and at the right moment. -- It turns real objects, real surroundings, real goals, and real interests into language practice. -- It remembers the outcome of every lesson and conversation. -- It uses that memory to make the next interaction sharper. - -The essence is memory plus agency plus real interaction. - -## Product Identity - -FluentAI should be understood as an adaptive tutor studio. - -It has two first-class modes: - -- Lesson Mode: structured learning, explanation, quiz, evaluation, spaced review, and deliberate practice. -- Conversation Mode: live spoken interaction, target-language immersion, tutor-led steering, video-context awareness, and speaking-confidence growth. - -Those modes should not feel like separate products. They should feel like two ways the same tutor helps the same learner. - -Lesson Mode builds the learner's foundation. - -Conversation Mode makes the learner use it. - -Memory connects both. - -## Non-Negotiable Product Principles - -### 1. The Tutor Must Be Agentic - -FluentAI should not wait for the learner to design their own course. - -The tutor should decide: - -- what topic to teach next -- what skill needs attention -- whether to review or introduce new material -- whether the learner needs easier examples -- whether the learner is ready for harder practice -- when to move from lesson to quiz -- when to move from quiz to conversation -- when to slow down -- when to stop correcting and let fluency flow -- when to explain in English -- when to return to the target language -- when to schedule review -- when to celebrate a real breakthrough - -The learner should still be able to choose goals, language, difficulty preference, voice, schedule, and interests. But the app's core promise is that the tutor carries the cognitive load of planning. - -### 2. Memory Is The Product - -The memory file is not just storage. It is the tutor's understanding of the learner. - -In the ideal version, every meaningful user interaction updates a learner model: - -- level -- active target language -- known words -- weak words -- known grammar patterns -- weak grammar patterns -- pronunciation profile -- listening comprehension -- reading comprehension -- writing accuracy -- speaking fluency -- speaking confidence -- topic preferences -- avoided topics -- recurring mistakes -- correction history -- review schedule -- interests -- real-life goals -- travel or work context -- preferred tone -- energy patterns -- lesson completion behavior -- conversation anxiety signals -- best time of day for practice -- retention curve -- fluency milestones - -The user should be able to inspect this memory. Nothing important should be hidden in a black box. - -### 3. Lessons And Conversations Must Feed Each Other - -The ideal FluentAI loop is: - -1. The tutor reads memory. -2. Lesson Mode teaches the next useful concept. -3. The learner practices through a quiz. -4. The evaluator identifies what changed. -5. Conversation Mode creates a live scenario where the learner must use the new concept. -6. The tutor evaluates real usage, not just quiz answers. -7. Memory updates. -8. Spaced review schedules future reinforcement. -9. The next lesson adapts. - -Example: - -- The learner misses present-tense conjugation in Lesson Mode. -- Conversation Mode later asks simple daily-routine questions. -- The learner says "yo estudiar" instead of "yo estudio." -- The tutor gives a tiny correction without killing the conversation. -- Memory records that conjugation is still weak under speaking pressure. -- The next lesson includes short, high-repetition present-tense practice. - -That is the core magic. - -### 4. The Tutor Must Feel Human, But Stay Useful - -FluentAI should feel like a real tutor session, especially in Conversation Mode. - -That means: - -- the tutor initiates -- the tutor waits for the learner to finish -- the tutor does not interrupt thinking pauses -- the tutor gives short prompts instead of monologues -- the tutor notices confusion -- the tutor explains when asked -- the tutor adapts to the learner's language choice -- the tutor corrects selectively -- the tutor asks follow-up questions -- the tutor remembers previous conversations -- the tutor has a warm, steady personality - -It should not overperform. It should not flood the learner with explanations. It should not turn every sentence into a grammar lecture. - -The tutor's job is to create progress, not show off intelligence. - -### 5. The App Must Be Demoable And Honest - -The app should always be able to prove what it is doing. - -A judge or user should be able to see: - -- memory was loaded -- level was judged -- topic was selected for a reason -- lesson was generated -- quiz was generated -- answer was evaluated -- progress changed -- spaced review was scheduled -- conversation topic was selected -- video context was analyzed -- speaking turn was scored -- next speaking goal changed -- state was saved - -The visible agent decision log is essential. It should be elegant, collapsible, and readable, but it should exist because transparency is part of the product. - -## Ideal User Experience - -### First Launch - -The first launch should feel like meeting a tutor, not configuring software. - -The app asks only what it needs: - -- name or nickname -- target language -- why the learner wants the language -- current self-estimated level, if known -- comfort with speaking -- preferred session length -- whether voice/video can be used -- whether local memory should stay private on the device - -Then the tutor runs a short placement flow: - -- a few comprehension questions -- one tiny writing prompt -- one short speaking prompt -- optional pronunciation sample -- optional visual prompt if camera is enabled - -The output is a clear learner profile: - -- judged level -- strongest skill -- weakest skill -- first practice goal -- first conversation goal -- recommended daily plan - -### Home Screen - -The home screen should not be a marketing landing page. It should be the actual tutor workspace. - -Ideal home screen regions: - -- current learner profile -- target language selector -- today's recommended action -- Lesson Mode launch -- Conversation Mode launch -- review queue -- recent progress -- speaking confidence -- agent decision log -- memory inspector - -The main action should be obvious: - -- "Start today's lesson" -- "Start live tutor call" -- "Review due phrases" -- "Practice yesterday's weak topic" - -The UI should be calm, polished, and practical. It should feel like a premium learning tool, not a toy. - -### Daily Session - -An ideal daily session could look like this: - -1. The app opens with a short tutor message: "You missed present tense yesterday, so today we will practice simple daily routines." -2. Lesson Mode teaches five useful phrases. -3. The learner answers six adaptive questions. -4. The evaluator gives immediate feedback. -5. The app says: "Now let's use this in a real conversation." -6. Conversation Mode starts a voice call. -7. The tutor asks simple daily routine questions. -8. The learner struggles with one verb. -9. The tutor gives a short correction and keeps going. -10. The session ends with a summary and next goal. -11. Memory updates automatically. - -The learner should feel that the app knew what it was doing the whole time. - -## Lesson Mode Ideal - -Lesson Mode should be a complete adaptive teaching engine. - -### Lesson Inputs - -The lesson generator should consider: - -- target language -- judged level -- current skill scores -- weak topics -- weak grammar patterns -- weak vocabulary -- due reviews -- recent topics -- conversation mistakes -- user goals -- interests -- time available -- preferred tone -- confidence level -- prior lesson outcomes -- prior conversation outcomes -- retention history - -### Lesson Structure - -An ideal lesson contains: - -- title -- reason this lesson was chosen -- target level -- target skill -- topic -- estimated duration -- five to ten useful phrases -- one core grammar idea -- three to five examples -- pronunciation notes -- listening examples -- cultural note when relevant -- micro-task -- quiz -- speaking bridge into Conversation Mode - -The reason matters. The learner should understand why they are seeing this lesson: - -"You missed present-tense daily routine answers in yesterday's conversation, so this lesson practices short sentences like 'I wake up,' 'I work,' and 'I study.'" - -### Explanation Style - -Explanations should be: - -- short -- level-appropriate -- practical -- example-first -- not overly academic unless the learner is advanced - -For A1: - -- one rule -- one pattern -- two examples -- immediate practice - -For B2 and above: - -- nuance -- register -- exceptions -- comparison with native usage -- argumentation patterns - -### Quiz Behavior - -The quiz should adapt question by question. - -Question types: - -- multiple choice -- fill in the blank -- translation -- listening comprehension -- pronunciation repeat -- sentence ordering -- open-ended writing -- short spoken answer -- visual naming -- role-play response -- error correction - -The quiz should not only mark correct/incorrect. It should identify why: - -- vocabulary missing -- wrong conjugation -- wrong gender -- wrong tense -- word order issue -- pronunciation issue -- comprehension issue -- answer too short -- answer understandable but unnatural - -### Feedback - -Feedback should be immediate and specific: - -- "Correct." -- "Almost. You used the infinitive. For 'I speak,' use 'hablo.'" -- "Understandable, but a native speaker would say..." -- "Good answer. Now add one detail." -- "You got the meaning, but missed the accent/pronunciation." - -The learner should never be left wondering what to do next. - -### Adaptation - -After each lesson: - -- XP updates -- skill scores update -- topic mastery updates -- weak topics recalculate -- review queue updates -- recent topics update -- history records outcome -- next lesson recommendation changes -- conversation goal changes if relevant - -The ideal system should distinguish between: - -- remembered in recognition -- remembered in recall -- used correctly in writing -- used correctly in speaking -- used correctly under real-time pressure - -This matters because a learner might pass a flashcard but fail to use the phrase in conversation. - -## Conversation Mode Ideal - -Conversation Mode is the heart of FluentAI's emotional experience. - -It should feel like a FaceTime-style tutor call. - -### Core Behavior - -The tutor should: - -- initiate the conversation -- know the learner's current level -- know the learner's weak topics -- choose the right topic -- speak in the target language by default -- use English only when helpful -- adapt to learner replies -- detect confusion -- wait through pauses -- avoid interrupting -- keep prompts short -- correct gently -- ask follow-ups -- use visible objects when video is on -- update progress after the session - -### Beginner Conversation - -For A1/A2 learners, conversation should be simple, concrete, and confidence-building. - -Topics: - -- name -- country -- weather -- food -- likes -- family -- daily routine -- simple objects -- colors -- numbers -- greetings - -Tutor behavior: - -- one short sentence at a time -- one question at a time -- model answers available -- English rescue when needed -- lots of repetition -- slow pacing -- positive reinforcement - -Example: - -Tutor: "Hola. Me llamo Ana. ¿Como te llamas?" - -Learner: "Me llamo Johan." - -Tutor: "Mucho gusto, Johan. ¿Te gusta el cafe?" - -If the learner says "What does that mean?", the tutor should answer directly: - -"It means 'Do you like coffee?' You can say: 'Si, me gusta el cafe.'" - -Then continue gently. - -### Advanced Conversation - -For B2/C1/C2 learners, conversation should become rich and intellectually alive. - -Topics: - -- politics -- environment -- culture -- work -- ethics -- debate -- current affairs -- personal goals -- persuasion -- humor -- literature -- professional communication -- negotiation - -Tutor behavior: - -- ask for reasons -- challenge assumptions -- request examples -- introduce nuance -- correct register -- push for idiomatic phrasing -- simulate real scenarios -- ask the learner to defend a position - -Example: - -"Defend a nuanced position: how should a city balance economic growth with environmental protection?" - -### Turn-Taking - -Natural turn-taking is critical. - -The tutor should: - -- wait for real silence before responding -- treat filler words as part of the learner's turn -- allow self-correction -- avoid talking over the learner -- give a short check-in after long silence -- avoid long monologues -- adapt pause duration by level and confidence - -Beginners need more wait time. - -Confident advanced learners can handle quicker rhythm. - -### Correction Policy - -The tutor should not correct everything. - -It should prioritize: - -- errors that block meaning -- errors related to the lesson target -- recurring mistakes -- level-appropriate corrections -- pronunciation issues that affect comprehension - -It should avoid: - -- interrupting every sentence -- overcorrecting beginners -- giving long grammar explanations in the middle of a conversation -- correcting advanced learners only on trivial issues - -Correction modes: - -- recast: learner says it wrong, tutor repeats naturally -- explicit correction: "Use 'hablo,' not 'hablar,' for 'I speak.'" -- model phrase: "Try: Me gusta el cafe." -- delayed feedback: notes after the conversation -- pronunciation replay: compare model and learner audio - -### Conversation Memory - -Every conversation should update: - -- sessions completed -- total turns -- fluency score -- speaking confidence -- recent conversation topics -- missed phrases -- last video context -- next speaking goal -- pronunciation targets -- hesitation patterns -- English-help frequency -- correction acceptance - -The ideal summary after a call: - -- what the learner did well -- one correction to remember -- one phrase to review -- one next speaking goal -- whether confidence improved -- what the next conversation should practice - -## Video Context Ideal - -Video is not a gimmick. It is how the tutor enters the learner's world. - -When video is on, the tutor should be able to use visible context: - -- objects -- room setting -- food -- weather through a window -- written text -- gestures -- flashcards -- a notebook -- travel items -- clothing colors -- real-life scenes - -For a beginner: - -- "I see an apple. In Spanish, apple is manzana. Say: Es una manzana." - -For an intermediate learner: - -- "Describe where the apple is and what you might do with it." - -For an advanced learner: - -- "Use the object as the opening image for a short persuasive argument about food waste." - -### Video Safety And Accuracy - -The vision system should: - -- avoid guessing when uncertain -- say when the scene is unclear -- avoid identifying sensitive personal attributes -- detect synthetic test feeds -- avoid storing raw images unless explicitly allowed -- store only useful learning summaries by default - -The current product direction already values uncertainty guardrails. The ideal version should make them first-class. - -## Voice Ideal - -Voice should feel like a real tutor, not text-to-speech bolted onto a chat app. - -The ideal voice system should have: - -- low latency -- natural prosody -- target-language pronunciation -- level-appropriate speed -- interruption control -- silence detection -- emotion-aware pacing -- pronunciation feedback -- replay mode -- side-by-side model and learner audio -- accent-tolerant recognition -- confidence scoring - -For language learning, the voice model must not merely transcribe. It must teach through sound. - -### Pronunciation Feedback - -The ideal pronunciation engine should identify: - -- missing sounds -- stress errors -- vowel issues -- rhythm issues -- intonation issues -- syllable timing -- likely L1 interference - -Feedback should be kind and practical: - -- "Your meaning was clear." -- "Try making the final vowel shorter." -- "Listen: hablo. Now you try." -- "Good improvement. One more time, slightly slower." - -## Learner Memory Model - -The ideal learner state should be structured, inspectable, portable, and privacy-aware. - -### Core Profile - -Fields: - -- learner id -- display name -- target language -- native language -- current CEFR level -- level confidence -- active goals -- learning motivation -- preferred session length -- preferred correction style -- preferred tutor tone -- accessibility preferences -- privacy settings - -### Skill Scores - -Skills: - -- vocabulary -- grammar -- conjugation -- listening -- speaking -- pronunciation -- reading -- writing -- translation -- conversation management -- cultural understanding -- register -- fluency -- comprehension under speed - -Each skill should have: - -- current score -- confidence interval -- trend -- last practiced date -- evidence examples -- recommended next action - -### Topic Mastery - -Topic mastery should track: - -- introductions -- weather -- food -- travel -- work -- family -- daily routines -- shopping -- health -- culture -- politics -- environment -- debate -- professional communication -- learner-created custom topics - -Each topic should track: - -- recognition -- recall -- spoken use -- written use -- listening comprehension -- review interval -- last success -- last failure - -### Mistake Memory - -Mistakes should not be stored as shame. They should be stored as teaching signals. - -Each recurring mistake could include: - -- incorrect form -- corrected form -- context -- skill -- topic -- first seen -- last seen -- frequency -- severity -- whether meaning was blocked -- whether it recurred during speech -- next review date - -### Spaced Review - -The ideal review system should be more nuanced than a single queue. - -It should schedule: - -- words -- phrases -- grammar patterns -- pronunciation targets -- conversation moves -- listening clips -- visual object vocabulary -- personal sentences the learner has used - -Intervals should adapt to: - -- quiz performance -- conversation performance -- confidence -- hesitation -- time since last exposure -- number of correct recalls -- modality of success - -Speaking recall should weigh more than recognition. - -## Agent Architecture - -The ideal FluentAI should be a coordinated agent system. - -### Memory Agent - -Responsibilities: - -- load learner state -- summarize current profile -- expose relevant history -- protect privacy -- decide what memory is relevant now -- write safe updates after sessions -- prevent noisy memory bloat - -### Curriculum Agent - -Responsibilities: - -- choose what to teach next -- balance review and new material -- map CEFR level to content -- align lessons with learner goals -- keep progression coherent - -### Lesson Generator Agent - -Responsibilities: - -- generate level-appropriate explanations -- create examples -- create vocabulary lists -- create grammar focus -- create micro-tasks -- keep content practical - -### Quiz Agent - -Responsibilities: - -- generate mixed question types -- adapt difficulty -- validate answer options -- avoid ambiguous questions -- test recall, not just recognition - -### Evaluator Agent - -Responsibilities: - -- grade answers -- identify error category -- update skill evidence -- produce clear feedback -- distinguish correct, partially correct, and understandable-but-unnatural - -### Conversation Orchestrator - -Responsibilities: - -- start the live session -- select topic -- manage turn-taking -- decide whether to correct now or later -- keep the conversation moving -- end with a useful summary - -### Voice Agent - -Responsibilities: - -- manage realtime audio -- tune turn detection -- prevent interruption -- adapt speed -- handle silence -- provide pronunciation feedback - -### Vision Context Agent - -Responsibilities: - -- analyze camera frames -- extract safe learning context -- detect uncertainty -- avoid sensitive inferences -- turn objects/scenes into prompts - -### Progress Reporter Agent - -Responsibilities: - -- summarize progress -- explain what changed -- show next action -- keep motivation grounded in evidence - -### Demo Orchestrator - -Responsibilities: - -- run a visible 5-10 minute demo -- show agent decisions -- show state changes -- prove both modes work -- keep flow smooth for judges - -## Data Model Ideal - -The ideal state should remain local-first by default, but support optional sync. - -High-level structure: - -```json -{ - "learner": {}, - "languages": {}, - "active_language": "Spanish", - "skills": {}, - "topic_mastery": {}, - "vocabulary_memory": {}, - "grammar_memory": {}, - "pronunciation_memory": {}, - "conversation_memory": {}, - "review_queue": {}, - "history": [], - "preferences": {}, - "privacy": {}, - "updated_at": "..." -} -``` - -### Event History - -Every meaningful learning event should be stored as an event: - -- lesson started -- lesson completed -- quiz answered -- answer evaluated -- review scheduled -- conversation started -- tutor prompt created -- learner replied -- pronunciation evaluated -- video context used -- progress updated - -Events make the system auditable and debuggable. - -### Derived State - -The app should separate raw events from derived profile values. - -Raw event: - -- learner answered "yo hablar espanol" - -Derived updates: - -- conjugation weakness increased -- present tense review scheduled -- speaking confidence slightly decreased or unchanged -- next lesson should revisit first-person present tense - -This separation keeps the system honest. - -## Ideal UI - -The ideal UI should feel sophisticated, dense enough for real work, and calm enough for learning. - -### Visual Direction - -FluentAI should feel like: - -- a premium tutor studio -- a live language-learning cockpit -- warm but not childish -- modern but not noisy -- demoable but not fake - -It should avoid: - -- giant empty space -- childish gamification -- unreadable generated-looking colors -- floating panels that overlap core work -- model-name clutter -- controls fighting with lesson content - -### Lesson UI - -Ideal Lesson Mode layout: - -- main lesson panel -- sticky learning rail -- vocabulary cards -- grammar explanation -- example sentences -- phrase lab -- quiz area -- instant feedback -- progress summary -- review schedule - -The learner should always know: - -- what they are learning -- why it was chosen -- what to do next -- whether they got it right -- how it changed their progress - -### Conversation UI - -Ideal Conversation Mode layout: - -- large call stage -- tutor video/avatar/audio presence -- learner camera preview when enabled -- call controls that never overlap context -- live transcript -- text fallback -- visual context card -- pronunciation feedback -- session goals -- post-call summary - -The voice/video call should be the main stage, not a side widget. - -### Agent Decision Log - -The agent log should: - -- be collapsible -- stay readable -- avoid blocking lesson/call content -- show agent names -- show concrete decisions -- avoid leaking secrets -- be useful in a judging demo - -Example logs: - -- `[Memory Agent] Loaded level A1; weak topics: conjugations, daily routines.` -- `[Lesson Generator Agent] Selected present tense because it is due for review.` -- `[Adaptive Quiz Agent] Prepared 6 mixed questions.` -- `[Vision Context Agent] Saw: apple on desk, confidence high.` -- `[Speaking Tutor Agent] Slowed pacing because learner confidence is low.` -- `[Memory Agent] Saved progress to data/progress.json.` - -## Multilingual Ideal - -Current FluentAI supports Spanish, French, and Hindi in important parts of the app. The ideal version supports many languages while keeping quality high. - -Language support should include: - -- Spanish -- French -- Hindi -- Mandarin -- Japanese -- Korean -- Arabic -- German -- Italian -- Portuguese -- English as a target language -- learner-added languages where model quality is sufficient - -But quality matters more than count. - -Each language should have: - -- CEFR or equivalent level mapping -- beginner topic ladders -- grammar progression -- pronunciation guidance -- cultural context -- script support -- romanization when useful -- native examples -- conversation scenarios - -For Hindi and other non-Latin scripts, the app should support: - -- native script -- optional transliteration -- pronunciation hints -- typing assistance -- script-learning mini-lessons - -## Personalization Ideal - -The tutor should adapt around the learner's real life. - -Possible learner goals: - -- travel -- work meetings -- interviews -- dating -- family communication -- school -- immigration -- reading books -- watching shows -- debating -- pronunciation improvement -- heritage-language reconnection - -The app should ask for goals and then make them concrete. - -If the learner wants Spanish for travel: - -- airport -- hotel -- directions -- food -- emergency phrases -- small talk - -If the learner wants French for work: - -- introductions -- meetings -- email phrasing -- polite disagreement -- presentations -- negotiation - -If the learner wants Hindi for family: - -- greetings -- food -- family roles -- affection -- basic questions -- listening practice with natural speech - -## Assessment Ideal - -The ideal assessment model should be continuous. - -It should not rely only on one placement test. - -It should infer level from: - -- quiz accuracy -- speed of recall -- speaking length -- hesitation -- grammar accuracy -- vocabulary range -- listening comprehension -- pronunciation clarity -- ability to recover from confusion -- ability to ask for clarification -- ability to sustain topic -- ability to use new material later - -The system should maintain: - -- current level -- level confidence -- skill-specific levels -- evidence for level changes - -A learner might be: - -- A2 in reading -- A1 in speaking -- B1 in listening -- A2 in grammar - -The app should adapt by skill, not flatten the learner into one number. - -## Demo Ideal - -A perfect judging demo should run for 5-10 minutes and show the full loop. - -### Demo Script - -1. Open FluentAI. -2. Show learner memory. -3. Show current level and weak topics. -4. Start Lesson Mode. -5. Agent log shows why the lesson was chosen. -6. Lesson teaches a short, personalized topic. -7. Learner answers quiz. -8. Evaluator grades answers. -9. Progress updates visibly. -10. Start Conversation Mode. -11. Tutor initiates. -12. Learner replies. -13. Tutor adapts. -14. Turn on video. -15. Tutor uses visible context. -16. End session. -17. Show updated `data/progress.json`. -18. Show next lesson recommendation changed. - -### Demo Success Criteria - -The demo succeeds if a judge can say: - -- this is agentic -- the tutor has memory -- the app adapts -- the two modes are connected -- the AI initiates and steers -- video context matters -- progress is saved -- the app could become a real product - -## Privacy Ideal - -Language learning can be personal. The app may hear voice, see camera context, and store learning weaknesses. - -Ideal privacy principles: - -- local-first memory by default -- no secret values in logs -- clear camera/mic permission prompts -- visible indicator when video/audio is active -- raw audio/video not stored unless explicitly enabled -- learning summaries stored instead of raw sensitive data -- user can inspect memory -- user can delete memory -- user can export memory -- user can reset a language profile -- optional encrypted cloud sync - -The learner should trust the tutor. - -## Reliability Ideal - -The ideal app should be robust in real usage. - -It should handle: - -- missing API key -- expired realtime session -- microphone blocked -- camera blocked -- weak network -- model timeout -- malformed model output -- empty tutor response -- invalid progress file -- interrupted lesson -- interrupted call -- app restart mid-session - -The app should always recover with a clear message and preserve state. - -## Testing Ideal - -The ideal test suite should cover: - -- state migration -- lesson generation -- quiz generation -- answer evaluation -- progress updates -- spaced review scheduling -- conversation topic selection -- conversation memory updates -- English-help behavior -- non-Spanish language behavior -- realtime turn detection -- camera-frame analysis -- web endpoints -- desktop bridge -- renderer states -- call controls -- text fallback -- no secret leakage -- demo smoke flow - -There should be mocked providers for CI and real-provider smoke checks for local/demo readiness. - -## Engineering Ideal - -The codebase should preserve the current good instincts: - -- simple runnable slices -- local state under `data/` -- inspectable agent logic -- API-key-first real usage -- mocked providers in tests -- visible logs -- demoable commands - -As the product grows, ideal architecture should include: - -- typed state schemas -- event-sourced learning history -- explicit memory migrations -- provider abstraction -- realtime session manager -- vision context manager -- curriculum engine -- evaluation engine -- review scheduler -- UI state store -- test fixtures for lessons/conversations/vision -- local-first persistence with optional sync - -## Ideal Roadmap - -### Phase 1: Perfect The Core Loop - -- lesson reason shown clearly -- quiz feedback instant and specific -- conversation uses lesson target immediately -- state updates visible -- review schedule reliable - -### Phase 2: Make Voice Feel Real - -- lower latency -- better pause handling -- better silence prompts -- pronunciation feedback -- replay model phrases -- speech confidence scoring - -### Phase 3: Make Video Truly Useful - -- recurring safe frame analysis -- object-to-lesson bridge -- visual conversation prompts -- OCR for learner notes -- uncertainty display - -### Phase 4: Deep Memory - -- mistake memory -- phrase memory -- pronunciation memory -- skill-specific levels -- evidence-backed progress - -### Phase 5: Curriculum Depth - -- richer lessons for Spanish, French, Hindi -- CEFR-aligned progression -- custom goals -- real-life scenarios -- advanced debate and work modules - -### Phase 6: Product Polish - -- premium UI -- seamless desktop and browser parity -- graceful error handling -- onboarding -- export/reset memory -- demo mode - -## What FluentAI Should Never Become - -FluentAI should not become: - -- a generic chatbot with a language prompt -- a flashcard clone -- a passive content library -- a grammar textbook with AI decoration -- a gamified streak machine with shallow learning -- a voice demo that does not update memory -- a video demo that guesses objects unreliably -- a black box that claims adaptation without showing evidence - -If a feature does not improve memory, adaptation, teaching quality, conversation quality, or learner confidence, it should be questioned. - -## The Final Ideal - -In its perfect form, FluentAI is the tutor that knows exactly what to do next. - -The learner opens the app and does not need to ask, "What should I study?" - -The tutor already knows. - -It knows yesterday's mistakes. It knows today's review. It knows whether the learner needs encouragement or challenge. It can teach through a structured lesson, quiz the learner, speak naturally in a live call, use the learner's real surroundings, explain in English when needed, return to the target language at the right time, and save every meaningful result. - -The perfect FluentAI feels personal because it remembers. - -It feels alive because it initiates. - -It feels useful because it adapts. - -It feels trustworthy because it shows its work. - -That is the essence: a living, memory-driven, agentic language tutor that turns every lesson and every conversation into the next best step toward real fluency. +See [architecture and memory decisions](GAP_AND_DESIGN.md), [curriculum](CURRICULUM_DESIGN.md), and [historical evidence](../notes.md). diff --git a/docs/GAP_AND_DESIGN.md b/docs/GAP_AND_DESIGN.md index 82cf763..7f16a03 100644 --- a/docs/GAP_AND_DESIGN.md +++ b/docs/GAP_AND_DESIGN.md @@ -1,594 +1,52 @@ -# FluentAI Gap Analysis and Implementation Design - -Scope: compare `docs/FLUENTAI_LIMITLESS_IDEAL.md` with the current implementation and design the next two workstreams: Memory Model v2 (WS1) and Core Adaptive Loop (WS2). Current-code citations use `path:function` or `path:function-name`. - -## Part 1: Gap Inventory - -| Capability | Status | Current behavior and grounding | -|---|---:|---| -| Memory model depth | PARTIAL | Flat single learner dict with scores, topics, review queue, history, and conversation memory in `fluent_ai/state.py:default_state`, migrated by `fluent_ai/state.py:migrate_state`. No skill evidence, mistake objects, privacy metadata, or event-sourced raw signals. | -| Per-language state | PARTIAL | Language switch mutates `learner.target_language` in `fluent_ai/desktop_bridge.py:_load`; lessons use generic content for non-Spanish in `fluent_ai/agent.py:_generic_lesson_bank`; no separate memory per language. | -| Event history | PARTIAL | `state["history"]` stores last 25 lesson/conversation summary dicts in `fluent_ai/agent.py:update_progress`, `fluent_ai/conversation.py:update_conversation_progress`, and `fluent_ai/desktop_bridge.py:apply_conversation_turn_progress`; not typed, not auditable event records. | -| Mistake memory | PARTIAL | Conversation corrections are appended as strings to `conversation_memory.missed_phrases` in `fluent_ai/conversation.py:update_conversation_progress` and `fluent_ai/desktop_bridge.py:apply_conversation_turn_progress`; quiz misses only reorder `weak_topics` in `fluent_ai/agent.py:update_progress`. | -| Spaced review | PARTIAL | Topic-level review scheduling exists in `fluent_ai/agent.py:update_review_schedule` and due-topic selection in `fluent_ai/agent.py:due_review_items` / `next_due_review_topic`; no item-level words, phrases, mistakes, modality, or confidence adjustment. | -| Lesson reason | MISSING | `fluent_ai/agent.py:choose_topic` selects a topic but returns no reason; `fluent_ai/agent.py:generate_lesson`, `fluent_ai/desktop_bridge.py:lesson_start`, and `desktop/electron/renderer.html:lessonHero` do not surface why the lesson was chosen. | -| Adaptive quiz and error categorization | PARTIAL | Mixed quiz types and adaptive count exist in `fluent_ai/agent.py:generate_quiz`; correctness and feedback exist in `fluent_ai/agent.py:evaluate_answers`, `is_correct`, and `feedback_for`; no error category, severity, corrected form, or OpenAI-backed grading. | -| Lesson-to-conversation goal feed | PARTIAL | `conversation_memory.next_speaking_goal` changes after conversation in `fluent_ai/conversation.py:next_speaking_goal` and bridge turn updates, but lesson outcomes do not explicitly set the next conversation goal in `fluent_ai/agent.py:update_progress`. | -| Conversation-to-lesson mistake feed | PARTIAL | Conversation corrections affect `missed_phrases`, `skills`, `topic_mastery`, and `weak_topics` in `fluent_ai/conversation.py:update_conversation_progress` and `fluent_ai/desktop_bridge.py:apply_conversation_turn_progress`; `fluent_ai/agent.py:choose_topic` does not read conversation mistakes directly. | -| Post-call summary | MISSING | Text fallback returns turn feedback in `fluent_ai/desktop_bridge.py:conversation_reply` and renderer chat bubbles in `desktop/electron/renderer.html:sendConversationReply`; no end-of-call summary object or rendering. Voice calls end client-side in `desktop/electron/renderer.html:endRealtimeCall` without persisting call outcomes. | -| Onboarding and placement | MISSING | State defaults to "Demo Learner" A1 in `fluent_ai/state.py:default_state`; renderer starts status and auto lesson in `desktop/electron/renderer.html:refreshStatus` / `ensureLessonStarted`; no placement flow. | -| Home workspace | PARTIAL | Electron renderer has profile, mode buttons, lesson, call stage, review counts, and agent logs in `desktop/electron/renderer.html:renderProfile`, `renderLesson`, and `renderChat`; no true home dashboard, memory inspector, or recommended action model. | -| Memory inspector | MISSING | `/api/progress` returns raw state in `fluent_ai/web.py:FluentAIHandler.do_GET`, but no UI inspector, filtered memory view, or export affordance. | -| Privacy export/delete/reset | MISSING | No privacy fields in `fluent_ai/state.py:default_state`; no export/delete/reset commands in `fluent_ai/desktop_bridge.py:COMMANDS`, `fluent_ai/web.py:FluentAIHandler`, or `desktop/electron/preload.js`. | -| Reliability handling | PARTIAL | API-key errors, bridge process errors, media permission errors, model failures, and JSON bounds exist in `fluent_ai/desktop_bridge.py:_openai_required`, `desktop/electron/main.js:runBridge`, `desktop/electron/main.js:media:request_access`, `desktop/electron/renderer.html:mediaErrorMessage`, and `fluent_ai/web.py:_read_json`; invalid/corrupt progress files and interrupted sessions are not recovered. | -| Mac app packaging | PARTIAL | Electron `.app` packaging exists in `scripts/build_mac_app.sh`; Electron shell in `desktop/electron/main.js`; Python Tk fallback in `macos/desktop_app.py:FluentAIDesktop`. No signing/notarization/update pipeline. | -| Pronunciation feedback | MISSING | Realtime voice session and turn detection exist in `fluent_ai/openai_provider.py:realtime_client_secret` and `_realtime_turn_detection`; no pronunciation scoring, replay, phoneme/stress feedback, or persisted pronunciation memory. | - -## Part 2: Memory Model v2 (WS1) - -### Proposed `data/progress.json` Shape - -Use plain dicts, JSON-serializable values, and `fluent_ai/state.py:migrate_state`. Add helper functions instead of new dependencies. - -```json -{ - "schema_version": 2, - "learner": { - "id": "local-demo-learner", - "display_name": "Demo Learner", - "native_language": null, - "motivation": "", - "active_goals": ["Hold a simple 5-minute conversation", "Build useful daily vocabulary", "Improve conjugation accuracy"], - "preferred_session_length_minutes": 10, - "preferred_correction_style": "gentle", - "preferred_tutor_tone": "encouraging and specific", - "accessibility": {"transliteration": false, "larger_text": false, "reduced_motion": false}, - "created_at": "2026-07-08T00:00:00+00:00" - }, - "active_language": "Spanish", - "languages": { - "Spanish": { - "profile": { - "target_language": "Spanish", - "current_level": "A1", - "level_confidence": 0.45, - "xp": 0, - "streak_days": 1, - "learning_goals": ["Hold a simple 5-minute conversation", "Build useful daily vocabulary", "Improve conjugation accuracy"], - "last_session_at": null - }, - "skills": { - "vocabulary": {"score": 0.34, "trend": "flat", "last_practiced": null, "evidence": []}, - "grammar": {"score": 0.28, "trend": "flat", "last_practiced": null, "evidence": []}, - "conjugations": {"score": 0.26, "trend": "flat", "last_practiced": null, "evidence": []}, - "listening": {"score": 0.30, "trend": "flat", "last_practiced": null, "evidence": []}, - "speaking": {"score": 0.30, "trend": "flat", "last_practiced": null, "evidence": []}, - "pronunciation": {"score": 0.30, "trend": "flat", "last_practiced": null, "evidence": []}, - "reading": {"score": 0.32, "trend": "flat", "last_practiced": null, "evidence": []}, - "writing": {"score": 0.30, "trend": "flat", "last_practiced": null, "evidence": []}, - "translation": {"score": 0.30, "trend": "flat", "last_practiced": null, "evidence": []}, - "fluency": {"score": 0.30, "trend": "flat", "last_practiced": null, "evidence": []} - }, - "topic_mastery": { - "cafe orders": {"recognition": 0.34, "recall": 0.30, "spoken_use": 0.30, "written_use": 0.30, "listening": 0.30, "review_interval_days": 1, "last_success": null, "last_failure": null, "evidence": []}, - "introductions": {"recognition": 0.36, "recall": 0.30, "spoken_use": 0.30, "written_use": 0.30, "listening": 0.30, "review_interval_days": 1, "last_success": null, "last_failure": null, "evidence": []} - }, - "weak_topics": ["past tense", "conjugations", "vocabulary"], - "recent_topics": [], - "review_queue": { - "review_topic_past_tense": { - "id": "review_topic_past_tense", - "item_type": "topic", - "target": "past tense", - "skill": "conjugations", - "topic": "past tense", - "source": "lesson", - "due_at": "2026-07-09T00:00:00+00:00", - "interval_days": 1, - "missed_count": 0, - "success_count": 0, - "last_score": "0/0", - "created_at": "2026-07-08T00:00:00+00:00", - "updated_at": "2026-07-08T00:00:00+00:00" - } - }, - "conversation_memory": { - "sessions_completed": 0, - "total_turns": 0, - "fluency_score": 0.30, - "speaking_confidence": 0.30, - "recent_topics": [], - "missed_phrases": [], - "last_video_context": {"summary": null, "primary_object": null, "confidence": null, "used_at": null}, - "next_speaking_goal": "Answer simple questions in full Spanish sentences.", - "next_conversation_goal": null, - "post_call_summaries": [] - }, - "mistake_memory": { - "mistake_yo_hablar_hablo": { - "id": "mistake_yo_hablar_hablo", - "incorrect_form": "yo hablar", - "corrected_form": "yo hablo", - "context": "Learner used infinitive for first-person present tense.", - "skill": "conjugations", - "topic": "daily routines", - "error_category": "wrong_conjugation", - "first_seen": "2026-07-08T00:00:00+00:00", - "last_seen": "2026-07-08T00:00:00+00:00", - "frequency": 1, - "severity": "medium", - "blocked_meaning": false, - "speech_recurrence": false, - "next_review": "2026-07-09T00:00:00+00:00" - } - }, - "history": [], - "daily_summary": {"last_sent_at": null, "lessons_completed": 0}, - "updated_at": "2026-07-08T00:00:00+00:00" - } - }, - "preferences": {"lesson_minutes": 10, "daily_quiz_questions": 6, "tone": "encouraging and specific", "voice": "alloy", "video_default": "off"}, - "privacy": { - "local_only": true, - "store_raw_audio": false, - "store_raw_video": false, - "store_camera_summaries": true, - "allow_export": true, - "allow_delete": true, - "allow_language_reset": true, - "last_exported_at": null - }, - "events": [], - "updated_at": "2026-07-08T00:00:00+00:00" -} -``` - -### Event History Design - -Event record: - -```json -{ - "id": "evt_20260708_000001", - "type": "quiz_answered", - "occurred_at": "2026-07-08T00:00:00+00:00", - "language": "Spanish", - "session_id": "lesson_20260708_000001", - "source": "lesson_mode", - "summary": "Answered fill_blank for conjugations on daily routines.", - "payload": { - "topic": "daily routines", - "skill": "conjugations", - "question_type": "fill_blank", - "prompt": "Fill in the blank: Yo ___ espanol.", - "answer": "hablar" - } -} +# FluentAI architecture and memory decisions + +## Runtime + +| Component | Responsibility | +| --- | --- | +| `fluent_ai/app.py` | CLI lesson and conversation orchestration | +| `fluent_ai/agent.py`, `conversation.py` | Topic selection, quizzes, scoring, review scheduling, and conversation progress | +| `fluent_ai/openai_provider.py` | OpenAI lesson generation, grading, tutor responses, Realtime, vision, and phrase audio | +| `fluent_ai/desktop_bridge.py` | Shared commands for interactive sessions, onboarding, memory controls, and checkpoints | +| `fluent_ai/web.py` | Local HTTP server exposing the bridge and shared renderer | +| `desktop/electron/` | Electron main/preload processes and shared browser/desktop renderer | +| `fluent_ai/state.py` | Versioned local JSON state, migrations, transactions, and reset boundaries | + +Electron invokes a Python bridge process per command in development and a bundled bridge executable when packaged. The web UI uses HTTP bridge endpoints. The CLI uses the same core lesson, conversation, provider, and state modules. The obsolete Tk launcher has been retired. + +## State schema v2 + +`data/progress.json` is the development default. Packaged desktop storage is described in [packaging](PACKAGING_DESIGN.md). The top-level structure is: + +```text +schema_version: 2 +memory_generation, storage_revision, completed_sessions +learner, active_language, preferences, privacy +languages[language]: + profile, skills, topic_mastery, weak_topics, recent_topics + review_queue, mistake_memory, conversation_memory + history, daily_summary, updated_at +events, event_counter, updated_at ``` -Required event types: `lesson_started`, `lesson_completed`, `quiz_answered`, `answer_evaluated`, `review_scheduled`, `conversation_started`, `learner_replied`, `video_context_used`, `progress_updated`. - -Policy: -- Append via `fluent_ai/state.py:add_event(state, event)`. -- Store language-scoped events in `languages[active_language].history` and top-level audit events in `events`. -- Cap `languages[language].history` to last 300 events per language. -- Cap top-level `events` to last 1000 events. -- Keep `payload` concise: no raw audio, raw video, full image data URLs, API keys, or full prompts. Store summaries and learning signals. -- Rotation happens inside `save_state` or `add_event`, never at arbitrary call sites. - -### Mistake Memory Record - -Fields: -- `id`: stable deterministic slug from normalized `incorrect_form`, `corrected_form`, `skill`, and `topic`. -- `incorrect_form`: learner text or shortest incorrect phrase. -- `corrected_form`: model/correct phrase. -- `context`: one sentence explaining where it happened. -- `skill`: one of vocabulary, grammar, conjugations, listening, speaking, pronunciation, reading, writing, translation, fluency. -- `topic`: lesson or conversation topic. -- `error_category`: vocabulary_missing, wrong_conjugation, wrong_tense, word_order, comprehension, too_short, unnatural, pronunciation_issue, other. -- `first_seen`, `last_seen`: ISO UTC strings. -- `frequency`: incremented on repeats. -- `severity`: low, medium, high. -- `blocked_meaning`: boolean. -- `speech_recurrence`: boolean. -- `next_review`: ISO UTC string or null. - -### Skill Score Records - -Each skill becomes a record: -- `score`: bounded float 0.05 to 0.99. -- `trend`: `up`, `down`, or `flat`, based on latest score delta. -- `last_practiced`: ISO UTC string. -- `evidence`: bounded list of last 8 evidence records: `{ "event_id": "...", "mode": "lesson|conversation", "topic": "...", "delta": 0.045, "note": "..." }`. - -### Exact v1-to-v2 Migration Plan - -Migration is idempotent: if `schema_version == 2` and `languages` exists, normalize missing defaults only; do not remap again. For v1, create v2 and preserve every current value. - -| Current v1 field | New v2 mapping | -|---|---| -| `learner.name` | `learner.display_name` | -| `learner.target_language` | `active_language` and `languages[language].profile.target_language` | -| `learner.current_level` | `languages[language].profile.current_level` | -| `learner.level` | fallback for `languages[language].profile.current_level`; do not keep duplicate except in compatibility helper output | -| `learner.xp` | `languages[language].profile.xp` | -| `learner.streak_days` | `languages[language].profile.streak_days` | -| `learner.learning_goals` | `learner.active_goals` and `languages[language].profile.learning_goals` | -| `skills.{skill}: float` | `languages[language].skills.{skill}.score`; add `trend`, `last_practiced`, bounded `evidence` | -| `skills.conjugation` | normalize to `skills.conjugations` as current `fluent_ai/state.py:migrate_state` already does | -| `topic_mastery.{topic}: float` | `languages[language].topic_mastery.{topic}.recognition`; initialize recall/spoken/written/listening from same score unless evidence says otherwise | -| `weak_topics` | `languages[language].weak_topics` | -| `preferences.lesson_minutes` | top-level `preferences.lesson_minutes` and `learner.preferred_session_length_minutes` | -| `preferences.daily_quiz_questions` | top-level `preferences.daily_quiz_questions` | -| `preferences.tone` | top-level `preferences.tone` and `learner.preferred_tutor_tone` | -| `recent_topics` | `languages[language].recent_topics` | -| `review_queue.{topic}` | `languages[language].review_queue.review_topic_{slug(topic)}`; preserve all nested keys in the new record plus `id`, `item_type`, `source` | -| `history[]` lesson dicts | convert to `languages[language].history[]` typed `lesson_completed` or `progress_updated` events with original dict under `payload.legacy` | -| `history[]` where `mode == conversation` or `conversation_turn` | convert to typed `conversation_started`, `learner_replied`, or `progress_updated` events with original dict under `payload.legacy` | -| `daily_summary` | `languages[language].daily_summary` | -| `conversation_memory.sessions_completed` | `languages[language].conversation_memory.sessions_completed` | -| `conversation_memory.total_turns` | `languages[language].conversation_memory.total_turns` | -| `conversation_memory.fluency_score` | `languages[language].conversation_memory.fluency_score` and `languages[language].skills.fluency.score` if no stronger skill score exists | -| `conversation_memory.speaking_confidence` | `languages[language].conversation_memory.speaking_confidence` | -| `conversation_memory.recent_topics` | `languages[language].conversation_memory.recent_topics` | -| `conversation_memory.missed_phrases[]` | keep as `conversation_memory.missed_phrases[]`; also create best-effort `mistake_memory` records with `incorrect_form: ""`, `corrected_form: phrase`, `source: legacy_missed_phrase` | -| `conversation_memory.last_video_object` | `languages[language].conversation_memory.last_video_context.primary_object` | -| `conversation_memory.next_speaking_goal` | `languages[language].conversation_memory.next_speaking_goal` | -| `conversation_memory.last_session_at` | `languages[language].profile.last_session_at` and `conversation_memory.last_session_at` if kept for compatibility | -| `updated_at` | top-level `updated_at` and `languages[language].updated_at` | - -Lossless rule: preserve unrecognized v1 keys under `languages[language]["legacy_extra"]` and preserve original history dicts inside event payloads. Do not delete data during migration. - -### State Access Helpers - -Add to `fluent_ai/state.py`: -- `active_language(state) -> str` -- `language_state(state, language=None) -> dict` -- `profile_state(state, language=None) -> dict` -- `skill_scores(state, language=None) -> dict[str, float]` -- `set_skill_score(state, skill, score, language=None, evidence=None) -> None` -- `topic_scores(state, language=None) -> dict[str, float]` -- `set_topic_score(state, topic, score, language=None, modality="recognition", evidence=None) -> None` -- `conversation_memory(state, language=None) -> dict` -- `review_queue(state, language=None) -> dict` -- `append_history_event(state, event, language=None) -> None` -- `record_mistake(state, mistake, language=None) -> dict` - -During WS1, keep temporary compatibility helpers that can read either v1 or v2. By the end of WS1, `load_state` should always return v2. - -### Moving Call Sites - -| Current site | Reads/writes moving fields | Change | -|---|---|---| -| `fluent_ai/agent.py:snapshot_progress` | `skills`, `topic_mastery`, `learner.xp`, level | Use `skill_scores`, `topic_scores`, `profile_state`, `current_level`. | -| `fluent_ai/agent.py:current_level` | `learner.current_level`, `learner.level` | Read `profile_state(state).current_level`; keep v1 fallback during migration. | -| `fluent_ai/agent.py:weakest_skill` | `state["skills"]` | Use `skill_scores(state)`. | -| `fluent_ai/agent.py:performance_band` | `history[-3:]` | Use typed `languages[language].history` lesson_completed / answer_evaluated events. | -| `fluent_ai/agent.py:due_review_items` | `review_queue` | Use `review_queue(state)`; support new item ids and old topic keys during migration. | -| `fluent_ai/agent.py:choose_topic` | `weak_topics`, `recent_topics`, due review | Use `language_state`; also consider mistake memory due items in WS2. | -| `fluent_ai/agent.py:generate_lesson` | `learner.target_language`, `learner.learning_goals`, `preferences.lesson_minutes` | Use `active_language`, `profile_state.learning_goals`, top-level preferences; add `reason`. | -| `fluent_ai/agent.py:answer_quiz` | average of `skills` | Use `skill_scores`. | -| `fluent_ai/agent.py:update_progress` | writes skills, topic_mastery, learner xp/level, weak_topics, recent_topics, daily_summary, history | Use setter helpers, per-language profile, typed events, and bounded evidence. | -| `fluent_ai/agent.py:update_review_schedule` | writes `review_queue` | Write per-language review records and `review_scheduled` event. | -| `fluent_ai/agent.py:recommendation` | `weak_topics`, `weakest_skill` | Use `language_state.weak_topics`. | -| `fluent_ai/agent.py:progress_report` | skills, `learner.streak_days`, `daily_summary` | Use per-language profile and daily summary. | -| `fluent_ai/conversation.py:choose_conversation_topic` | target language and `conversation_memory.recent_topics` | Use `active_language`, `conversation_memory`. | -| `fluent_ai/conversation.py:conversation_topics_for` | `learner.target_language` | Use `active_language`. | -| `fluent_ai/conversation.py:visual_reply_options` | target language | Use `active_language`. | -| `fluent_ai/conversation.py:evaluate_reply` | level | Uses migrated `current_level`. | -| `fluent_ai/conversation.py:build_follow_up` | level, target language | Use `current_level` and `active_language`. | -| `fluent_ai/conversation.py:update_conversation_progress` | conversation memory, skills, topic_mastery, weak_topics, history | Use helpers, record mistakes, append typed events. | -| `fluent_ai/conversation.py:next_speaking_goal` | level | Uses migrated `current_level`; may read `next_conversation_goal`. | -| `fluent_ai/desktop_bridge.py:lesson_start` | level, weak_topics, lesson payload | Use helper-backed functions; return `lesson.reason`. | -| `fluent_ai/desktop_bridge.py:lesson_submit` | snapshot/update/save/profile | Helper-backed update plus event writes. | -| `fluent_ai/desktop_bridge.py:conversation_start` | topic selection and session | Helper-backed topic selection; append `conversation_started`. | -| `fluent_ai/desktop_bridge.py:conversation_reply` | turn progress | Use shared conversation progress helper, not bridge-only duplicate logic. | -| `fluent_ai/desktop_bridge.py:apply_conversation_turn_progress` | duplicate writes to conversation memory, skills, topic_mastery, history | Replace with shared `fluent_ai/conversation.py:apply_conversation_turn` or move helper to state/agent module. | -| `fluent_ai/desktop_bridge.py:profile_for` | learner/profile/review/conversation fields | Build stable renderer profile from v2. Keep existing keys; add optional `lesson_reason`, `next_conversation_goal`, `memory_counts`. | -| `fluent_ai/desktop_bridge.py:_load` | mutates `learner.target_language` | Set `active_language`; create `languages[language]` via migration defaults without overwriting other languages. | -| `fluent_ai/web.py:FluentAIHandler.do_GET` | status reads `learner.current_level`, weak topics; `/api/progress` returns raw state | Use `profile_for` for status; optionally return v2 raw progress unchanged for inspector/export. | -| `fluent_ai/web.py:run_lesson_cycle` | lesson update path | Helper-backed update and lesson reason output. | -| `fluent_ai/web.py:run_conversation_cycle` | next speaking goal | Read via `conversation_memory(state)`. | -| `fluent_ai/app.py:run_loop` | weak topics, goals, streak, update progress | Use helpers and print lesson reason. | -| `fluent_ai/app.py:run_conversation_loop` | weak topics, next speaking goal | Use helpers. | -| `desktop/electron/renderer.html:renderProfile` | profile fields from bridge | No raw state read; keep keys stable and render added memory counts/reasons when present. | -| `desktop/electron/renderer.html:refreshStatus` | may persist language | Continue passing `language`; backend sets `active_language`, not `learner.target_language`. | -| `desktop/electron/renderer.html:startLesson` / `lessonHero` | lesson payload | Render `lesson.reason`. | -| `desktop/electron/renderer.html:submitQuiz` / `renderQuizResults` | result payload | Render `error_category` and corrected form if present. | -| `desktop/electron/renderer.html:startConversation` / `sendConversationReply` | session, turn, profile | Render post-call summary when bridge returns it. | -| `desktop/electron/renderer.html:startRealtimeCall` / `endRealtimeCall` | voice call has no state update path | WS2 should add a backend summary/update endpoint for voice transcripts or explicitly mark voice persistence as not implemented. | - -### Backward Compatibility Strategy - -- `load_state` always calls `migrate_state` and returns v2. -- Helpers accept old v1 dicts in tests until fixtures are updated. -- Keep `profile_for` response keys stable: `name`, `language`, `level`, `xp`, `streak_days`, `weak_topics`, `learning_goals`, `fluency_score`, `speaking_confidence`, `next_speaking_goal`, review counts. -- Update tests coherently by changing assertions to v2 paths or helper output: - - `tests/test_agent.py`: default state shape, progress update, review queue, conversation memory. - - `tests/test_conversation_naturalness.py`: confidence under v2 conversation memory. - - `tests/test_web.py`: status and smoke endpoints should not assume flat state. - - `tests/test_renderer_ui.py`: add string checks for lesson reason and error category rendering. - - `tests/test_vision_context.py`: unchanged except active-language helper in prompts if needed. -- No new pip dependencies. Use plain dict helpers and small dataclasses only for local result records. - -## Part 3: Core Adaptive Loop (WS2) - -### Lesson Reason - -Generate in `fluent_ai/agent.py:generate_lesson` after `choose_topic` by adding `lesson_reason_for(state, topic, source_signal)`. - -Reason priority: -1. Due review: "This is due for review because the last score was X and the interval is Y day(s)." -2. Mistake memory: "You missed `incorrect_form` -> `corrected_form` in conversation, so this lesson practices `topic`." -3. Weak topic: "Your weakest current topic is `topic` based on recent lesson and conversation evidence." -4. Rotation: "This avoids repeating recent topics while staying at level `current_level`." - -Surface path: -- Add `lesson["reason"]`. -- Include `reason` in `fluent_ai/desktop_bridge.py:lesson_start` response and logs. -- Include `reason` in `fluent_ai/web.py:run_lesson_cycle` and `fluent_ai/app.py:print_lesson`. -- Render it in `desktop/electron/renderer.html:lessonHero` under the title. - -### Quiz Evaluation Error Categories - -Add fields to `QuizResult`: `error_category: str | None`, `corrected_form: str | None`, `severity: str`, `confidence: float`. - -Required categories: -- `vocabulary_missing` -- `wrong_conjugation` -- `wrong_tense` -- `word_order` -- `comprehension` -- `too_short` -- `unnatural` - -Local/mock assignment: -- Empty answer or fewer than 2 tokens for open-ended prompts: `too_short`. -- Multiple choice vocabulary miss: `vocabulary_missing`. -- Fill blank with focus skill `conjugations`: `wrong_conjugation`. -- Prompt/topic includes "past tense" and answer lacks expected past-tense marker: `wrong_tense`. -- Normalized token set overlaps expected answer but order differs materially: `word_order`. -- Open-ended answer has no expected keywords: `comprehension`. -- Correct by keyword but not exact/natural enough: `unnatural`. - -OpenAI-backed grading: -- Add `OpenAIProvider.evaluate_quiz_answer(state, question, answer, lesson) -> dict`. -- Prompt returns strict JSON: `correct`, `error_category`, `feedback`, `corrected_form`, `severity`, `confidence`. -- `fluent_ai/desktop_bridge.py:lesson_submit` uses provider grading when available; tests patch provider with deterministic category outputs. -- `fluent_ai/agent.py:evaluate_answers` remains local deterministic fallback for unit tests and smoke mode. - -### Lesson Outcome to Conversation Goal - -In `fluent_ai/agent.py:update_progress`: -- Identify weakest missed category/topic from results. -- Write `languages[language].conversation_memory.next_conversation_goal`. -- Update `next_speaking_goal` if lesson misses are speaking-relevant. -- Add `progress_updated` event with `next_conversation_goal`. - -Example: -- Missed `wrong_conjugation` on daily routines -> `next_conversation_goal = "Ask simple daily-routine questions that force first-person present tense."` -- Missed `vocabulary_missing` on cafe orders -> `next_conversation_goal = "Use cafe-order phrases in a short role-play."` - -In `fluent_ai/conversation.py:choose_conversation_topic`: -- Check `next_conversation_goal` before random ladder selection. -- Prefer matching topic/scenario unless video context is active and useful. - -### Conversation Mistake to Next Lesson - -In `fluent_ai/conversation.py:evaluate_reply` and bridge/OpenAI grading: -- Return correction metadata, not only correction text: `incorrect_form`, `corrected_form`, `error_category`, `skill`, `topic`, `blocked_meaning`. -- `update_conversation_progress` and bridge turn updates call `record_mistake`. -- `record_mistake` increments frequency, updates `last_seen`, schedules `next_review`, and appends review queue item. -- `fluent_ai/agent.py:choose_topic` considers due mistake reviews before generic weak topics. -- `generate_lesson` includes the top relevant mistake as a lesson input and reason source. - -### Post-call Summary - -Summary content: -- `session_id` -- `topic` -- `turn_count` -- `average_score` -- `did_well` -- `correction_to_remember` -- `phrase_to_review` -- `next_speaking_goal` -- `confidence_change` -- `next_conversation_should_practice` - -Where it renders: -- Text fallback: when `fluent_ai/desktop_bridge.py:conversation_reply` returns `done: true`, include `post_call_summary`; render via `desktop/electron/renderer.html:sendConversationReply` in the chat log and a compact summary panel. -- CLI/web: append summary in `fluent_ai/app.py:run_conversation_loop` and `fluent_ai/web.py:run_conversation_cycle`. -- Voice: add a later `conversation:end` bridge endpoint that accepts sanitized transcript snippets from realtime events, computes a summary, and writes state. Until then, voice call summaries are UI-only and should not claim persistent memory updates. - -## Part 4: Sequenced Implementation Plan - -### WP1: State v2 Skeleton and Migration - -Files touched: -- `fluent_ai/state.py` -- `tests/test_agent.py` - -Work: -- Add v2 defaults, helper accessors, migration, event append/rotation. -- Preserve v1 migration idempotently and losslessly. - -Tests: -- New tests for v1-to-v2 mapping, idempotent migration, default v2 shape, and event cap. - -Acceptance: -- `python -m unittest tests.test_agent -q` -- `npm run check` - -### WP2: Update Core Lesson State Call Sites - -Files touched: -- `fluent_ai/agent.py` -- `fluent_ai/app.py` -- `fluent_ai/web.py` -- `tests/test_agent.py` -- `tests/test_web.py` - -Work: -- Replace flat state reads/writes with helpers. -- Keep public lesson/quiz behavior stable. -- Write typed lesson/quiz/review/progress events. - -Tests: -- Progress updates mutate v2 skill records, topic records, review queue, daily summary, and typed history. -- Web smoke still passes. - -Acceptance: -- `python -m unittest tests.test_agent tests.test_web -q` -- `npm run check` - -### WP3: Update Conversation State Call Sites - -Files touched: -- `fluent_ai/conversation.py` -- `fluent_ai/desktop_bridge.py` -- `fluent_ai/app.py` -- `fluent_ai/web.py` -- `tests/test_agent.py` -- `tests/test_conversation_naturalness.py` - -Work: -- Use v2 helpers for active language, conversation memory, skills, topic mastery, weak topics, and history. -- Remove or delegate duplicate bridge conversation progress logic. -- Set `active_language` without overwriting other language memory. - -Tests: -- Conversation updates per-language memory. -- Switching Hindi/French/Spanish preserves separate profiles. -- Realtime turn detection reads v2 confidence. - -Acceptance: -- `python -m unittest tests.test_agent tests.test_conversation_naturalness -q` -- `npm run check` - -### WP4: Lesson Reason and Renderer Surface - -Files touched: -- `fluent_ai/agent.py` -- `fluent_ai/desktop_bridge.py` -- `fluent_ai/web.py` -- `fluent_ai/app.py` -- `desktop/electron/renderer.html` -- `tests/test_agent.py` -- `tests/test_renderer_ui.py` - -Work: -- Add `lesson.reason`. -- Include reason in logs and bridge payload. -- Render reason in lesson hero. - -Tests: -- Due review, mistake memory, weak topic, and rotation each produce a reason. -- Renderer contains reason rendering path. - -Acceptance: -- `python -m unittest tests.test_agent tests.test_renderer_ui -q` -- `npm run check` - -### WP5: Error Categories and Mistake Memory - -Files touched: -- `fluent_ai/agent.py` -- `fluent_ai/openai_provider.py` -- `fluent_ai/desktop_bridge.py` -- `fluent_ai/state.py` -- `desktop/electron/renderer.html` -- `tests/test_agent.py` -- `tests/test_renderer_ui.py` - -Work: -- Extend `QuizResult`. -- Add local category heuristics. -- Add optional OpenAI grading through mocked provider in bridge submit path. -- Record mistakes and review items from lesson misses. -- Render category/corrected form in quiz feedback. - -Tests: -- Each required category has a deterministic local test. -- Mocked OpenAI grader category overrides local category. -- Mistake memory increments on repeated miss. - -Acceptance: -- `python -m unittest tests.test_agent tests.test_renderer_ui -q` -- `npm run check` - -### WP6: Lesson and Conversation Feedback Loop - -Files touched: -- `fluent_ai/agent.py` -- `fluent_ai/conversation.py` -- `fluent_ai/desktop_bridge.py` -- `tests/test_agent.py` - -Work: -- Lesson outcomes set `next_conversation_goal`. -- Conversation topic selection consumes that goal. -- Conversation corrections write mistake memory and influence next lesson choice. - -Tests: -- Missed conjugation lesson leads to daily-routine/present-tense conversation goal. -- Conversation correction creates mistake record and next lesson reason references it. - -Acceptance: -- `python -m unittest tests.test_agent -q` -- `npm run check` +Per-language blocks prevent one language's progress from overwriting another. Skill/topic records retain evidence and confidence alongside scores. Mistake records retain corrected forms and scheduling information. V1 migration preserves legacy profile, scores, review queue, history, and conversation memory; normalization is idempotent and removes obsolete compatibility mirrors. -### WP7: Post-call Summary +Histories retain the latest 300 events per language and 1,000 globally. A persistent event counter keeps event IDs unique after truncation. This bounded history is an audit aid, not a complete event-sourced database. -Files touched: -- `fluent_ai/conversation.py` -- `fluent_ai/desktop_bridge.py` -- `fluent_ai/app.py` -- `fluent_ai/web.py` -- `desktop/electron/renderer.html` -- `tests/test_agent.py` -- `tests/test_renderer_ui.py` -- `tests/test_web.py` +## Adaptive loop -Work: -- Generate summary after text/CLI/web conversation completion. -- Persist summary in `conversation_memory.post_call_summaries`. -- Render summary when text fallback reaches turn target. -- Document voice persistence gap in logs until `conversation:end` exists. +Lesson selection prioritizes due review, due mistake memory, weak topics, then rotation. The lesson carries a selection reason. Misses update mistake memory and review dates; lesson outcomes supply a next-conversation goal. Conversation corrections map back to teachable topics and become immediately due practice. Successful mistake practice reschedules the topic to avoid a perpetual due loop. -Tests: -- Summary contains required fields. -- Renderer has post-call summary rendering. -- Web conversation output includes summary. +OpenAI can grade free answers and conversation turns; malformed grading output falls back to local scoring. This grading fallback does not provide an offline user-facing tutor: real generation still requires OpenAI. -Acceptance: -- `python -m unittest tests.test_agent tests.test_renderer_ui tests.test_web -q` -- `npm run check` +## Durability -### WP8: Privacy and Inspector Foundation +State updates use cross-process locks, short reload/apply transactions, and atomic file replacement. Revisions reject stale snapshots. Memory generations invalidate sessions after delete-all. Lesson submission and voice completion carry stable session IDs so retries reuse committed results rather than award progress twice. -Files touched: -- `fluent_ai/state.py` -- `fluent_ai/desktop_bridge.py` -- `fluent_ai/web.py` -- `desktop/electron/preload.js` -- `desktop/electron/main.js` -- `desktop/electron/renderer.html` -- `tests/test_agent.py` -- `tests/test_web.py` -- `tests/test_renderer_ui.py` +Lesson and call checkpoints live beside progress under `sessions/`. The renderer captures a call's language and uses a shared retryable finalizer for navigation, language changes, and window close. Failed saves remain available for retry. Corrupt progress is backed up before recovery; [delete-all](ONBOARDING_HOME_DESIGN.md) also clears managed recovery artifacts. -Work: -- Add bridge commands for export, delete all memory, and reset active language. -- Add safe memory-inspector payload that excludes raw image/audio and secrets. -- Render a compact memory inspector. +## Verification -Tests: -- Export returns v2 state. -- Reset removes one language only. -- Delete resets to default v2. -- Renderer exposes inspector controls. +Run the README's Python suite, mocked smoke, and `npm run check`. State migration, language isolation, concurrent writes, stale sessions, memory deletion, and renderer finalization have regression coverage in `tests/`. Use temporary state paths for any manual experiment; do not overwrite the checked-in demo profile. -Acceptance: -- `python -m unittest tests.test_agent tests.test_web tests.test_renderer_ui -q` -- `npm run check` +See [limitations and remaining work](FLUENTAI_LIMITLESS_IDEAL.md). diff --git a/docs/ONBOARDING_HOME_DESIGN.md b/docs/ONBOARDING_HOME_DESIGN.md index 848662f..4c9899f 100644 --- a/docs/ONBOARDING_HOME_DESIGN.md +++ b/docs/ONBOARDING_HOME_DESIGN.md @@ -1,475 +1,30 @@ -# FluentAI Onboarding, Home Workspace, and Memory Inspector Design +# Onboarding, home, and privacy boundaries -Scope: WS3 onboarding/placement and WS4 home tutor workspace. This is a design-only handoff. Do not introduce a frontend framework; keep the shared single-file renderer used by Electron and `fluent_ai/web.py`. +## Current flow -Current anchors: -- State v2 is implemented in `fluent_ai/state.py`: `load_state`, `migrate_state`, `active_language`, `language_state`, `profile_state`, `skill_scores`, `set_skill_score`, `topic_scores`, `set_topic_score`, `conversation_memory`, `review_queue`, `record_mistake`, `add_event`, `save_state`. -- Lesson evaluation exists in `fluent_ai/agent.py`: `LESSON_BANK`, `generate_lesson`, `generate_quiz`, `evaluate_answers`, `update_progress`, `due_review_items`, `due_mistake_items`, `choose_topic_with_reason`. -- Conversation evaluation exists in `fluent_ai/conversation.py`: `choose_conversation_topic`, `evaluate_reply_with_metadata`, `apply_turn_progress`, `persist_post_call_summary`. -- Bridge surface is `fluent_ai/desktop_bridge.py:COMMANDS`; current commands are `status`, `realtime_client_secret`, `vision_analyze_frame`, `lesson_start`, `lesson_submit`, `conversation_start`, `conversation_reply`, `conversation_end`. -- Web bridge proxy is `fluent_ai/web.py:FluentAIHandler._run_bridge_command` via `POST /api/bridge/`. `/api/progress` currently returns raw v2 state. -- Electron exposes bridge calls in `desktop/electron/preload.js`; `desktop/electron/main.js:runBridge` maps IPC handlers to Python bridge commands. -- Renderer structure is `desktop/electron/renderer.html`: header (`headerMeta`, `headerPills`, `languageSelect`, mode tabs), workspace (`workspace`), launch controls (`startLessonBtn`, `startConversationBtn`), lesson pane (`lessonPane`, `lessonContent`), conversation pane (`conversationPane`, `callStage`, `chatLog`), and agent rail (`agentLogPanel`, `agentLogs`). The file already has the 2026-07-08 aurora studio CSS override and post-call summary rendering. +The shared Electron/browser renderer starts with onboarding when required, then Home. Onboarding records native/target language, motivation, goals, session length, correction/tutor preferences, accessibility preferences, and speaking comfort. A learner may start as a beginner or complete placement. Placement combines quiz/written/conversation evidence to seed a level, confidence, strengths, weaknesses, and initial practice goals; it is not a certified assessment. -## Part 1: WS3 Onboarding and Placement +`home_summary` returns the next recommendation, review preview, recent progress, and speaking-confidence summary. Recommendations reuse the lesson/conversation selection logic so Home and the next activity agree. -### Trigger and Lifecycle +## Memory controls -First launch means either: -- the selected `state_path` file is absent before `load_state` creates it, or -- the loaded v2 state lacks `learner.onboarded_at`. +| Bridge command | Behavior | +| --- | --- | +| `memory_inspect` | Returns a sanitized view of learner and selected-language memory | +| `memory_export` | Exports one language by default, or all languages with `scope=all`; records export time | +| `memory_reset_language` | Requires `RESET `; resets that language while preserving other languages and top-level learner/preferences/privacy | +| `memory_delete_all` | Requires `DELETE ALL MEMORY`; replaces progress with a fresh generation and removes managed session checkpoints, TTS cache, corrupt backups, and temporary recovery files | -`onboarding_status` must check `Path.exists()` before `load_state` so the UI can distinguish "brand new file" from "existing profile missing onboarding metadata". Re-run onboarding from the home workspace through a "Meet your tutor again" affordance. Re-running must preserve learning memory by default and only update onboarding fields unless the user chooses placement again. +Electron exports through a save dialog; the browser downloads JSON. Exports omit secrets, raw media, and internal fields through the bridge sanitizer. They are selected memory views, not full round-trip database backups. Delete-all prevents stale in-flight sessions from restoring deleted learning memory. Previously exported copies outside managed storage and the separately managed API key are not learner-memory deletion targets. -Stop conditions: -- Onboarding is complete when `learner.onboarded_at` is set and `onboarding_status.requires_onboarding == false`. -- Placement is complete when `languages[lang].profile.placement_completed_at` is set and a `placement_completed` event exists for that language. -- Skipping placement writes an explicit `placement_completed` event with `payload.method == "skip_beginner"` so later logic can explain why the learner starts at A1. +## Local storage versus provider access -### Onboarding Steps and State Mapping +The schema's `privacy.local_only` describes local learner-memory storage. The app still sends relevant learner context and text to OpenAI for teaching and grading, audio for Realtime calls, camera frames for enabled vision context, and phrase text for speech synthesis. It is not an offline app. -Use one full-screen overlay stage before the normal workspace. Keep copy short, like a tutor intake, not a settings wizard. +Raw microphone recordings and raw camera video are not persisted as learner memory. Transcripts, corrections, session checkpoints, and camera summaries can contain personal context. Generated phrase audio is cached under `cache/tts` beside the state file and removed by delete-all. Camera sampling skips unchanged frames before inference, with explicit capture and a bounded refresh; fake test feeds must not become claimed real-world objects. -| Step | Answer | Existing state writes | Additions needed | -|---|---|---|---| -| Name or nickname | free text | `learner.display_name` | none | -| Target language | `Hindi`, `Spanish`, `French` | `active_language`; create `languages[lang]` through `language_state`; `languages[lang].profile.target_language` | none | -| Why / goal | short free text plus optional chips | `learner.motivation`; `learner.active_goals`; `languages[lang].profile.learning_goals` | none | -| Self-estimated level | `new`, `A1`, `A2`, `B1+`, `not sure` | if placement skipped, seed `languages[lang].profile.current_level`; otherwise do not trust it as final | `languages[lang].profile.self_reported_level` | -| Speaking comfort | `quiet`, `some`, `comfortable` | seed `languages[lang].conversation_memory.speaking_confidence` conservatively (`0.22`, `0.35`, `0.50`) | `languages[lang].profile.speaking_comfort` | -| Session length | `5`, `10`, `15` minutes | `learner.preferred_session_length_minutes`; `preferences.lesson_minutes` | none | -| Voice/video expectations | voice on/off preference, video default off/on | `preferences.voice`; `preferences.video_default`; do not request mic/camera yet | optional `preferences.voice_default` if voice availability should be separate from selected voice name | -| Privacy note | local memory acknowledgement | `privacy.local_only = true`; `privacy.store_raw_audio = false`; `privacy.store_raw_video = false`; `privacy.store_camera_summaries = true` | `learner.onboarded_at`; optional `privacy.local_memory_notice_seen_at` | +The privacy fields document intent and supported controls; do not treat them as a general consent engine or a guarantee about provider retention. Keep API keys out of progress, exports, and logs. Electron key storage and precedence are described in [packaging](PACKAGING_DESIGN.md). -Recommended additions normalized by `fluent_ai/state.py:_normalize_v2_state` / `_normalize_language_state`: -- `learner.onboarded_at: str | null` -- `learner.last_onboarding_at: str | null` for reruns -- `privacy.local_memory_notice_seen_at: str | null` -- `languages[lang].profile.self_reported_level: str | null` -- `languages[lang].profile.speaking_comfort: "quiet" | "some" | "comfortable" | null` -- `languages[lang].profile.placement_completed_at: str | null` -- `languages[lang].profile.placement_method: "adaptive" | "skip_beginner" | null` -- `languages[lang].profile.first_practice_goal: str | null` -- `languages[lang].profile.judged_strengths: list[str]` -- `languages[lang].profile.judged_weaknesses: list[str]` +## Validation limits -### Placement Flow - -Placement is short and adaptive, not a separate test product. - -Default flow: -1. `placement_start` builds 3-5 comprehension/recall items from existing lesson bank material. Use `fluent_ai/agent.py:LESSON_BANK` for Spanish and `_generic_lesson_bank` via a generated lesson for non-Spanish. Prefer mixed `multiple_choice`, `fill_blank`, and reading comprehension questions from `generate_quiz`. -2. Optional written prompt: one `open_ended` item evaluated through `evaluate_answers`. If OpenAI grading is available, reuse `_apply_openai_quiz_grading` exactly as `lesson_submit` does; otherwise local evaluator still works. -3. Optional typed/spoken prompt: use `conversation.py:evaluate_reply_with_metadata` against a topic from `choose_conversation_topic`. Spoken input may be transcript text; do not store raw audio. -4. Output judged level, confidence, strongest skills, weakest skills, first lesson/practice goal, and first conversation goal. - -Skippable path: -- CTA: "Start as beginner instead". -- Set `profile.current_level = "A1"`, `profile.level_confidence = 0.35`, `profile.placement_method = "skip_beginner"`, `profile.first_practice_goal = "Build basic introductions and daily phrases."`, and `conversation_memory.next_conversation_goal` to an A1 introductions goal. - -Judging rule: -- Compute quiz accuracy and optional prompt scores as deterministic signals. -- If self-report is `new`, `A1`, or `not sure`, cap judged level at `B1` even with perfect results. -- Suggested bands: `<45% => A1`, `45-72% => A2`, `>72% => B1`; only allow `B2+` later through continuous assessment, not first-run placement. -- `profile.level_confidence`: `0.35` skip, `0.50` quiz-only, `0.60` quiz+written, `0.70` quiz+written+conversation. -- Skill seeds: aggregate by question `skill`; optional written affects `writing` and focus skill; optional speaking affects `speaking`, `fluency`, and topic `spoken_use`. -- Strongest/weakest: top/bottom 2 skills after bounded seed updates through `set_skill_score`. - -Placement writes: -- `languages[lang].profile.current_level` -- `languages[lang].profile.level_confidence` -- `languages[lang].profile.placement_completed_at` -- `languages[lang].profile.placement_method` -- `languages[lang].profile.first_practice_goal` -- `languages[lang].profile.judged_strengths` -- `languages[lang].profile.judged_weaknesses` -- `languages[lang].weak_topics` -- `languages[lang].conversation_memory.speaking_confidence` -- `languages[lang].conversation_memory.next_speaking_goal` -- `languages[lang].conversation_memory.next_conversation_goal`, shaped like current lesson-derived goals: `{source, topic, skill, instruction, reason}` -- skill/topic evidence with `evidence.mode == "placement"` -- top-level and language history event `type: "placement_completed"` through `add_event` - -Example `placement_completed.payload`: -```json -{ - "method": "adaptive", - "self_reported_level": "A1", - "judged_level": "A2", - "level_cap_applied": false, - "quiz_score": "4/5", - "written_score": 0.62, - "conversation_score": 0.48, - "strongest_skills": ["vocabulary", "reading"], - "weakest_skills": ["speaking", "conjugations"], - "first_practice_goal": "Practice daily introductions with complete sentences.", - "first_conversation_goal": "Answer two simple introduction questions in full Spanish sentences." -} -``` - -### New Bridge Commands - -Add to `fluent_ai/desktop_bridge.py:COMMANDS`, expose through `preload.js`, map in `main.js`, and rely on existing `/api/bridge/` in `web.py`. - -`onboarding_status` -- Payload: `{ "state_path"?: string, "language"?: string }` -- Response: -```json -{ - "ok": true, - "requires_onboarding": true, - "is_first_launch": true, - "requires_placement": true, - "profile": {}, - "defaults": {"language": "Spanish", "session_minutes": 10, "video_default": "off"}, - "logs": ["[Onboarding Agent] First launch detected."] -} -``` -- Semantics: no destructive writes except normal `load_state` creation/migration. If file existed and `learner.onboarded_at` exists, return `requires_onboarding: false`. - -`onboarding_submit` -- Payload: -```json -{ - "display_name": "Johan", - "language": "Spanish", - "motivation": "Travel and conversation", - "goals": ["Hold a 5-minute conversation"], - "self_reported_level": "A1", - "speaking_comfort": "some", - "session_minutes": 10, - "voice_default": "openai", - "video_default": "off", - "privacy_local_only": true, - "state_path": "data/progress.json" -} -``` -- Response: `{ "ok": true, "profile": profile_for(...), "requires_placement": true, "logs": [...] }` -- Semantics: update only intake fields, set `learner.onboarded_at` if missing, always set `learner.last_onboarding_at`, add `onboarding_completed` event. Do not run placement here. - -`placement_start` -- Payload: `{ "language": "Spanish", "include_written": true, "include_conversation": true, "state_path"?: string }` -- Response: -```json -{ - "ok": true, - "session": { - "id": "placement_20260708_000001", - "language": "Spanish", - "items": [], - "written_prompt": {}, - "conversation_prompt": {} - }, - "profile": {}, - "logs": ["[Placement Agent] Built a 5-item adaptive check from the lesson bank."] -} -``` -- Semantics: read state and produce questions; do not save score yet unless a lightweight `placement_started` event is desired. Use existing quiz item shape so renderer can reuse `questionNode` patterns. - -`placement_submit` -- Payload: -```json -{ - "session": {}, - "answers": ["Nice to meet you", "Soy", "..."], - "written_answer": "Me llamo Johan.", - "conversation_answer": "Me llamo Johan y vivo en Nueva York.", - "skip_beginner": false, - "state_path": "data/progress.json" -} -``` -- Response: -```json -{ - "ok": true, - "profile": {}, - "placement": { - "judged_level": "A2", - "level_confidence": 0.6, - "strongest_skills": ["vocabulary"], - "weakest_skills": ["speaking"], - "first_practice_goal": "...", - "first_conversation_goal": "..." - }, - "logs": ["[Placement Agent] Judged starting level A2 from 4/5 placement items."] -} -``` -- Semantics: evaluate via existing evaluators, update v2 state, save, add `placement_completed`. - -### Renderer Design - -Add a full-screen overlay before workspace: -- DOM after `` start, before `
`: - - `section#onboardingOverlay.onboarding-overlay.hidden` - - `div#onboardingStage` - - `div#onboardingProgress` - - `form#onboardingForm` - - `div#placementStage.hidden` - - `div#placementItems` - - `button#onboardingNextBtn` - - `button#placementSkipBtn` - - `button#placementSubmitBtn` -- Add CSS near aurora override for `.onboarding-overlay`, `.onboarding-card`, `.onboarding-grid`, `.placement-item`, `.privacy-note`; use existing variables and 8-24px radius rhythm from the aurora studio style. -- Add `state.onboarding`, `state.placementSession`. -- Extend `window.fluentAI` fallback with `onboardingStatus`, `submitOnboarding`, `startPlacement`, `submitPlacement`. -- Add to `els`: overlay/stage/form/buttons. -- New JS functions: - - `initOnboarding()`: call `window.fluentAI.onboardingStatus(bridgePayload())` before `refreshStatus`. - - `showOnboardingOverlay(status)` - - `renderOnboardingStep(index)` - - `collectOnboardingAnswers()` - - `submitOnboarding(event)` - - `startPlacement()` - - `renderPlacement(session)` - - `submitPlacement({skipBeginner=false})` - - `finishOnboarding(result)`: hide overlay, `renderProfile`, `appendLogs`, then `refreshStatus(true)` and `ensureLessonStarted`. -- Startup order changes from `refreshStatus().then(ensureLessonStarted)` to `initOnboarding()`, which either runs the overlay or continues to existing status/lesson auto-start. -- Re-run affordance: add a small button in the future home/memory region, not the header tabs, labeled "Meet your tutor again". It calls `showOnboardingOverlay({rerun:true})`. - -## Part 2: WS4 Home Tutor Workspace - -### Home Summary Command - -Add a dedicated `home_summary` bridge command rather than bloating `status`. - -`home_summary` -- Payload: `{ "language": "Spanish", "state_path"?: string }` -- Response: -```json -{ - "ok": true, - "profile": {}, - "today": { - "kind": "due_review", - "title": "Review due phrases", - "body": "2 items are due; start with past tense.", - "cta": "Review due phrases", - "mode": "lesson", - "topic": "past tense", - "reason": "Due reviews come before fresh lessons." - }, - "review_preview": [], - "recent_progress": [], - "speaking_confidence": {"score": 0.34, "trend": "up", "recent": []}, - "memory_counts": {"events": 34, "mistakes": 3, "reviews_due": 2}, - "logs": ["[Home Agent] Recommended due review before fresh lesson."] -} -``` - -Deterministic recommendation rule, computed in `desktop_bridge.py:home_summary` using v2 helpers: -1. If `due_review_items(state)` is non-empty, recommend lesson mode with CTA "Review due phrases". -2. Else if `due_mistake_items(state)` is non-empty, recommend lesson mode with CTA "Practice yesterday's weak topic". -3. Else if `conversation_neglected(state)` is true, recommend conversation mode with CTA "Start live tutor call". -4. Else if `conversation_memory.next_conversation_goal` exists and the last completed lesson produced it but no later conversation has addressed it, recommend conversation mode with CTA "Use this in conversation". -5. Else if `profile.first_practice_goal` exists and `daily_summary.lessons_completed == 0`, recommend lesson mode with CTA "Start today's lesson". -6. Else recommend fresh lesson mode with CTA "Start today's lesson". - -Conversation neglected rule: -- Let `lesson_completed_count_since_last_conversation` be the count of `lesson_completed` events after the last `conversation_started` event in `languages[lang].history`. -- If count >= 3, or `conversation_memory.sessions_completed == 0` and at least one lesson is complete, conversation is neglected. -- Reason text: "You have practiced lessons without a tutor call recently." - -CTA behavior: -- `Start today's lesson`: calls existing `startLesson()`. -- `Review due phrases`: calls existing `startLesson()`; backend topic selection already prioritizes due reviews. -- `Practice yesterday's weak topic`: calls existing `startLesson()`; topic selection already checks due mistakes before weak topics. -- `Start live tutor call`: `setMode("conversation")`, keep current `voiceMode`, then `startConversation()`. -- `Use this in conversation`: same as call CTA; `choose_conversation_topic` already reads `next_conversation_goal`. - -### Home Regions Mapped to Current Renderer - -Current renderer starts in lesson mode and auto-starts via `ensureLessonStarted`. WS4 should make home the default first workspace surface, then let CTA launch modes. - -New: -- `div#homePane.pane` as the default pane in `workspace`, replacing the initial empty-state-first experience. -- `section#todayPanel` for recommendation card. -- `section#homeProfilePanel` with profile summary and "Meet your tutor again". -- `section#reviewPreviewPanel` with top 3 due/scheduled reviews. -- `section#recentProgressPanel` from typed events. -- `section#speakingTrendPanel` with confidence score and trend. -- `details#memoryInspectorPanel` for sanitized memory. - -Reuse: -- Header language selector: `languageSelect`. -- Mode launchers: existing `lessonModeBtn`, `conversationModeBtn`, `startLessonBtn`, `startConversationBtn`; move/duplicate CTA affordances into home panel without changing backend semantics. -- Lesson area: `lessonPane`, `lessonContent`, `lessonHero`, `submitQuiz`. -- Call stage: `conversationPane`, `callStage`, `chatLog`, voice/text fallback. -- Agent log: `agentLogPanel`, `agentLogs`, `appendLogs`, `syncAgentLogPanel`. -- Profile rendering: extend `renderProfile`; keep compatibility keys from `profile_for`. - -Renderer functions: -- Add `state.homeSummary`. -- Add `loadHomeSummary()`, `renderHome(summary)`, `runTodayAction()`, `showHome()`. -- `refreshStatus` can still update profile; `loadHomeSummary` owns today/review/recent/memory counts. -- After `submitQuiz`, `sendConversationReply`, and `endRealtimeCall`, call `loadHomeSummary()` after `renderProfile`. -- Disable lesson auto-start on initial app load once home exists. `ensureLessonStarted` should only run from explicit lesson CTA or if preserving current demo behavior behind a `?autostart=lesson` style flag later. - -### Memory Inspector - -Recommend a dedicated sanitized command: `memory_inspect`. Do not drive the inspector from `/api/progress`, because `/api/progress` is raw state and can include future sensitive payloads. - -`memory_inspect` -- Payload: `{ "language": "Spanish", "state_path"?: string }` -- Response: -```json -{ - "ok": true, - "language": "Spanish", - "learner": {"display_name": "Johan", "motivation": "...", "onboarded_at": "..."}, - "profile": {"current_level": "A2", "level_confidence": 0.6, "learning_goals": []}, - "skills": [{"name": "speaking", "score": 0.34, "trend": "up", "last_practiced": "..."}], - "topic_mastery": [{"topic": "introductions", "recognition": 0.5, "recall": 0.42, "spoken_use": 0.35}], - "mistakes": [{"incorrect_form": "...", "corrected_form": "...", "skill": "speaking", "topic": "introductions", "frequency": 2, "next_review": "..."}], - "review_queue": [{"id": "...", "target": "past tense", "skill": "conjugations", "due_at": "...", "source": "lesson"}], - "conversation": {"speaking_confidence": 0.34, "next_speaking_goal": "...", "next_conversation_goal": {}, "post_call_summaries": []}, - "recent_events": [{"id": "evt_...", "type": "lesson_completed", "occurred_at": "...", "summary": "..."}], - "privacy": {"local_only": true, "store_raw_audio": false, "store_raw_video": false, "store_camera_summaries": true}, - "redactions": ["raw audio", "raw video", "image data URLs", "API keys"] -} -``` - -Sanitization rules: -- Include summaries and learning signals only. -- Do not include raw image data URLs, raw audio, raw video, API keys, realtime client secrets, raw bridge `raw` stderr, or full transcript beyond bounded post-call summaries already stored. -- `last_video_context.summary` and `primary_object` are allowed because v2 is designed to store camera summaries, not images. - -Inspector UI: -- `details#memoryInspectorPanel` collapsed by default. -- Sections: Profile, Skills, Topic mastery, Mistake memory, Review queue, Conversation memory, Recent events, Privacy. -- Use small tables/lists, not raw JSON by default. -- Include "Show JSON" toggle for the sanitized payload only. -- Agent log every open/export/reset/delete action. - -Export and delete commands: - -`memory_export` -- Payload: `{ "language"?: "Spanish" | null, "scope": "all" | "language", "state_path"?: string }` -- Response: `{ "ok": true, "filename": "fluentai-memory-20260708.json", "data": {...}, "logs": [...] }` -- Semantics: bridge returns sanitized export data and updates `privacy.last_exported_at`; Electron uses `dialog.showSaveDialog` in `main.js` to write JSON to a user-chosen location, while web fallback downloads a Blob. Browser download must not require filesystem paths. - -`memory_reset_language` -- Payload: `{ "language": "Spanish", "confirm": "RESET Spanish", "state_path"?: string }` -- Response: `{ "ok": true, "profile": {}, "memory": {}, "logs": [...] }` -- Semantics: replace only `languages[language]` with a fresh `_default_language_state(language)` via a public helper such as `reset_language_state(state, language)`. Preserve top-level `learner`, `preferences`, `privacy`, other languages, and `learner.onboarded_at`. If resetting active language, keep it active but fresh. Add `language_reset` event to top-level `events` after reset if possible. - -`memory_delete_all` -- Payload: `{ "confirm": "DELETE ALL MEMORY", "state_path"?: string }` -- Response: `{ "ok": true, "profile": {}, "logs": [...] }` -- Semantics: replace the file with `default_state(active_language_or_payload_language)` via a public helper such as `delete_all_memory(path, language)`. This removes onboarding and placement, so next `onboarding_status` returns `requires_onboarding: true`. Do not delete the file path itself; write fresh v2 JSON for reliability. - -Electron IPC additions: -- `preload.js`: `homeSummary`, `memoryInspect`, `memoryExport`, `memoryResetLanguage`, `memoryDeleteAll`, plus onboarding/placement methods. -- `main.js`: map IPC to bridge commands. For export, main should own `dialog.showSaveDialog` and `fs.writeFile`; renderer should receive `{ok, path?}` but never need arbitrary filesystem access. -- Web fallback in renderer: call `/api/bridge/memory_export`, then `URL.createObjectURL(new Blob([JSON.stringify(data, null, 2)]))`. - -## Part 3: Sequenced Implementation Plan - -Renderer packages must be sequential. Do not run two renderer-editing agents in parallel. - -### WP1: State Metadata and Onboarding Bridge - -Files touched: -- `fluent_ai/state.py` -- `fluent_ai/desktop_bridge.py` -- `desktop/electron/main.js` -- `desktop/electron/preload.js` -- `tests/test_state_v2.py` -- `tests/test_agent.py` -- `tests/test_web.py` - -New bridge commands: `onboarding_status`, `onboarding_submit`. - -Renderer regions: none in this package. - -Tests: -- state defaults normalize new onboarding/profile fields. -- absent file returns `is_first_launch: true`. -- existing state without `learner.onboarded_at` requires onboarding. -- `onboarding_submit` writes exact mapped fields and `onboarding_completed`. -- `/api/bridge/onboarding_status` and `/api/bridge/onboarding_submit` work through `web.py`. - -Acceptance: `npm run check` green. - -### WP2: Placement Bridge and Evaluator Reuse - -Files touched: -- `fluent_ai/desktop_bridge.py` -- `fluent_ai/state.py` only if reset/helper additions from WP1 did not cover placement metadata normalization -- `tests/test_agent.py` -- `tests/test_web.py` - -New bridge commands: `placement_start`, `placement_submit`. - -Renderer regions: none in this package. - -Tests: -- `placement_start` returns 3-5 existing quiz-shaped items. -- `placement_submit` adaptive path writes judged level, skill evidence, first goals, `next_conversation_goal`, and `placement_completed`. -- beginner self-report cap prevents `B2+`. -- skip path writes A1 beginner state and `payload.method == "skip_beginner"`. -- OpenAI quiz grading remains mocked and no new model interface is introduced. -- web bridge endpoint accepts both placement commands. - -Acceptance: `npm run check` green. - -### WP3: Onboarding and Placement Renderer Overlay - -Files touched: -- `desktop/electron/renderer.html` -- `desktop/electron/preload.js` -- `desktop/electron/main.js` -- `tests/test_renderer_ui.py` -- `tests/test_web.py` - -New bridge commands exposed to renderer: `onboarding_status`, `onboarding_submit`, `placement_start`, `placement_submit`. - -Renderer regions: -- `onboardingOverlay` -- `onboardingStage` -- `onboardingForm` -- `placementStage` -- `placementItems` -- placement skip/submit buttons - -Tests: -- renderer string assertions for overlay IDs, fallback web bridge methods, "Start as beginner instead", privacy local-memory copy, and startup `initOnboarding`. -- preload exposes onboarding/placement APIs. -- main maps IPC to bridge commands. -- web fallback can post to `/api/bridge/onboarding_status`. - -Acceptance: `npm run check` green and first launch visibly blocks workspace until onboarding is submitted or completed. - -### WP4: Home Workspace and Memory Inspector - -Files touched: -- `fluent_ai/desktop_bridge.py` -- `fluent_ai/state.py` -- `fluent_ai/web.py` only if adding a convenience download route; otherwise existing bridge proxy is enough -- `desktop/electron/main.js` -- `desktop/electron/preload.js` -- `desktop/electron/renderer.html` -- `tests/test_agent.py` -- `tests/test_web.py` -- `tests/test_renderer_ui.py` - -New bridge commands: `home_summary`, `memory_inspect`, `memory_export`, `memory_reset_language`, `memory_delete_all`. - -Renderer regions: -- `homePane` -- `todayPanel` -- `homeProfilePanel` -- `reviewPreviewPanel` -- `recentProgressPanel` -- `speakingTrendPanel` -- `memoryInspectorPanel` - -Tests: -- home recommendation priority: due reviews > due mistakes > neglected conversation > lesson-to-conversation goal > first practice goal > fresh lesson. -- sanitized memory payload excludes raw image data URLs and secret-looking strings. -- export returns sanitized v2 payload and updates `privacy.last_exported_at`. -- reset language preserves other languages and top-level learner/preferences/privacy. -- delete all resets to default and makes onboarding required. -- renderer assertions for home pane IDs, memory inspector sections, export/reset/delete confirmations, and no initial `ensureLessonStarted` autostart when home is present. -- web endpoint tests for all new bridge commands. - -Acceptance: `npm run check` green. Demo can show: intake, placement result, home recommendation reason, mode launch, progress update, memory inspector, export/reset/delete controls, and agent decision logs throughout. +Tests cover sanitization, export/reset behavior, complete managed-memory deletion, and stale-session rejection. Physical permission dialogs and clean-account desktop acceptance remain manual checks in the [roadmap](FLUENTAI_LIMITLESS_IDEAL.md). diff --git a/docs/PACKAGING_DESIGN.md b/docs/PACKAGING_DESIGN.md index 1ad0ebe..74bf88e 100644 --- a/docs/PACKAGING_DESIGN.md +++ b/docs/PACKAGING_DESIGN.md @@ -1,387 +1,49 @@ -# FluentAI Mac Packaging Design +# macOS packaging and key management -## Scope +## Current package -Target: a double-clickable consumer `FluentAI.app` that works without Terminal, a repo checkout, `node_modules`, `.venv`, or system Python. It remains API-key-first, local-first, and keeps the Python agent core. +Electron Builder packages `desktop/electron/**` and the PyInstaller bridge from `dist-py/fluentai-bridge/` using `electron-builder.yml`. Packaged Electron runs `Contents/Resources/bridge/fluentai-bridge/fluentai-bridge`; development runs `python -m fluent_ai.desktop_bridge` from the checkout. -Audited files: `scripts/build_mac_app.sh`, `desktop/electron/main.js`, `preload.js`, `renderer.html`, `fluent_ai/desktop_bridge.py`, `openai_provider.py`, `state.py`, `config.py`, `package.json`, and `docs/FLUENTAI_LIMITLESS_IDEAL.md`. +This keeps one Python learning engine across CLI, browser, and desktop while removing a checkout/Python-install dependency from the packaged runtime. A bundled Python distribution remains an alternative if PyInstaller becomes unsuitable; rewriting the bridge in Node would duplicate the learning engine and is not the current design. The Tk launcher is retired. -## Current Packaging Audit +The PyInstaller spec collects the OpenAI dependency graph, curriculum assets, and certificates. Direct provider HTTPS calls use certifi for certificate verification. Build checks must exercise the binary outside the checkout to catch accidental source/environment dependencies. -### Actual output +## Build and local verification -`scripts/build_mac_app.sh` creates `dist/FluentAI.app` by copying the dev Electron app and installing a repo-pinned launcher: - -- Computes `ROOT_DIR` from the repo (`scripts/build_mac_app.sh:4`). -- Copies `node_modules/electron/dist/Electron.app` to `dist/FluentAI.app` (`:5-6`, `:25`). -- Replaces `Contents/Resources/app` with a tiny `package.json` and generated `main.js` (`:26-35`, `:47-54`). -- The generated launcher sets `FLUENTAI_PROJECT_ROOT` to the absolute repo path and `require()`s the repo's `desktop/electron/main.js` (`:47-54`). -- Edits bundle display name/id/version in `Info.plist` (`:56-70`). - -### Build dependencies - -- macOS and `/usr/bin/ditto` (`scripts/build_mac_app.sh:9-10`, `:25`). -- `node_modules/electron/dist/Electron.app`; user must run `npm install` first (`:19-22`). -- `python3` is checked (`:14-17`), but the script actually invokes `python` for generation (`:37`). That mismatch can fail on modern macOS. -- Writable `dist/`. - -### Runtime dependencies - -The produced `.app` is not standalone: - -- It requires the original repo at the exact build-time path because the app launcher imports repo `main.js` (`scripts/build_mac_app.sh:47-54`). -- `main.js` loads repo `preload.js` and `renderer.html` using `__dirname` (`desktop/electron/main.js:82-83`, `:106`). -- `runBridge()` spawns `python -m fluent_ai.desktop_bridge` with `cwd: projectRoot` and `PYTHONPATH: projectRoot` (`desktop/electron/main.js:37-45`). -- Python resolution is `PYTHON_EXECUTABLE`, then repo `.venv/bin/python`, then system `python3` (`desktop/electron/main.js:10-23`). -- The chosen Python must have `openai>=1.0.0` importable (`pyproject.toml:9-10`; `fluent_ai/openai_provider.py:294-301`). -- `OPENAI_API_KEY` is read from env or repo-root `.env` because the bridge cwd is `projectRoot` and `OpenAIProvider.__init__()` calls `load_env_file()` (`fluent_ai/openai_provider.py:24-31`; `fluent_ai/config.py:7-20`). -- State defaults to repo `data/progress.json` (`fluent_ai/desktop_bridge.py:33`, `:384-385`). - -### Consumer readiness - -Current `dist/FluentAI.app` is a local demo launcher, not a consumer app. Moving the repo breaks it; clean Macs lack the repo files, Python environment, dependencies, key path, and repo `data/` state. It also lacks camera/mic usage strings, signing, notarization, and app-support persistence. - -## Recommended Packaging Architecture - -### Option A: `electron-builder` + PyInstaller bridge binary - -Package Electron with `electron-builder`; package `fluent_ai.desktop_bridge` as a standalone PyInstaller executable. In packaged mode, `main.js` launches `process.resourcesPath/bridge/fluentai-bridge`; in dev mode, keep today's `python -m fluent_ai.desktop_bridge`. - -Layout: - -```text -FluentAI.app/Contents/Resources/ - app.asar - bridge/fluentai-bridge -``` - -Pros: - -- Keeps Python lesson, state, provider, and conversation logic intact. -- Removes system Python, `.venv`, repo, and `node_modules` runtime dependencies. -- Preserves the JSON stdin/stdout bridge and visible agent logs. -- Lets existing Python tests keep covering the product core. -- Uses mainstream Electron signing/resource packaging. - -Cons: - -- PyInstaller may need hidden imports/data for `openai`, `httpx`, `pydantic`, certs, and metadata. -- One-file mode can add cold-start latency; one-dir mode may be faster but has more files to sign. -- Universal macOS builds need per-arch bridge builds or a merged universal binary. - -Test impact: - -- Existing `python -m unittest discover -s tests -q` remains the main agent/state test suite. -- Add a bridge-binary smoke test: run `dist-py/fluentai-bridge status` with temp `state_path`. -- Add main-process tests/smoke checks for packaged bridge and state path resolution. - -Recommendation: choose Option A. - -### Option B: `electron-builder` + bundled Python runtime - -Ship `python-build-standalone`, installed `site-packages`, and run bundled `python -m fluent_ai.desktop_bridge`. - -Pros: closest to dev mode, easier Python debugging, fewer PyInstaller import surprises. - -Cons: larger app, many files/dylibs to sign, more path/cert setup, more risk of writing into read-only app resources. - -Test impact: existing Python tests remain, plus integration tests for bundled runtime importability and cert behavior. - -Use only if PyInstaller cannot package the dependency graph reliably. - -### Option C: rewrite bridge in Node - -Port bridge, state migration, lesson/quiz logic, conversation scoring, and provider code to Node. - -Pros: one runtime, no child Python process, potentially smaller. - -Cons: rewrites working core logic, duplicates/invalidates tests, high regression risk, delays packaging. Do not choose for the next standalone app. - -## API Key UX Design - -### First-run screen - -Before auto-starting Lesson Mode, Electron checks key availability. If missing, show a setup panel with: - -- Password input for OpenAI API key. -- `Validate Key` action. -- `Use Dev .env` only when `!app.isPackaged`. -- `Replace key` and `Remove key` in settings after setup. - -States: - -- Missing: "Add your OpenAI API key to start FluentAI." -- Validating: disable actions, run validation ping. -- Invalid: "That key did not validate. Check it and try again." -- Valid: store securely, continue to `status`, then lesson/conversation. - -### Storage and precedence - -Store the key with Electron `safeStorage`: encrypt the plaintext key, persist only a base64 encrypted blob in: - -```text -~/Library/Application Support/FluentAI/settings.json -``` - -Shape: - -```json -{ - "openai_api_key_encrypted": "base64-safeStorage-ciphertext", - "created_at": "iso-time", - "last_validated_at": "iso-time" -} -``` - -If `safeStorage.isEncryptionAvailable()` is false, allow session-only use and do not persist. Do not put plaintext keys in logs, renderer storage, `progress.json`, crash reports, or normal bridge payloads. - -Key precedence: - -1. `OPENAI_API_KEY` already in Electron process env, for CI/demo explicit launches. -2. Packaged secure storage. -3. Dev-only repo `.env` fallback when `!app.isPackaged` or `FLUENTAI_DEV_ENV_FALLBACK=1`. - -Consumer packaged builds must not depend on repo `.env`. - -### Validation and bridge injection - -Add IPC handlers: - -- `key:status` -- `key:validate` -- `key:save` -- `key:delete` - -`key:validate` spawns the bridge with `OPENAI_API_KEY=` and a `validate_key` command that calls `OpenAIProvider.health_check()` (`fluent_ai/openai_provider.py:186-190`) or a cheaper ping. Store only after success. - -For normal commands, main injects the resolved key into the child process env: - -```js -env: { - ...allowlistedEnv, - OPENAI_API_KEY: resolvedKey, - FLUENTAI_STATE_PATH: statePath -} -``` - -Rules: - -- Do not send the key through renderer-visible payloads. -- Redact `sk-...` patterns from bridge `raw` output before returning to renderer. -- Avoid forwarding all `process.env` in packaged mode. - -## State Location - -### Current behavior - -`desktop_bridge.DEFAULT_PROGRESS_PATH` is `data/progress.json` (`fluent_ai/desktop_bridge.py:33`). `_path()` uses payload `state_path` or that default (`:384-385`). `load_state()` creates missing files and migrates valid JSON (`fluent_ai/state.py:77-140`); `save_state()` writes JSON (`:158-163`). - -### Packaged behavior - -Consumer state path: - -```text -~/Library/Application Support/FluentAI/progress.json -``` - -Resolution: - -- If `FLUENTAI_STATE_PATH` is set, use it for tests/demos. -- Else if `app.isPackaged`, use `path.join(app.getPath("userData"), "progress.json")`. -- Else use repo `data/progress.json`. - -Main should inject `state_path` into every bridge payload or set `FLUENTAI_STATE_PATH`; renderer should not know filesystem paths. - -### Migration - -On first packaged launch: - -1. If app-support `progress.json` exists, use it. -2. Else, if `FLUENTAI_PROJECT_ROOT/data/progress.json` or explicit `FLUENTAI_LEGACY_STATE_PATH` exists, copy it. -3. Log only `[Memory Agent] Migrated learner profile to Application Support.` -4. If no legacy state exists, let `load_state()` create default state. - -Do not scan the filesystem. If JSON is invalid, move it to `progress.corrupt..json`, start a fresh profile, and tell the user a backup was saved. - -## Reliability Matrix - -| Failure | Current behavior | Designed behavior | -| --- | --- | --- | -| Missing key | `OpenAIProvider.status()` reports unset key (`fluent_ai/openai_provider.py:38-43`). Lesson/conversation return `_openai_required()` (`fluent_ai/desktop_bridge.py:105-111`, `:169-176`, `:388-394`). Renderer shows generic lesson/text errors or realtime secret failure (`desktop/electron/renderer.html:2406-2418`, `:2675-2678`, `:1992-2001`). | First-run key gate. Message: "Add your OpenAI API key to start FluentAI." Recovery: validate and store key; dev may use `.env`. | -| Expired realtime session | Client secret is requested for 600s (`fluent_ai/openai_provider.py:66-68`). Renderer does not track expiry (`desktop/electron/renderer.html:1992-2058`). | Track `expires_at`; refresh at T-60s. Message: "Voice session is refreshing." Recovery: reconnect or continue in text with transcript preserved. | -| Microphone blocked | Main checks/asks macOS media access and returns blocked error (`desktop/electron/main.js:158-186`). Renderer maps permission errors (`desktop/electron/renderer.html:1964-2075`). | Keep detection; add "Open System Settings"; disable voice until fixed; offer text mode. | -| Camera blocked | Main returns combined mic/camera error (`desktop/electron/main.js:158-186`). Camera toggle shows raw error message (`desktop/electron/renderer.html:2316-2340`). | Split camera from mic. Message: "Camera is blocked. Continue voice-only or allow camera." Recovery: switch video off, keep call active. | -| Weak network | Provider timeouts: realtime 20s, vision 12s, SDK 30s (`fluent_ai/openai_provider.py:92-99`, `:158-165`, `:303-327`). Realtime `fetch()` has no explicit timeout (`desktop/electron/renderer.html:2032-2044`). | Add `AbortController`, retry once, and offline/weak-network banner. Preserve lesson draft/call transcript. | -| Model timeout | `_text_response()` catches SDK errors, stores safe error, returns empty (`fluent_ai/openai_provider.py:321-327`, `:389-393`). Bridge often reports generation failure (`fluent_ai/desktop_bridge.py:105-111`, `:173-176`). | Distinguish timeout from missing key. Message: "The model timed out. Your progress is safe." Recovery: retry same request, optional lower-latency model. | -| Malformed model output | Lesson JSON parse failure returns base lesson, then bridge rejects missing `source=openai` (`fluent_ai/openai_provider.py:224-238`; `fluent_ai/desktop_bridge.py:108-111`). Vision has a generic fallback (`fluent_ai/openai_provider.py:167-184`). | Schema-validate, retry once with repair prompt, then show malformed-response message and leave state unchanged. | -| Empty tutor response | Empty conversation text returns `None` (`fluent_ai/openai_provider.py:288-292`). Start errors (`fluent_ai/desktop_bridge.py:173-176`). Reply currently can save progress and return `tutor_message: null` because it does not guard after `_tutor_reply()` (`:238-253`). | Never render/persist empty tutor output. Retry once; if still empty, use a short recovery prompt and log the recovery. | -| Invalid progress file | `json.load()` has no corruption handling (`fluent_ai/state.py:77-86`). Bridge catches and returns `JSONDecodeError` (`fluent_ai/desktop_bridge.py:421-426`). | Backup corrupt file, create fresh state, show "Your progress file was damaged; a backup was saved." | -| Interrupted lesson | Generated lesson/quiz are in renderer memory only (`desktop/electron/renderer.html:2406-2424`). State saves only on quiz submit (`fluent_ai/desktop_bridge.py:130-161`). | Save `sessions/current_lesson.json` after generation and during answers. Resume/discard on next launch. | -| Interrupted call | `beforeunload` ends the call (`desktop/electron/renderer.html:2811-2813`). `endRealtimeCall()` clears in-memory call state (`:2230-2255`). Voice transcripts are not saved to progress; text saves after reply (`fluent_ai/desktop_bridge.py:242-257`). | Checkpoint transcript/metadata every turn. On drop, offer reconnect or text. Summarize checkpoint through bridge to update memory. | -| Restart mid-session | Renderer state is in memory (`desktop/electron/renderer.html:1693-1714`); persisted state changes only after quiz submit/text reply. | Read session checkpoint files on startup and show Resume/Discard before auto-starting a new lesson. Avoid duplicate progress updates. | - -## Build Pipeline - -### Commands and scripts - -Install build tools: +From the repository root on macOS: ```bash -npm install --save-dev electron-builder -python -m pip install pyinstaller -``` - -New npm scripts: - -```json -{ - "package:bridge": "python -m PyInstaller packaging/fluentai_bridge.spec --noconfirm --clean", - "package:mac": "npm run package:bridge && electron-builder --mac dir", - "dist:mac": "npm run package:bridge && electron-builder --mac dmg" -} -``` - -Bridge build command shape: - -```bash -python -m PyInstaller \ - --name fluentai-bridge \ - --onefile \ - --collect-all openai \ - --collect-all httpx \ - --collect-all pydantic \ - packaging/bridge_entry.py -``` - -`packaging/bridge_entry.py` should only call `fluent_ai.desktop_bridge.main()`. Prefer a `.spec` file once hidden imports/certs are known. - -### Electron builder config - -Add config in `package.json` or `electron-builder.yml`: - -```yaml -appId: local.fluentai.desktop -productName: FluentAI -files: - - desktop/electron/** - - package.json -extraResources: - - from: dist-py/fluentai-bridge - to: bridge/fluentai-bridge -mac: - target: [dir, dmg] - category: public.app-category.education - hardenedRuntime: false - gatekeeperAssess: false - extendInfo: - NSMicrophoneUsageDescription: FluentAI uses the microphone for live language tutoring calls. - NSCameraUsageDescription: FluentAI uses the camera only when you enable video context for tutoring. +python3 -m venv .venv +source .venv/bin/activate +python -m pip install -e '.[dev]' +npm ci +npm run check +npm run build:app ``` -### Signing and notarization +`build:app` invokes `package:mac`, which runs `scripts/build_bridge.sh` followed by Electron Builder. The bridge script requires `.venv/bin/python`, installs PyInstaller if missing, and checks standalone status, missing-key behavior, and SDK import using temporary state outside the checkout. No paid API call is needed for these build smokes. -No paid Apple Developer ID for now: +On Apple Silicon the app normally appears at `dist/mac-arm64/FluentAI.app`; other architectures can use `dist/mac/FluentAI.app`. Use the actual output directory reported by Electron Builder. `npm run sign:mac` applies ad-hoc signing to these paths. To check the result: ```bash -npm run package:mac -codesign --force --deep --sign - "dist/mac/FluentAI.app" -open "dist/mac/FluentAI.app" +codesign --verify --deep --strict dist/mac-arm64/FluentAI.app +open dist/mac-arm64/FluentAI.app ``` -Ad-hoc signing helps local launch but is not notarization. Downloaded/shared builds may still hit Gatekeeper quarantine; personal users may need right-click Open or `xattr -dr com.apple.quarantine FluentAI.app`. Public distribution later needs Developer ID signing, hardened runtime, `xcrun notarytool`, and stapling. - -Size expectations: - -- Option A: about 230-350 MB compressed DMG. -- Option B: about 300-450 MB. -- Option C: closer to Electron-only size, but too risky now. - -## Sequenced Implementation Plan - -### 1. Packaging path abstraction +Local/ad-hoc signing is not Developer ID signing or notarization. Hardened runtime and Gatekeeper assessment are currently disabled in the builder configuration. Public distribution still needs a deliberate signing/notarization pipeline and clean-account acceptance. -Files: `desktop/electron/main.js`, `fluent_ai/desktop_bridge.py`, tests. +## State location and migration -Work: add dev/package bridge resolution, app-support state resolution, and main-side `state_path` injection. +Development defaults to `data/progress.json`. Packaged Electron uses `app.getPath("userData")/progress.json` (normally `~/Library/Application Support/FluentAI/`). `FLUENTAI_STATE_PATH` overrides the file and `FLUENTAI_USER_DATA_PATH` can isolate a test profile. -Tests: dev path uses repo `data/progress.json`; packaged-like env uses temp app-support path. +First packaged launch copies a legacy profile only when the destination is missing and an explicit `FLUENTAI_LEGACY_STATE_PATH`, or `FLUENTAI_PROJECT_ROOT` containing `data/progress.json`, identifies the source. It does not search arbitrary directories. Corrupt-state backup/recovery is handled by the Python state layer. -Acceptance: `npm run check` green; memory logs still show load/save. - -### 2. State migration and corruption recovery - -Files: `desktop/electron/main.js`, `fluent_ai/state.py`, `fluent_ai/desktop_bridge.py`, tests. - -Work: copy legacy `data/progress.json` once, backup corrupt JSON, return recovery metadata. - -Tests: missing state creates default; legacy copies once; invalid JSON backs up and recovers. - -Acceptance: packaged state writes to `~/Library/Application Support/FluentAI/`; dev/tests keep repo or temp paths. - -### 3. API key manager - -Files: `desktop/electron/main.js`, `preload.js`, `renderer.html`, `fluent_ai/desktop_bridge.py`, `openai_provider.py`, tests. - -Work: add safeStorage key IPC, first-run UI, `validate_key`, env injection, and raw-output redaction. - -Tests: mocked validation, no secret in logs/raw/progress, dev `.env` fallback only in dev. - -Acceptance: no-key app shows setup; valid key enables lesson/call; no secret is printed. - -### 4. PyInstaller bridge build - -Files: `packaging/bridge_entry.py`, `packaging/fluentai_bridge.spec`, `package.json`. - -Work: build `dist-py/fluentai-bridge` with OpenAI dependency graph and certs. - -Tests: binary `status` with temp `state_path`; binary missing-key response is clean. - -Acceptance: bridge runs without `.venv`; `npm run check` green. - -### 5. Electron Builder packaging - -Files: `package.json` or `electron-builder.yml`, optional `build/entitlements.mac.plist`, `desktop/electron/main.js`. - -Work: include bridge in `extraResources`, add usage descriptions, add ad-hoc signing flow. - -Tests: `npm run package:mac`; `codesign --verify --deep --strict dist/mac/FluentAI.app`. - -Acceptance: `open dist/mac/FluentAI.app` launches. - -### 6. Reliability pass - -Files: Electron files, `desktop_bridge.py`, `openai_provider.py`, `state.py`, tests. - -Work: add fetch/bridge timeouts, retry paths, lesson/call checkpoints, empty-response recovery, malformed-output repair, and camera voice-only fallback. - -Tests: missing key, invalid progress, empty tutor, timeout, malformed output, interrupted lesson, call disconnect. - -Acceptance: every Reliability Ideal row has a message and recovery path; no learner state loss in simulations. - -### 7. Clean-account acceptance - -Commands: - -```bash -npm run check -npm run package:mac -codesign --force --deep --sign - "dist/mac/FluentAI.app" -open "dist/mac/FluentAI.app" -``` +## API keys -Manual checks: +The Electron main process resolves keys in this order: `OPENAI_API_KEY`, session key, encrypted stored key, then development-only `.env`. Packaged operation must not depend on a repository `.env`. -- Clean macOS user account. -- Repo moved aside or inaccessible. -- Double-click launches app. -- First-run key entry validates and stores key. -- Lesson starts, quiz submit updates app-support `progress.json`. -- Conversation starts voice-only. -- Camera denial falls back to voice-only. -- Quit mid-lesson shows resume on next launch. +Keys can be validated through the bridge and persisted using Electron `safeStorage` in `settings.json` under user data. If encrypted storage is unavailable, the app keeps the key for the session. Keys are injected into the bridge environment; packaged subprocesses receive a restricted environment plus `FLUENTAI_*` overrides. Main/bridge errors redact secrets rather than returning raw credentials to the renderer or learner memory. -Final acceptance: +## Manual acceptance still required -- `npm run check` stays green. -- `open dist/FluentAI.app` launches on a clean user account without the repo. -- Consumer mode requires no `.env`, repo path, terminal, or printed secret value. +Use a clean macOS account with the repository inaccessible and an isolated profile. Verify double-click launch, first-run key entry, lesson submission and restart/resume, voice-only conversation, camera-denial recovery, navigation/language-change/window-close finalization, and export/delete controls. Mocked tests, bridge smokes, and a successful bundle build do not establish those hardware/window behaviors. Prior acceptance evidence and its limits are recorded in [notes](../notes.md). diff --git a/macos/desktop_app.py b/macos/desktop_app.py deleted file mode 100644 index e21b076..0000000 --- a/macos/desktop_app.py +++ /dev/null @@ -1,105 +0,0 @@ -#!/usr/bin/env python3 -"""Tiny native fallback launcher for FluentAI on macOS. - -The primary desktop app is Electron. This Tk fallback keeps the original one-shot -MVP contract alive on machines where a Python-only window is preferable during -local demos. It calls the same desktop bridge functions as the Electron shell and -never requires OpenAI credentials. -""" -from __future__ import annotations - -import tkinter as tk -from pathlib import Path -from tkinter.scrolledtext import ScrolledText - -from fluent_ai.desktop_bridge import conversation_start, lesson_start, lesson_submit, status - -ROOT = Path(__file__).resolve().parents[1] -STATE_PATH = ROOT / "data" / "progress.json" - - -class FluentAIDesktop(tk.Tk): - def __init__(self) -> None: - super().__init__() - self.title("FluentAI") - self.configure(bg="#fff5f5") - self.attributes("-fullscreen", True) - self.bind("", lambda _event: self.attributes("-fullscreen", False)) - self.bind("", lambda _event: self.destroy()) - self.geometry("1180x760") - - self.language = tk.StringVar(value="Spanish") - self.turns = tk.IntVar(value=4) - self.video = tk.StringVar(value="off") - self.visible_object = tk.StringVar(value="apple") - - self._build_ui() - self.refresh_status() - - def payload(self) -> dict: - return {"state_path": str(STATE_PATH), "language": self.language.get()} - - def _build_ui(self) -> None: - header = tk.Frame(self, bg="#ef4444", padx=18, pady=14) - header.pack(fill="x") - tk.Label(header, text="FluentAI", bg="#ef4444", fg="white", font=("Helvetica", 26, "bold")).pack(side="left") - self.status_label = tk.Label(header, text="Loading...", bg="#ef4444", fg="white", font=("Helvetica", 13)) - self.status_label.pack(side="right") - - controls = tk.Frame(self, bg="#fff5f5", padx=18, pady=12) - controls.pack(fill="x") - tk.Label(controls, text="Language", bg="#fff5f5").grid(row=0, column=0, sticky="w") - tk.OptionMenu(controls, self.language, "Spanish", "Hindi", "French").grid(row=1, column=0, sticky="ew", padx=(0, 10)) - tk.Label(controls, text="Turns", bg="#fff5f5").grid(row=0, column=1, sticky="w") - tk.Spinbox(controls, from_=2, to=8, textvariable=self.turns, width=8).grid(row=1, column=1, padx=(0, 10)) - tk.Label(controls, text="Video", bg="#fff5f5").grid(row=0, column=2, sticky="w") - tk.OptionMenu(controls, self.video, "off", "on").grid(row=1, column=2, sticky="ew", padx=(0, 10)) - tk.Label(controls, text="Visible object", bg="#fff5f5").grid(row=0, column=3, sticky="w") - tk.Entry(controls, textvariable=self.visible_object, width=18).grid(row=1, column=3, padx=(0, 10)) - tk.Button(controls, text="Run Lesson", command=self.run_lesson, bg="#ef4444", fg="white").grid(row=1, column=4, padx=6) - tk.Button(controls, text="Start Conversation", command=self.run_conversation, bg="#f59e0b").grid(row=1, column=5, padx=6) - tk.Button(controls, text="Refresh", command=self.refresh_status).grid(row=1, column=6, padx=6) - - self.output = ScrolledText(self, wrap="word", font=("Menlo", 13), padx=14, pady=14) - self.output.pack(fill="both", expand=True, padx=18, pady=(0, 18)) - self.output.insert("end", "Press Run Lesson or Start Conversation. Escape leaves fullscreen. Command-Q quits.\n") - - def refresh_status(self) -> None: - result = status(self.payload()) - profile = result.get("profile", {}) - self.status_label.config( - text=f"{profile.get('language', 'Spanish')} · {profile.get('level', 'A1')} · XP {profile.get('xp', 0)} · OpenAI required" - ) - - def run_lesson(self) -> None: - start = lesson_start(self.payload()) - answers = [question.get("answer", "") for question in start.get("quiz", [])] - result = lesson_submit({**self.payload(), "lesson": start.get("lesson"), "quiz": start.get("quiz"), "answers": answers}) - self._write("\n".join(start.get("logs", []) + result.get("logs", []))) - self._write(f"\nScore: {result.get('summary', {}).get('score')}\n{result.get('summary', {}).get('report')}\n") - self.refresh_status() - - def run_conversation(self) -> None: - result = conversation_start( - { - **self.payload(), - "turns": self.turns.get(), - "video": self.video.get(), - "object": self.visible_object.get(), - } - ) - self._write("\n".join(result.get("logs", []))) - self._write(f"\nTutor: {result.get('tutor_message')}\n") - self.refresh_status() - - def _write(self, text: str) -> None: - self.output.insert("end", text + "\n") - self.output.see("end") - - -def main() -> None: - FluentAIDesktop().mainloop() - - -if __name__ == "__main__": - main() diff --git a/notes.md b/notes.md index 1d9d647..bf76c6c 100644 --- a/notes.md +++ b/notes.md @@ -1,62 +1,26 @@ -# Notes - -- 2026-05-17: Workspace started empty. -- Product direction: agentic language learning app with an always-on tutor, local learner state, autonomous daily lessons/quizzes, answer evaluation, progress updates, and visible 5-10 minute demo loop. -- Implementation choice: start with a small Python CLI loop and deterministic fallback logic so the app runs without external credentials. -- Goal-mode instruction added: keep `plan.md` current, record real progress and durable decisions here, and adapt calmly if `plan.md` changes mid-run. -- Built initial Python MVP: local state model, adaptive lesson/quiz engine, auto and interactive CLI modes, bounded demo loop, and a standard-library smoke test. -- Upgraded MVP requirements: `data/progress.json` is now canonical memory; lessons include vocabulary, grammar, examples; quizzes produce 5-8 mixed question types; console logs show Memory, Lesson Generator, Notification, Adaptive Quiz, Evaluator, Progress Reporter, and Orchestrator agents. -- Added optional `fluent_ai.ui` Gradio entry point that falls back to console instructions if Gradio is not installed. -- New product direction: add Conversation Mode, a FaceTime-like AI tutor that initiates conversation, adapts topic complexity to measured ability, supports video on/off, and can use camera input such as pointing at an apple to continue the target-language conversation. -- Built Conversation Mode MVP: CLI supports `--product-mode conversation`, video on/off, visible object labels such as `apple`, adaptive topic selection, tutor-initiated turns, fluency feedback, and `conversation_memory` updates in `data/progress.json`. -- Added `.env` loading and OpenAI Responses API support. Verified `.env` is present, OpenAI status reports enabled with `gpt-5.5`, health check passed, API-backed lesson enhancement worked, and API-backed apple conversation worked. -- Added no-dependency local web app at `python -m fluent_ai.web --port 7860`; verified browser UI runs Lesson Mode and Conversation Mode with video on and visible object `apple`. -- Added standalone fullscreen Electron desktop app launched by `npm run desktop`; it runs outside the browser and calls the existing Python/OpenAI agents. -- 2026-05-17: User asked for clean Duolingo-inspired UI mockups before changing the real app. Added `mockups/duolingo-inspired/index.html` with three selectable concepts: Learning Path Home, Tutor Call First, and Demo Studio. Added Electron screenshot helper and rendered PNGs for all three concepts. Do not apply a final UI direction until the user chooses one. -- 2026-05-17: Fixed desktop MVP after user reported the app felt non-interactive. Added `fluent_ai.desktop_bridge`, Electron IPC bridge calls, real lesson quiz submission, turn-by-turn Conversation Mode replies, visible feedback, and logs. -- Verified desktop window through Electron debug context: Conversation Mode accepted "Me llamo Johan.", produced feedback and a follow-up; Lesson Mode accepted quiz answers, graded them, and updated `data/progress.json`. -- 2026-05-17: User asked to switch the theme to red and apply it. Updated mockups to a red Duolingo-inspired palette, regenerated mockup screenshots, rethemed the richer Electron UI in `desktop/electron/renderer.html`, rethemed the browser fallback in `fluent_ai/web.py`, and updated the native Tk Mac app source plus bundled `dist/FluentAI.app` resource. Verified unit tests, Python compile, localhost status, and browser render. -- 2026-05-17: Fixed Electron window lifetime by keeping a global BrowserWindow reference and rebuilt `dist/FluentAI.app` as a real Electron app copy instead of the old Tk/Wish wrapper. -- 2026-05-17: Added model selector: default fast local tutor, optional OpenAI model. Live renderer test verified OpenAI conversation and lesson paths use `gpt-5.5` from `.env`. -- 2026-05-17: Added real camera preview request for Video On. Live renderer test showed `Camera On` with video element and object context `apple`; conversation used visible object `manzana`. -- 2026-05-17: Live renderer test results: app starts with zero conversation bubbles, OpenAI tutor accepts learner replies, quiz submits and updates `data/progress.json`. Screenshots saved in `/tmp/fluentai_test_conversation_openai.png` and `/tmp/fluentai_test_lesson_openai.png`. -- 2026-05-17: User asked to confirm the approved design was truly implemented and usable. Found the browser route was still serving the older two-column UI. Fixed it by serving the same red Demo Studio renderer used by Electron and adding `/api/bridge/*` web endpoints. Updated the live renderer closer to the approved Demo Studio layout: mission/sidebar, large Lesson/Conversation mode cards, main work panel, and dark visible-agent-decision rail. Verified web status, lesson generation, quiz submission, conversation start, video-object fallback, chat reply, no browser console errors, and 7 unit tests. Live verification screenshot saved at `mockups/duolingo-inspired/live-app-verified.png`. -- 2026-05-17: User asked for a more polished Lesson UI and voice mode by default for Conversation. Updated `desktop/electron/renderer.html`: Lesson Mode now has a red hero card, metrics, polished vocabulary/example cards, card-style quiz choices, free-answer styling, and improved quiz summary. Conversation Mode now defaults to Voice tutor on, speaks tutor messages with browser speech synthesis, offers a Speak button using SpeechRecognition/webkitSpeechRecognition when available, attempts voice listening after tutor prompts, and keeps typed replies as fallback. Verified tests, browser render, polished lesson DOM, voice default state, conversation bridge start/reply, and saved screenshot `mockups/duolingo-inspired/polished-lesson-voice-ui.png`. -- 2026-05-17: User clarified Conversation Mode must use an OpenAI voice model, not browser speech synthesis, and video/voice should live in the main chat area like the initial call mockup. Replaced the browser speech hooks with an OpenAI Realtime WebRTC client-secret bridge, added a call-style main stage with Call/Mute/Video/End controls, kept typed chat as Text fallback, and made video context update the main call surface. Video can be selected before a call or toggled into the main stage during an active call. Verified Realtime client-secret creation with `gpt-realtime`/`alloy`, browser UI layout, text fallback conversation, no browser console errors, JS syntax checks, and 7 unit tests. -- 2026-05-17: User reported a voice-call error and asked to use the app first. Reproduced the browser-runner failure as microphone permission denial. Fixed Electron media permission handling with both permission-check and permission-request handlers, added a macOS media access IPC bridge, moved mic/camera acquisition before Realtime session creation, and added clear blocked-mic UI copy. Verified the actual Electron app with fake media via remote debugging: Realtime call connected, tutor spoke through `gpt-realtime`, transcript appeared, video toggled into the main stage, mute changed state, and End returned controls to reconnect. -- 2026-05-17: User rejected the typed visible-object field and asked for OpenAI to process video. Removed the visible-object UI from the Electron/browser renderer, added `vision_analyze_frame` bridge command, and implemented OpenAI Responses API image input analysis from a captured camera frame. Video calls now analyze a frame, show OpenAI vision context in the call card, and pass that context into the `gpt-realtime` tutor. Verified in Electron with fake camera/mic: no object input, OpenAI vision saw the fake green camera scene, Realtime tutor used that context in Spanish, video was in the main stage, and the call ended cleanly. -- 2026-05-17: User reported calls failing midway, video not appearing, green shape artifacts, and text fallback not feeling like a real chat. Fixed the Electron renderer so Lesson Mode auto-generates with no setup/top workspace bar, Conversation Mode fills the workspace, OpenAI voice is forced as the startup default, text fallback becomes a full chat interface with a clear return-to-voice control, and video is selected before call startup instead of racing an audio-only call. Removed the green visual accent from the call stage and cleaned stale `green shape` memory from `data/progress.json`. -- 2026-05-17: Reworked call startup so the video feed renders immediately, OpenAI Realtime connects first, and OpenAI vision analysis runs only after the voice call is live or when video is toggled into an active call. Verified with the actual Electron app plus fake mic/camera over remote debugging: auto lesson generated, quiz submitted and rendered feedback, Conversation filled the screen, text fallback sent/received messages, return-to-voice worked, video appeared in the main call stage, Realtime reached `Live`, mute toggled, and End reset controls. Rebuilt `dist/FluentAI.app`. -- 2026-05-17: User clarified the green shape must not be treated as camera content; video should be the real camera feed and the tutor should respond to what it sees. Added media diagnostics for fake-camera mode, explicit real-camera constraints, live recurring OpenAI vision sampling while video is on, synthetic test-feed filtering so fake green patterns are labeled and not sent as tutor context, and a vision prompt guard against inventing objects from test patterns. Verified Electron with fake media: app labels the feed as `Test camera feed`, video still renders, Realtime reaches `Live`, and controls reset. Rebuilt `dist/FluentAI.app`. -- 2026-05-17: User asked to remove the visible `OpenAI enabled with model gpt-5.5` text and make the tutor more responsive when the learner asks for English meaning. Header now shows only level and weak topics; provider status now says `OpenAI connected.` instead of naming the model. Realtime and text tutor prompts now explicitly answer comprehension/English-help requests in English first, then return to one simple Spanish model phrase. Added a unit test for the English-help fallback and rebuilt `dist/FluentAI.app`. -- 2026-05-17: Initialized Git, committed the FluentAI project, created public GitHub repo `Jo2234/FluentAI`, and pushed `main`. `.env`, `node_modules/`, and `dist/` are ignored. -- 2026-05-17: Added first-class Hindi, Spanish, and French selection to the desktop/web renderer. The selected language is persisted in local learner state, lessons/quizzes use language-specific content, and text/Realtime/vision prompts now tell the tutor to respond in the language the learner uses when it is Hindi, Spanish, or French, with English explanations for comprehension help. -- 2026-05-17: Made the agent decision log collapsible in the shared desktop/web renderer. The panel stays visible by default for demo transparency, but users can hide it and give the lesson area more room. -- 2026-06-30: Upgraded project hygiene for a stronger public/demo repo: fixed editable packaging with explicit setuptools package discovery, added Python/Node CI, added offline smoke script, added web endpoint smoke test, hardened web JSON parsing/turn bounds, made progress JSON preserve non-ASCII target-language text, and refreshed README quickstart/test/demo docs. -- 2026-06-30: User asked to remove deterministic fallback. Real CLI/web/desktop Lesson and Conversation Mode now require `OPENAI_API_KEY`; tests and CI use mocked OpenAI providers for reliability without secrets. -- 2026-06-30: User said Realtime tutor cuts them off / talks too soon and should know when to wait, prompt, or explain in English. Added natural turn policy, server VAD silence/idle tuning, no-interrupt setting, language-aware help behavior, stopped camera-context updates from forcing immediate speech, and added regression tests. -- 2026-07-03: Daily product polish finished prior voice/camera work: dynamic Realtime VAD now adapts silence duration by learner level and speaking confidence, camera-frame analysis defaults to fast `gpt-4.1-mini` with uncertainty guardrails and shorter timeout, video context refreshes more often and shows model/confidence, and the agent decision log starts collapsed to reduce conversation clutter. -- 2026-07-04: Added spaced repetition memory for Lesson Mode. Quiz results now maintain a `review_queue` in local learner state, due reviews override recent-topic rotation so missed topics come back first, and the desktop/web profile surfaces review count/next review topic. Smoke tests now verify review scheduling. -- 2026-07-05: Daily product polish clarified spaced-review status. Profiles now distinguish reviews due now from the total future schedule, include the next due timestamp, and the desktop/web header says `Reviews N due` only when immediate practice is needed. -- 2026-07-06: Daily product polish localized Conversation Mode scaffolds for French and Hindi. Beginner topic openings, help model phrases, and known visible-object prompts now use the selected target language instead of Spanish-only fallback text; `scripts/smoke_demo.py` also runs directly from the README command without needing `PYTHONPATH`. -- 2026-07-07: Daily product polish localized the remaining Conversation Mode fallback follow-ups and visual auto-replies for French/Hindi, so mocked/offline tutor flows no longer drift back into Spanish after the opening turn. -- 2026-07-08: User requested a substantial UI overhaul after seeing excess empty space, a floating square agent-decision panel, basic lessons, slow quiz feedback, unreadable/generated-looking colors, and Conversation controls overlapping the visual-context card. Updated the shared desktop/web renderer with a full-bleed aurora studio aesthetic, a tall side-rail agent log that pushes content left when expanded, sticky Lesson learning rail, Phrase Lab, per-question answered/correct/review status chips, separated call context/controls, and higher-contrast text/fallback controls. -- 2026-07-08: Synced local `main` to GitHub `origin/main` commit `63a3514` (`Overhaul FluentAI lesson and call UI`), stashed local progress/notes edits as `stash@{0}`, refreshed Python/Node installs, verified 28 Python tests, smoke demo, and `npm run check`, then restarted the web app at `http://127.0.0.1:7860`. -- 2026-07-08: Added `docs/FLUENTAI_LIMITLESS_IDEAL.md` as the product north-star spec for the ideal unbounded FluentAI: a living, memory-driven, agentic language tutor connecting Lesson Mode, Conversation Mode, voice, video, local learner memory, agent logs, privacy, tests, and roadmap. -- 2026-07-08: Implemented state v2 WP2/WP3 migration: lesson, conversation, bridge, web, CLI, provider, and smoke call sites now use v2 helpers/direct language blocks; compat mirrors were removed from `state.py`; event IDs now use persistent `event_counter`; added tests for typed events, v2 review queue, bridge language isolation, bridge delegation, and post-cap event ID uniqueness. -- 2026-07-08: Fixed WP2/WP3 language-switch persistence regression: `load_state` no longer overrides active language for existing files; bridge language switches persist only when payloads explicitly include `language`; persisted-file bridge isolation now covers French/no-language/Spanish switching. -- 2026-07-08: Started WP4/WP5 implementation. Constraints: do not touch `data/progress.json`, preserve existing topic-selection semantics except inserting due mistake memory after due review and before weak topics, keep OpenAI required for real Lesson/Conversation while tests use mocked providers. -- 2026-07-08: Completed WP4/WP5. Lessons now carry `reason` from due review, due mistake memory, weak topic, or rotation; desktop/web/CLI surfaces show the reason. Quiz results now include deterministic error categories, corrected forms, severity, and confidence; desktop lesson submit can use mocked/OpenAI grading for fill-blank/open-ended answers with local fallback. Lesson misses call `record_mistake`, schedule `next_review`, and can drive the next lesson before weak-topic selection. -- 2026-07-08: Completed WP6/WP7. Lesson misses now set `conversation_memory.next_conversation_goal` and clean lessons clear stale lesson-sourced goals. Conversation topic selection consumes goals unless video context wins; tutor prompts and realtime instructions include goal guidance. Conversation corrections now store real learner text, map conversation topics to teachable lesson-bank topics (`likes and food`/`weather`/visible object -> `vocabulary`, `past weekend` -> `past tense`, direct lesson topics unchanged), and schedule review. Text fallback, voice `conversation_end`, web, and CLI now persist capped post-call summaries. -- 2026-07-08: Implemented WP4 Home workspace and privacy controls: `home_summary`, sanitized memory inspect/export, language reset, delete-all memory, Electron save-dialog export, and home-first renderer flow. -- 2026-07-08: Packaging steps 2/3 in progress: added Electron API key gate/IPC, safeStorage or session-only key handling, bridge `validate_key`, secret redaction, and corrupt `progress.json` backup/recovery notice path. -- 2026-07-08: Completed Packaging step 6 reliability pass: bridge session checkpoints, empty tutor retry/recovery, model-failure copy, renderer fetch timeouts/retry policy, realtime refresh scaffolding/fallback, camera voice-only fallback, tests, and live interrupted-lesson resume verification. -- 2026-07-08: Consumer acceptance run attempted against packaged `dist/mac-arm64/FluentAI.app` at `cf9c4d5` with isolated `FLUENTAI_USER_DATA_PATH` and `FLUENTAI_FAKE_MEDIA=1`. Host GUI capture/control was blocked (`computer-use` timeout/AX cannotComplete, `screencapture` could not create image, System Events hung, packaged Electron CDP endpoint hung), so true screenshot-based GUI acceptance is not complete. Packaged bridge fallback verified onboarding, placement, home summary, lesson feedback/adaptation, checkpoint resume, and schema-v2/no-secret state in temp profile. Defects found: Realtime client secret failed in packaged bridge with Python SSL certificate verification error; text tutor visibly corrected `yo estudiar por la manana` but structured turn metadata/mistake memory did not record that conversation mistake. -- 2026-07-08: Fixed the two acceptance-run defects. Direct provider HTTPS calls now use a certifi SSL context inside the PyInstaller bridge. Conversation turns can use one strict provider JSON grade per learner reply, falling back to the local keyword scorer on malformed output; provider correction forms/categories now flow through existing mistake memory and post-call summary paths. Verified `python3 -m pytest` (111 passed), `npm run check`, `./scripts/build_bridge.sh`, packaged `realtime_client_secret` with real `.env` key redacted (`ok`, model `gpt-realtime`, secret present), and packaged `vision_analyze_frame` no-key clean error. -- 2026-07-08: Fixed conversation-to-lesson mistake scheduling. Conversation corrections now make mistake-memory reviews immediately due; lesson-origin misses still schedule next review for +1 day. Successful mistake-memory lessons push the practiced mistake topic out by +1 day so the same due mistake does not loop forever. -- 2026-07-08: Final polish added phrase audio playback through cached OpenAI TTS, lesson pronunciation/culture cards, OpenAI enhancement context preservation for those fields, and post-call pronunciation practice notes; verified `npm run check` and one real `phrase_audio` call (`ok`, 36096 bytes). -- 2026-07-08: Fixed Lesson Mode quiz contrast in the aurora renderer. Multiple-choice labels now render light text on dark cards with subtle hover and indigo selected states; prompts, placeholders, feedback, and lesson Try It/listen controls were swept for dark-on-dark. Verified `python -m unittest discover -s tests -q`, `npm run test:js`, and a temp-state live web drive measuring a generated choice label at `rgb(238, 242, 255)` on `rgba(15, 16, 22, 0.78)`. -- 2026-07-08: Daily product polish made practice streaks real instead of static profile copy. Lesson and Conversation Mode now update language-specific `streak_days`, `last_practice_date`, `last_session_at`, and reset same-language daily counters when a new UTC practice day starts. -- 2026-09-06: Implemented approved audit fixes. State writes now use a shared process lock, short reload/apply transactions and atomic replacement; stale snapshots/reset generations cannot overwrite newer progress. Delete All Memory removes managed checkpoints, speech cache and recovery files, cancels renderer work and invalidates in-flight sessions. -- 2026-09-06: Lesson and voice completion carry stable session identity and cache committed results; duplicate quiz requests grade/award once. Voice navigation, language changes and window close share a retryable finalizer with captured language and durable checkpoints. Camera samples are checked before inference, with change/manual triggers and a 30-second refresh bound. -- 2026-09-06: Verified 131 Python tests without skips using real Python/OpenAI/certifi, 11 renderer VM scenarios, existing JS checks, Ruff and mocked smoke. Built and installed the wheel outside the source checkout; Spanish/French/Hindi curricula and transactional status smoke pass. Added missing curriculum package data and enabled renderer regression tests in CI. Physical camera/packaged window lifecycle remains a manual acceptance check; no paid API calls were used. +# Project decisions and historical evidence + +Current behavior is documented in [architecture](docs/GAP_AND_DESIGN.md), [curriculum](docs/CURRICULUM_DESIGN.md), [privacy](docs/ONBOARDING_HOME_DESIGN.md), and [packaging](docs/PACKAGING_DESIGN.md). Pending work lives in the [roadmap](docs/FLUENTAI_LIMITLESS_IDEAL.md). + +## Durable decisions + +- The project began with a CLI demo and evolved into a shared Electron/browser tutor. The early deterministic user-facing fallback was removed on 2026-06-30; real lesson/conversation runs require OpenAI, while tests mock providers. +- The user selected the red Duolingo-inspired Demo Studio direction in May, followed by an aurora UI overhaul in July. Retain the original mockups and screenshots in `mockups/duolingo-inspired/` as design provenance; they are not all screenshots of today's UI. +- Per-language schema v2 replaced flat memory; lesson and conversation mistakes share review scheduling. Avoid reintroducing compatibility mirrors or overwriting one language during another's session. +- Keep demo learner data intact during development. Use temporary files/profiles for tests and paid API experiments. + +## Historical validation, not a fresh acceptance claim + +- July 2026 packaging acceptance was attempted with isolated user data and fake media. Host GUI capture/control failed, so full screenshot-based clean-account acceptance was incomplete. Bridge checks exercised onboarding, placement, home, lesson adaptation, checkpoint resume, and secret-safe state. Certificate verification and structured conversation mistake recording defects found in that run were fixed. +- The 2026-09-06 audit recorded 131 Python tests, 11 renderer VM scenarios, JS checks, Ruff, and mocked smoke passing. A wheel installed outside the checkout loaded all three curricula and passed transactional status smoke. No paid API calls were used for that audit. Physical camera and packaged-window lifecycle acceptance remained manual. +- The September audit added transactional writes, reset generations, complete managed-memory deletion, duplicate-submission protection, retryable call finalization, and camera inference filtering. Tests and code are the current authority; old test counts are historical snapshots. + +## Cleanup validation (2026-09-10) + +- After retiring the Tk launcher and condensing these documents, `npm run check` passed 133 Python tests without skips, 11 renderer scenarios, JS syntax/redaction checks, and mocked smoke. Ruff and relative documentation links passed. +- The PyInstaller bridge passed its outside-checkout status/missing-key/SDK-import smokes. Electron packaging succeeded using the installed lockfile-matching Electron distribution; ad-hoc signing and strict deep signature verification passed. The host required Command Line Tools via a per-command `DEVELOPER_DIR` because its selected Xcode loader was broken. The default Electron download was blocked by sandbox DNS; no source/build-config workaround was committed. +- No paid API calls or physical-camera/clean-account GUI acceptance were performed. + +## Provenance + +The full construction log and original five design proposals remain available in [the pre-cleanup Git snapshot](https://github.com/Jo2234/FluentAI/tree/d923a6a49243ed19cbad3c3470e165157f2c34b0). In that snapshot, `notes.md` records dated work and validation, and `docs/` contains the original north-star, schema, onboarding, curriculum, and packaging proposals. Their work-package assignments, future-tense descriptions, and old source line numbers are historical, not current implementation instructions. Git history retains authorship and the evolution of those decisions. diff --git a/plan.md b/plan.md index b8cb6e7..12abddb 100644 --- a/plan.md +++ b/plan.md @@ -1,59 +1,7 @@ -# Plan +# Current plan -## Goal -Create a demoable agentic language learning app with Lesson Mode and Conversation Mode. Both modes read local learner state, adapt to current ability, update progress, and make agent decisions visible during a judging demo. +The core lesson/conversation loop, per-language state, spaced repetition, onboarding, memory controls, packaging path, and durable/retryable session completion are implemented. -## MVP Shape -- Python CLI loop for a reliable judging demo. -- Local state in `data/progress.json`. -- Autonomous demo mode that simulates learner answers so the loop can run unattended for 5-10 minutes. -- Interactive mode for a real learner. -- Standalone fullscreen Electron desktop app for manual testing outside the browser. -- Conversation mode: a FaceTime-like tutor that initiates and steers spoken/chat conversation based on learner ability, with video optional. -- Video-on conversation can use an object/image context such as "apple" to keep the target-language conversation grounded. -- Optional Gradio UI when the dependency is installed, with console fallback. -- OpenAI-backed generation for real lessons and conversations; mocked providers are used only in tests/CI. +Remaining work is maintained in the [product roadmap](docs/FLUENTAI_LIMITLESS_IDEAL.md#remaining-roadmap). Use the [architecture](docs/GAP_AND_DESIGN.md), [curriculum](docs/CURRICULUM_DESIGN.md), [privacy](docs/ONBOARDING_HOME_DESIGN.md), and [packaging](docs/PACKAGING_DESIGN.md) documents for current decisions and boundaries. -## Build Steps -- Scaffold project docs and concise Goal-mode instructions. Done. -- Implement local learner state model. Done. -- Implement lesson generation, quiz generation, evaluation, and adaptation. Done. -- Add a CLI loop with duration and interval settings. Done. -- Add a smoke test or command that proves state updates. Done. -- Add optional Gradio UI fallback. Done. -- Add Conversation Mode state, turn loop, adaptive topic steering, video-object handling, and memory updates. Done. -- Add `.env` loading and OpenAI Responses API support for lesson enhancement and conversation tutor utterances. Done. -- Add no-dependency local web app for browser testing. Done. -- Add standalone fullscreen Electron desktop app. Done. -- Replace transcript-only desktop output with real lesson quiz submission and turn-by-turn conversation. Done. -- Apply approved red Duolingo-inspired Demo Studio UI to the Electron and browser app. Done. -- Polish Lesson Mode UI and make Conversation Mode voice-first with text fallback. Done. -- Add OpenAI Realtime voice-agent integration for speech-to-speech conversation. Done. -- Put voice-only and video calls in the main Conversation Mode stage instead of a side panel. Done. -- Replace typed video-object demo labels with OpenAI camera-frame recognition. Done. -- Stabilize call/video/text fallback UX after live app testing. Done. -- Make video mode use the live camera feed with recurring OpenAI vision updates and filter synthetic green test feeds. Done. -- Remove model-name UI copy and improve English-help responsiveness in tutor conversation. Done. -- Add Hindi, Spanish, and French language selection across lessons, text fallback, realtime voice, and camera-context prompts. Done. -- Make the visible agent decision log collapsible. Done. -- Migrate Lesson and Conversation state call sites to state schema v2 helpers and remove temporary v1 compatibility mirrors. Done. - -## Next -- Add spaced repetition scheduling. Done. -- Implement WP4/WP5: lesson-selection reasons, quiz error categories, OpenAI grading override, and lesson-driven mistake memory. Done. -- Implement WP6/WP7: lesson-to-conversation goals, conversation mistakes feeding next lessons, and persistent post-call summaries for text/web/CLI/voice. Done. -- Add richer browser/desktop regression tests for voice and camera behavior. -- Tune Realtime voice turn-taking and English-help behavior for natural language-learning conversations. Done. -- Implement WP4 Home workspace, memory inspector/export/reset/delete controls, and privacy-safe memory payloads. -- Packaging steps 2/3: add API key manager, bridge validation, safe-storage/session key handling, and corrupt progress recovery. -- Packaging step 6 reliability pass: realtime refresh, renderer timeouts, model-failure copy, empty-tutor recovery, lesson/call checkpoints, and camera voice-only fallback. Done. -- Fix acceptance-run defects: packaged bridge Realtime SSL certificate setup and provider-graded Conversation mistake memory. Done. -- Final Phase 2/6 polish: phrase listening, pronunciation/culture lesson cards, and post-call pronunciation practice note. Done. - -## Approved audit fixes (2026-09-06) -- Delete managed learner memory completely and reject stale session writes. Done. -- Finalize calls consistently across navigation, language changes and window close; retain failed saves for retry. Done. -- Commit progress through cross-process transactions and atomic file replacement. Done. -- Deduplicate lesson submission before grading and progress updates. Done. -- Skip unchanged camera frames before inference, with bounded refresh and explicit capture. Done. -- Run concurrency/privacy/session regressions, existing suites, lint and installed-package smoke. Done; hardware camera/window acceptance remains manual. +For each change, verify its behavior with the relevant tests and update these concise documents if the contract changes. Use temporary learner state for verification. Keep actual validation evidence and unresolved acceptance limits in [notes](notes.md). diff --git a/tests/test_project_contract.py b/tests/test_project_contract.py index 2343186..abac643 100644 --- a/tests/test_project_contract.py +++ b/tests/test_project_contract.py @@ -9,7 +9,15 @@ class ProjectContractTests(unittest.TestCase): def test_required_demo_files_exist(self): root = Path(__file__).resolve().parents[1] - for relative in [".env.example", "macos/desktop_app.py", "scripts/smoke_demo.py", ".github/workflows/smoke.yml"]: + for relative in [ + ".env.example", + "desktop/electron/main.js", + "desktop/electron/preload.js", + "desktop/electron/renderer.html", + "fluent_ai/desktop_bridge.py", + "scripts/smoke_demo.py", + ".github/workflows/smoke.yml", + ]: self.assertTrue((root / relative).exists(), relative) def test_openai_status_is_non_secret_and_model_aware(self):