Add connector architecture and cleanup plans

CONNECTOR_ARCHITECTURE.md

@@ -0,0 +1,92 @@
+# Connector Architecture
+
+Draft for technical and privacy review.
+
+The final MVP should be connector-first. Contributors should not have to manually upload
+chat exports as the normal path. Instead, they should authorize connectors to supported
+LLM platforms, see exactly what will sync, and view their results on
+makeaivisible.org.
+
+Manual upload remains a fallback for early testing, unsupported platforms, and local
+developer demos.
+
+## Product Goal
+
+1. Contributor visits makeaivisible.org.
+2. Contributor chooses a supported LLM connector.
+3. Contributor reviews consent and sync scope.
+4. Contributor authorizes access with the platform.
+5. Eligible conversations sync in the background.
+6. Raw text is sent through the anonymization service.
+7. Contributor sees safe results and project status on the website.
+
+## Initial Connector Targets
+
+| Platform | MVP posture | Notes |
+| --- | --- | --- |
+| ChatGPT | Target connector | Prefer official export/API mechanisms when available. |
+| Claude | Target connector | Confirm supported account and data access patterns. |
+| Gemini | Target connector | Confirm Google account/API consent and revocation behavior. |
+| Copilot | Target connector | Confirm whether user chat history access is available and appropriate. |
+| Other LLMs | Later | Add only after consent, privacy, and technical feasibility review. |
+
+## Consent And Authorization Rules
+
+Connectors must be:
+
+- opt-in
+- revocable
+- clear about what data is collected
+- clear about sync frequency
+- clear about what is not published
+- scoped to the minimum permission needed
+- separate from passwords or credential collection
+
+The portal should never ask for platform account passwords. Authorization should happen
+through official OAuth or approved platform mechanisms.
+
+## Sync Model
+
+The MVP should start with conservative background sync:
+
+- pull only conversations after authorization
+- store connector tokens separately from conversation content
+- avoid raw message text in logs
+- record sync status, timestamps, and error codes
+- allow contributors to pause, disconnect, or delete synced data
+- send raw chat content directly to anonymization before review/scoring
+
+## Data Boundary
+
+```text
+LLM platform connector
+  -> connector ingest worker
+  -> anonymization service
+  -> privacy review queue
+  -> scoring engine
+  -> dataset publishing pipeline
+  -> aggregate dashboard
+```
+
+Raw chat content should exist only in the shortest-lived systems needed for sync and
+anonymization. Public systems should receive aggregate or synthetic outputs only.
+
+## Minimum Technical Components
+
+- Connector registry: supported platforms, scopes, status, and limitations.
+- Authorization callback handler: receives platform auth results.
+- Token store: encrypted token storage with revocation metadata.
+- Sync worker: fetches eligible conversations and tracks sync cursors.
+- Anonymization handoff: sends normalized messages to the anonymization service.
+- Contributor dashboard: shows connection status, sync history, and results.
+- Deletion path: deletes or invalidates data according to policy.
+
+## Open Questions
+
+- Which platforms provide official APIs or exports suitable for background sync?
+- Which platforms allow user-authorized chat history access?
+- How often should sync run by default?
+- What contributor controls are required before syncing starts?
+- How should deletion requests propagate through anonymized/scored/aggregate data?
+- What minimum age, assent, and caregiver consent rules apply before real submissions?
+- What data must never leave the contributor browser before expert review is complete?

END_TO_END_DEMO.md

@@ -0,0 +1,115 @@
+# End-To-End Demo
+
+Draft developer walkthrough.
+
+This document explains the current synthetic MVP path. It is not a production data flow
+and must not be used with real contributor conversations.
+
+## Current Demo Flow
+
+```text
+Portal-Frontend preview
+  -> Anonymization-Service /anonymize
+  -> NLP-Scoring-Engine baseline scorer
+  -> Dataset-Publishing-Pipeline release builder
+  -> Dashboard synthetic aggregate view
+```
+
+## 1. Portal Preview
+
+Repository: `Portal-Frontend`
+
+The portal repo currently shows the future connector/upload experience. It points users
+to the live website at makeaivisible.org and explains that the final MVP should use
+authorized connectors for major LLM platforms.
+
+The file picker is a fallback preview. It does not upload or persist files.
+
+## 2. Anonymization
+
+Repository: `Anonymization-Service`
+
+Run locally:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e '.[dev]'
+uvicorn makeaivisible_anonymizer.main:app --reload
+```
+
+Submit the synthetic request:
+
+```bash
+curl -s http://127.0.0.1:8000/anonymize \
+  -H 'content-type: application/json' \
+  --data @examples/request.json
+```
+
+Expected behavior:
+
+- direct identifiers are replaced with labels such as `[EMAIL_1]`
+- response includes an opaque conversation UUID
+- caller-supplied identifiers are rejected
+- no raw input is persisted by the service baseline
+
+## 3. Scoring
+
+Repository: `NLP-Scoring-Engine`
+
+Run:
+
+```bash
+python -m scoring_engine.cli examples/anonymized_conversation.json
+```
+
+Expected behavior:
+
+- returns deterministic dimension scores
+- includes evidence keywords
+- marks output as `validated_for_claims: false`
+
+## 4. Dataset Release
+
+Repository: `Dataset-Publishing-Pipeline`
+
+Run:
+
+```bash
+python -m release_builder.cli examples/reviewed_records.json --out-dir dist
+```
+
+Expected behavior:
+
+- fails unless records are marked `approved_for_release`
+- writes `aggregate_release.json`
+- writes `DATASET_CARD.md`
+- marks the synthetic release as not public-ready if below threshold
+
+## 5. Dashboard
+
+Repository: `Dashboard`
+
+Run:
+
+```bash
+python3 -m http.server 8000
+```
+
+Open `http://127.0.0.1:8000`.
+
+Expected behavior:
+
+- reads synthetic aggregate data
+- displays aggregate dimensions
+- suppresses counts below `min_public_cohort`
+- does not show individual records or quotes
+
+## Demo Limitations
+
+- Manual upload is only a fallback preview.
+- Connectors are not implemented yet.
+- PII detection is incomplete.
+- Scoring is unvalidated.
+- Dashboard data is synthetic.
+- Governance docs require expert review before real submissions.

GITHUB_ISSUE_CLEANUP.md

@@ -0,0 +1,75 @@
+# GitHub Issue Cleanup Plan
+
+Draft maintainer checklist.
+
+The MVP branches have landed, but several initial issues still describe work that is now
+partly complete. This file proposes how to update them without losing the next steps.
+
+## Close Or Update
+
+### Portal-Frontend
+
+- `#1 MVP: Build mobile-first consent and upload prototype`
+  - Suggested status: close as MVP prototype complete.
+  - Follow-up issue: connector-first authorization mock.
+
+- `#2 MVP: Add platform export guidance`
+  - Suggested status: keep open or replace.
+  - Update direction: platform guidance should focus on connector targets, with manual
+    exports as fallback only.
+
+### Anonymization-Service
+
+- `#2 MVP: Add PII detector test suite`
+  - Suggested status: keep open.
+  - Update direction: active detectors exist for direct identifiers; expected-failure
+    tests now document names, schools, addresses, and indirect identifiers.
+
+### NLP-Scoring-Engine
+
+- `#1 MVP: Create baseline scoring schema and deterministic rubric scorer`
+  - Suggested status: close as baseline complete.
+  - Follow-up issue: schema hardening and richer rubric examples.
+
+- `#2 MVP: Build evaluation harness for human-label comparison`
+  - Suggested status: keep open.
+  - Update direction: next step is synthetic human-label fixtures and agreement report.
+
+### Dataset-Publishing-Pipeline
+
+- `#1 MVP: Create release manifest and aggregate export builder`
+  - Suggested status: partially complete.
+  - Update direction: aggregate builder exists; release manifest can be expanded.
+
+- `#2 MVP: Draft dataset card and release checklist generator`
+  - Suggested status: partially complete.
+  - Update direction: dataset card draft exists; checklist generator still needs depth.
+
+### Dashboard
+
+- `#1 MVP: Build synthetic aggregate dashboard`
+  - Suggested status: close as synthetic baseline complete.
+
+- `#2 MVP: Add privacy threshold and suppression rules`
+  - Suggested status: partially complete.
+  - Update direction: suppression exists in UI; add tests and policy configuration.
+
+### Governance-Documentation
+
+- `#1 MVP: Draft data lifecycle and consent documents`
+  - Suggested status: partially complete.
+  - Update direction: drafts exist; expert/legal/IRB review still required.
+
+- `#2 MVP: Draft human review and scoring validation protocol`
+  - Suggested status: partially complete.
+  - Update direction: protocol exists; calibration metrics and validation thresholds need review.
+
+## Suggested New Issues
+
+- Portal: Design connector authorization mock
+- Portal: Document platform-specific connector feasibility
+- Governance: Review connector consent and revocation policy
+- Anonymization: Promote expected-failure PII gap tests into passing detectors
+- Scoring: Add synthetic human-label evaluation harness
+- Dataset: Expand release manifest and review checklist
+- Dashboard: Add automated suppression tests

LIVE_SITE_INTEGRATION.md

@@ -0,0 +1,63 @@
+# Live Site Integration Plan
+
+Draft implementation plan.
+
+The live public website is:
+
+https://makeaivisible.org
+
+The GitHub repositories should support that site, not compete with it. The live site is
+the public home; the repos hold implementation modules, demos, and governance material.
+
+## Current Split
+
+| Surface | Role |
+| --- | --- |
+| makeaivisible.org | Public website and future contributor-facing experience |
+| Portal-Frontend | Connector/upload preview and future portal reference |
+| Anonymization-Service | Privacy boundary for raw or synced conversations |
+| NLP-Scoring-Engine | Baseline scoring and future validation tooling |
+| Dataset-Publishing-Pipeline | Reviewed aggregate release generation |
+| Dashboard | Synthetic aggregate dashboard prototype |
+| Governance-Documentation | Policy, consent, review, and methodology source of truth |
+
+## What Belongs On The Live Site
+
+- project explanation
+- challenge/repo links
+- connector entry point when ready
+- contributor consent flow
+- connected-account status
+- sync status
+- personal result summaries
+- aggregate public dashboard when approved
+
+## What Belongs In GitHub Repos
+
+- source code
+- developer docs
+- issue tracking
+- CI checks
+- synthetic examples
+- governance drafts
+- implementation plans
+- privacy and review checklists
+
+## Near-Term Site Updates
+
+The live site should eventually reflect the GitHub state:
+
+- link each challenge card to the matching repository
+- show that the portal is connector-first and coming soon
+- avoid implying manual uploads are the final product
+- link governance docs before real data collection
+- state that current demos are synthetic
+
+## Integration Milestones
+
+1. Add a clear "Connectors coming soon" section to the live site.
+2. Link each GitHub repo from the relevant live-site section.
+3. Build a connector authorization mock on the site.
+4. Wire the mock to the anonymization service contract.
+5. Replace mock data with reviewed synthetic demo data.
+6. Start platform-specific connector spikes only after privacy review.

README.md

@@ -28,6 +28,10 @@ Publish draft governance documents that let volunteers understand the project, o
 - [CONSENT_DRAFT.md](CONSENT_DRAFT.md): plain-language contributor and caregiver draft copy.
 - [REVIEW_PROTOCOL.md](REVIEW_PROTOCOL.md): reviewer access, prohibited actions, review states, and calibration.
 - [MVP_REVIEW_CHECKLIST.md](MVP_REVIEW_CHECKLIST.md): gates before real uploads and public release.
+- [CONNECTOR_ARCHITECTURE.md](CONNECTOR_ARCHITECTURE.md): connector-first MVP direction, authorization, sync, and privacy boundaries.
+- [END_TO_END_DEMO.md](END_TO_END_DEMO.md): current synthetic demo path across the repos.
+- [LIVE_SITE_INTEGRATION.md](LIVE_SITE_INTEGRATION.md): how GitHub work should map back to makeaivisible.org.
+- [GITHUB_ISSUE_CLEANUP.md](GITHUB_ISSUE_CLEANUP.md): maintainer checklist for closing/updating MVP issues.
 
 These documents are working drafts. They are meant to expose decisions and open
 questions early, not to replace legal, IRB, privacy, or security review.