diff --git a/CONNECTOR_ARCHITECTURE.md b/CONNECTOR_ARCHITECTURE.md new file mode 100644 index 0000000..8587f1d --- /dev/null +++ b/CONNECTOR_ARCHITECTURE.md @@ -0,0 +1,92 @@ +# Connector Architecture + +Draft for technical and privacy review. + +The final MVP should be connector-first. Contributors should not have to manually upload +chat exports as the normal path. Instead, they should authorize connectors to supported +LLM platforms, see exactly what will sync, and view their results on +makeaivisible.org. + +Manual upload remains a fallback for early testing, unsupported platforms, and local +developer demos. + +## Product Goal + +1. Contributor visits makeaivisible.org. +2. Contributor chooses a supported LLM connector. +3. Contributor reviews consent and sync scope. +4. Contributor authorizes access with the platform. +5. Eligible conversations sync in the background. +6. Raw text is sent through the anonymization service. +7. Contributor sees safe results and project status on the website. + +## Initial Connector Targets + +| Platform | MVP posture | Notes | +| --- | --- | --- | +| ChatGPT | Target connector | Prefer official export/API mechanisms when available. | +| Claude | Target connector | Confirm supported account and data access patterns. | +| Gemini | Target connector | Confirm Google account/API consent and revocation behavior. | +| Copilot | Target connector | Confirm whether user chat history access is available and appropriate. | +| Other LLMs | Later | Add only after consent, privacy, and technical feasibility review. | + +## Consent And Authorization Rules + +Connectors must be: + +- opt-in +- revocable +- clear about what data is collected +- clear about sync frequency +- clear about what is not published +- scoped to the minimum permission needed +- separate from passwords or credential collection + +The portal should never ask for platform account passwords. Authorization should happen +through official OAuth or approved platform mechanisms. + +## Sync Model + +The MVP should start with conservative background sync: + +- pull only conversations after authorization +- store connector tokens separately from conversation content +- avoid raw message text in logs +- record sync status, timestamps, and error codes +- allow contributors to pause, disconnect, or delete synced data +- send raw chat content directly to anonymization before review/scoring + +## Data Boundary + +```text +LLM platform connector + -> connector ingest worker + -> anonymization service + -> privacy review queue + -> scoring engine + -> dataset publishing pipeline + -> aggregate dashboard +``` + +Raw chat content should exist only in the shortest-lived systems needed for sync and +anonymization. Public systems should receive aggregate or synthetic outputs only. + +## Minimum Technical Components + +- Connector registry: supported platforms, scopes, status, and limitations. +- Authorization callback handler: receives platform auth results. +- Token store: encrypted token storage with revocation metadata. +- Sync worker: fetches eligible conversations and tracks sync cursors. +- Anonymization handoff: sends normalized messages to the anonymization service. +- Contributor dashboard: shows connection status, sync history, and results. +- Deletion path: deletes or invalidates data according to policy. + +## Open Questions + +- Which platforms provide official APIs or exports suitable for background sync? +- Which platforms allow user-authorized chat history access? +- How often should sync run by default? +- What contributor controls are required before syncing starts? +- How should deletion requests propagate through anonymized/scored/aggregate data? +- What minimum age, assent, and caregiver consent rules apply before real submissions? +- What data must never leave the contributor browser before expert review is complete? diff --git a/END_TO_END_DEMO.md b/END_TO_END_DEMO.md new file mode 100644 index 0000000..97b2b2e --- /dev/null +++ b/END_TO_END_DEMO.md @@ -0,0 +1,115 @@ +# End-To-End Demo + +Draft developer walkthrough. + +This document explains the current synthetic MVP path. It is not a production data flow +and must not be used with real contributor conversations. + +## Current Demo Flow + +```text +Portal-Frontend preview + -> Anonymization-Service /anonymize + -> NLP-Scoring-Engine baseline scorer + -> Dataset-Publishing-Pipeline release builder + -> Dashboard synthetic aggregate view +``` + +## 1. Portal Preview + +Repository: `Portal-Frontend` + +The portal repo currently shows the future connector/upload experience. It points users +to the live website at makeaivisible.org and explains that the final MVP should use +authorized connectors for major LLM platforms. + +The file picker is a fallback preview. It does not upload or persist files. + +## 2. Anonymization + +Repository: `Anonymization-Service` + +Run locally: + +```bash +python -m venv .venv +source .venv/bin/activate +pip install -e '.[dev]' +uvicorn makeaivisible_anonymizer.main:app --reload +``` + +Submit the synthetic request: + +```bash +curl -s http://127.0.0.1:8000/anonymize \ + -H 'content-type: application/json' \ + --data @examples/request.json +``` + +Expected behavior: + +- direct identifiers are replaced with labels such as `[EMAIL_1]` +- response includes an opaque conversation UUID +- caller-supplied identifiers are rejected +- no raw input is persisted by the service baseline + +## 3. Scoring + +Repository: `NLP-Scoring-Engine` + +Run: + +```bash +python -m scoring_engine.cli examples/anonymized_conversation.json +``` + +Expected behavior: + +- returns deterministic dimension scores +- includes evidence keywords +- marks output as `validated_for_claims: false` + +## 4. Dataset Release + +Repository: `Dataset-Publishing-Pipeline` + +Run: + +```bash +python -m release_builder.cli examples/reviewed_records.json --out-dir dist +``` + +Expected behavior: + +- fails unless records are marked `approved_for_release` +- writes `aggregate_release.json` +- writes `DATASET_CARD.md` +- marks the synthetic release as not public-ready if below threshold + +## 5. Dashboard + +Repository: `Dashboard` + +Run: + +```bash +python3 -m http.server 8000 +``` + +Open `http://127.0.0.1:8000`. + +Expected behavior: + +- reads synthetic aggregate data +- displays aggregate dimensions +- suppresses counts below `min_public_cohort` +- does not show individual records or quotes + +## Demo Limitations + +- Manual upload is only a fallback preview. +- Connectors are not implemented yet. +- PII detection is incomplete. +- Scoring is unvalidated. +- Dashboard data is synthetic. +- Governance docs require expert review before real submissions. diff --git a/GITHUB_ISSUE_CLEANUP.md b/GITHUB_ISSUE_CLEANUP.md new file mode 100644 index 0000000..959ff75 --- /dev/null +++ b/GITHUB_ISSUE_CLEANUP.md @@ -0,0 +1,75 @@ +# GitHub Issue Cleanup Plan + +Draft maintainer checklist. + +The MVP branches have landed, but several initial issues still describe work that is now +partly complete. This file proposes how to update them without losing the next steps. + +## Close Or Update + +### Portal-Frontend + +- `#1 MVP: Build mobile-first consent and upload prototype` + - Suggested status: close as MVP prototype complete. + - Follow-up issue: connector-first authorization mock. + +- `#2 MVP: Add platform export guidance` + - Suggested status: keep open or replace. + - Update direction: platform guidance should focus on connector targets, with manual + exports as fallback only. + +### Anonymization-Service + +- `#2 MVP: Add PII detector test suite` + - Suggested status: keep open. + - Update direction: active detectors exist for direct identifiers; expected-failure + tests now document names, schools, addresses, and indirect identifiers. + +### NLP-Scoring-Engine + +- `#1 MVP: Create baseline scoring schema and deterministic rubric scorer` + - Suggested status: close as baseline complete. + - Follow-up issue: schema hardening and richer rubric examples. + +- `#2 MVP: Build evaluation harness for human-label comparison` + - Suggested status: keep open. + - Update direction: next step is synthetic human-label fixtures and agreement report. + +### Dataset-Publishing-Pipeline + +- `#1 MVP: Create release manifest and aggregate export builder` + - Suggested status: partially complete. + - Update direction: aggregate builder exists; release manifest can be expanded. + +- `#2 MVP: Draft dataset card and release checklist generator` + - Suggested status: partially complete. + - Update direction: dataset card draft exists; checklist generator still needs depth. + +### Dashboard + +- `#1 MVP: Build synthetic aggregate dashboard` + - Suggested status: close as synthetic baseline complete. + +- `#2 MVP: Add privacy threshold and suppression rules` + - Suggested status: partially complete. + - Update direction: suppression exists in UI; add tests and policy configuration. + +### Governance-Documentation + +- `#1 MVP: Draft data lifecycle and consent documents` + - Suggested status: partially complete. + - Update direction: drafts exist; expert/legal/IRB review still required. + +- `#2 MVP: Draft human review and scoring validation protocol` + - Suggested status: partially complete. + - Update direction: protocol exists; calibration metrics and validation thresholds need review. + +## Suggested New Issues + +- Portal: Design connector authorization mock +- Portal: Document platform-specific connector feasibility +- Governance: Review connector consent and revocation policy +- Anonymization: Promote expected-failure PII gap tests into passing detectors +- Scoring: Add synthetic human-label evaluation harness +- Dataset: Expand release manifest and review checklist +- Dashboard: Add automated suppression tests diff --git a/LIVE_SITE_INTEGRATION.md b/LIVE_SITE_INTEGRATION.md new file mode 100644 index 0000000..e099a74 --- /dev/null +++ b/LIVE_SITE_INTEGRATION.md @@ -0,0 +1,63 @@ +# Live Site Integration Plan + +Draft implementation plan. + +The live public website is: + +https://makeaivisible.org + +The GitHub repositories should support that site, not compete with it. The live site is +the public home; the repos hold implementation modules, demos, and governance material. + +## Current Split + +| Surface | Role | +| --- | --- | +| makeaivisible.org | Public website and future contributor-facing experience | +| Portal-Frontend | Connector/upload preview and future portal reference | +| Anonymization-Service | Privacy boundary for raw or synced conversations | +| NLP-Scoring-Engine | Baseline scoring and future validation tooling | +| Dataset-Publishing-Pipeline | Reviewed aggregate release generation | +| Dashboard | Synthetic aggregate dashboard prototype | +| Governance-Documentation | Policy, consent, review, and methodology source of truth | + +## What Belongs On The Live Site + +- project explanation +- challenge/repo links +- connector entry point when ready +- contributor consent flow +- connected-account status +- sync status +- personal result summaries +- aggregate public dashboard when approved + +## What Belongs In GitHub Repos + +- source code +- developer docs +- issue tracking +- CI checks +- synthetic examples +- governance drafts +- implementation plans +- privacy and review checklists + +## Near-Term Site Updates + +The live site should eventually reflect the GitHub state: + +- link each challenge card to the matching repository +- show that the portal is connector-first and coming soon +- avoid implying manual uploads are the final product +- link governance docs before real data collection +- state that current demos are synthetic + +## Integration Milestones + +1. Add a clear "Connectors coming soon" section to the live site. +2. Link each GitHub repo from the relevant live-site section. +3. Build a connector authorization mock on the site. +4. Wire the mock to the anonymization service contract. +5. Replace mock data with reviewed synthetic demo data. +6. Start platform-specific connector spikes only after privacy review. diff --git a/README.md b/README.md index ef56453..0647ea3 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,10 @@ Publish draft governance documents that let volunteers understand the project, o - [CONSENT_DRAFT.md](CONSENT_DRAFT.md): plain-language contributor and caregiver draft copy. - [REVIEW_PROTOCOL.md](REVIEW_PROTOCOL.md): reviewer access, prohibited actions, review states, and calibration. - [MVP_REVIEW_CHECKLIST.md](MVP_REVIEW_CHECKLIST.md): gates before real uploads and public release. +- [CONNECTOR_ARCHITECTURE.md](CONNECTOR_ARCHITECTURE.md): connector-first MVP direction, authorization, sync, and privacy boundaries. +- [END_TO_END_DEMO.md](END_TO_END_DEMO.md): current synthetic demo path across the repos. +- [LIVE_SITE_INTEGRATION.md](LIVE_SITE_INTEGRATION.md): how GitHub work should map back to makeaivisible.org. +- [GITHUB_ISSUE_CLEANUP.md](GITHUB_ISSUE_CLEANUP.md): maintainer checklist for closing/updating MVP issues. These documents are working drafts. They are meant to expose decisions and open questions early, not to replace legal, IRB, privacy, or security review.