A privacy-preserving synthetic education data engine that simulates a high-school math department for assessment analytics, dashboard development, and learning-systems prototyping.
Inspect the validation results | Open the longitudinal dashboard | Read the project brief
The local Python/SQL path reconciles 2,009 roster rows and models 4,018
assessment records. A clean rebuild on 19 October 2026 from public commit
2ecdb77 passed all 20 warehouse checks and regenerated the synthetic source
artifacts without changes. Checks cover unique enrollments, score bounds,
LMS extraction counts, and source-to-roster reconciliation.
Run make all and make warehouse after installing the documented DuckDB
dependency to reproduce the checks. The optional Supabase path was checked in
dry-run mode only; this verification does not establish a hosted deployment.
The separate Portfolio Data Lab offers an AI-assisted exploration interface over the synthetic warehouse. It is optional; the source, validation tables, and local rebuild do not depend on a live AI service. The Logos single-administration dashboard uses a different synthetic reporting dataset.
This repository demonstrates education analytics infrastructure without publishing protected student data, raw LMS exports, real gradebooks, teacher names, section labels, or school-private records.
It is not just a fake CSV generator. The project creates a coherent synthetic department system with students, teachers, courses, sections, enrollments, assessment scores, attendance/non-participation behavior, Canvas-style course artifacts, and validation checks.
- Synthetic education data generation for public-safe analytics workflows
- Nested education data structure: students, sections, courses, teachers, and enrollments
- Canvas-style all-school math assessment gradebook generation
- Synthetic Canvas API-style profile extraction into SQL tables
- Assessment score simulation with attendance/non-participation modeled separately from academic performance
- Course-track, teacher, section, growth, measurement-error, observation-noise, and regression-to-the-mean effects
- Bayesian-style readiness updates for reusable longitudinal score generation
- Grade-level calibration diagnostics for longitudinal modeling
- Optional DuckDB analytics warehouse with SQL validation queries and dashboard-ready mart exports
- LMS-to-SQL roster reconciliation from synthetic Canvas course profiles
- Star-schema reporting model with student, course, section, teacher, assignment, assessment-score, and LMS-enrollment tables
- Canonical state object generation with reproducible CSV and JSON exports
- Seven-year synthetic student churn with graduating seniors and replacement freshman cohorts
- Validation of counts, schema, enrollment consistency, score bounds, assignment population policy, Canvas-style profiles, and public-safety constraints
The generator separates the department structure from the assessment measurement process.
synthetic students, teachers, courses, sections, enrollments
-> assessment context by grade, course, track, teacher, and section
-> attendance / non-participation draw
-> latent readiness and observed assessment score
-> validation-ready public artifacts
Assignments 01-14 represent beginning- and end-of-year standardized assessment windows across seven synthetic academic years. Present-student scores are drawn or updated through a reusable longitudinal score engine. Attendance is modeled separately, so an observed zero means non-participation, not academic evidence.
The engine separates hidden latent readiness from observed posterior readiness. Hidden latent readiness advances at every assessment window through school-year growth or summer atrophy, while observed posterior readiness updates only when a student is present and produces score evidence. The model also adds course/track and teacher/section effects, includes regression-to-the-mean behavior, and separates growth noise from assessment observation noise.
If a student is absent for an assessment window, the observed score is 0 and observed posterior readiness is not updated. The hidden synthetic latent trajectory still advances, which prevents absence from freezing the student's underlying simulated academic state.
synthetic math department state
-> synthetic ASMA gradebook
-> yearly synthetic ASMA gradebooks
-> long assessment score export
-> synthetic course, section, and enrollment exports
-> validation
-> synthetic Canvas profile extraction into SQL raw tables
-> optional DuckDB SQL warehouse and mart exports
-> downstream assessment analysis
The canonical source of truth is:
data/synthetic/synthetic_school_state.json
Downstream CSV artifacts are rendered from that state:
data/synthetic/synthetic_asma_gradebook.csv
data/synthetic/synthetic_assessment_scores_long.csv
data/synthetic/synthetic_math_courses.csv
data/synthetic/synthetic_math_sections.csv
data/synthetic/synthetic_math_enrollments.csv
The generator also renders year-specific ASMA gradebooks and synthetic Canvas-style course profiles:
data/synthetic/assessment_shells/
data/synthetic/canvas_course_profiles/
Optional SQL mart exports are generated by the DuckDB analytics warehouse:
data/marts/
- docs/methodology.md explains the data-generating process, Assignment 01 score generation, the longitudinal score engine, Canvas-style artifacts, and validation checks.
- docs/star-schema-erd.md diagrams the DuckDB star schema used for downstream reporting.
- docs/data-lineage.md documents how synthetic artifacts flow into raw SQL tables, marts, hosted Postgres, and downstream reports.
- docs/supabase-postgres-deployment.md explains the optional Supabase/Postgres serving path without committing credentials.
- sql/examples/ contains readable SQL examples for enrollment, growth, missingness, readiness, reconciliation, and dashboard extracts.
- data/synthetic/synthetic_school_state.json is the canonical state object used to render downstream artifacts.
- data/synthetic/synthetic_asma_gradebook.csv is the public-safe all-school math assessment gradebook.
- scripts/generate_synthetic_math_department.py contains the simulation logic.
- scripts/validate_synthetic_math_department.py checks artifact shape, coherence, score policy, and public-safety boundaries.
- reports/grade-level-calibration/grade-level-calibration-report.md shows the aggregate calibration diagnostics used to support weak grade-level priors.
make allOr run the steps separately:
make generate
make validateThe project uses only the Python standard library.
Optional SQL analytics warehouse outputs can be built with DuckDB:
make analytics-install
make warehouseThe warehouse target creates a local warehouse/synthetic_math.duckdb database, runs SQL models from sql/marts/, exports public-safe mart CSVs to data/marts/, and checks mart.validation_summary.
See docs/duckdb-analytics-warehouse.md for the SQL workflow.
The warehouse also normalizes synthetic Canvas course profile JSON into raw_canvas SQL tables and reconciles those LMS-style records against canonical synthetic enrollments. See docs/canvas-workflow-simulation.md for the Canvas-to-SQL simulation.
The exported reporting layer includes both analytic marts and a star schema:
dim_student
dim_course
dim_section
dim_teacher
dim_assignment
fact_assessment_score
fact_lms_enrollment
An optional Supabase/Postgres serving path is scaffolded for the validated star-schema marts. DuckDB remains the reproducible local warehouse; Supabase/Postgres is the hosted serving layer for public-safe synthetic analytics tables and API-style access after a private connection string is supplied locally.
make postgres-install
make postgres-load-dry-runAfter setting SUPABASE_DATABASE_URL in a local .env or shell session, load the hosted database with:
make postgres-loadSee docs/supabase-postgres-deployment.md and docs/duckdb-analytics-warehouse.md.
Optional grade-level calibration diagnostics can be generated from a private gradebook path:
SOURCE_GRADEBOOK=/path/to/private/gradebook.csv make calibrate-grade-levelThe calibration target writes public-safe aggregate diagnostics only. It does not write source rows, identifiers, emails, section labels, or private paths to public outputs.
This repository is the synthetic data foundation.
The downstream assessment-intelligence project is responsible for the visual analytics and reporting layer: dashboards, distribution checks, growth diagnostics, decision-support reports, and leadership-facing interpretation.
synthetic-education-data -> data generation and validation
synthetic-education-data -> DuckDB SQL marts and reporting extracts
synthetic-education-data -> star-schema fact/dimension tables
synthetic-education-data -> optional Supabase/Postgres hosted deployment for synthetic analytics tables
assessment-intelligence -> analytics, dashboards, diagnostics, and reporting
This repository is designed to be public-safe from the first commit. It contains synthetic data and generalized methodology only.
Do not commit:
- real students, emails, IDs, or rosters
- raw LMS exports
- private assessment artifacts
- private teacher names
- internal section labels
- school-private paths
- private calibration/debug files
See docs/public-safety.md for the release boundary.
Current version is an active seven-year longitudinal synthetic math department simulation.
It generates academic years 2025-2026 through 2031-2032 with:
- 696 all-ever synthetic students
- 287 active students per school year
- 5 synthetic teachers per school year
- 9 math course entries
- 174 synthetic sections across the seven-year horizon
- 2,009 active student-year enrollments
- 62 synthetic Canvas course JSON profiles
- 14 assessment assignment fields
- all 14 assignment windows populated for active student-year records
- yearly ASMA gradebooks under
data/synthetic/assessment_shells/ - a long assessment-score export with 4,018 rows
- DuckDB mart exports for assessment facts, LMS enrollment facts, readiness, growth, missingness, and roster reconciliation
For a more analysis-facing view of the same synthetic data ecosystem, see the downstream assessment-intelligence project.
- Code is available under the MIT License.
- Original documentation and generated visual content are available under CC BY 4.0.
- Original synthetic datasets are available under CC BY 4.0.
Third-party materials, trademarks, personal likenesses, and acquired source material are excluded unless explicitly stated otherwise.