Repository navigation
A measured evaluation of the Schema Change pipeline, and a stand-in model server - #14
Merged
Merged
Conversation
…bases, and a stand-in model server scripts/schema-change-eval.ts builds PostgreSQL databases of 50, 300 and 1,000 tables with real foreign keys and populated tables, writes change requests in the words a person would use (typos, partial names, a column name only), and measures table selection, plan checks against the real catalog, lint, execution of the runbook statement by statement and the result, refusal of invalid requests, and the latency of every stage that does not wait for a model. The plan is the reference plan unless --live is given, so nothing in the numbers says anything about a model. scripts/lib/synthetic-schema.ts generates the schemas (with a test, which vitest now picks up from scripts/). scripts/fake-model-server.ts is a stand-in for the OpenAI Responses endpoint to try the page without a key. npm run eval:schema.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Four files the README already described but the repository never had, plus a test config line.
scripts/schema-change-eval.ts(npm run eval:schema): builds PostgreSQL databases of 50, 300 and 1,000 tables with real foreign keys and populated tables, writes change requests in the words a person would use (typos, partial names, a column name only), and measures table selection, plan checks against the real catalog, lint, execution of the runbook statement by statement and whether it leaves the schema as asked, refusal of invalid requests, and the latency of every stage that does not wait for a model. The planner is the reference plan unless--liveis given, so the numbers say nothing about a model's quality, and the model's own latency and tokens are reported as not measured.scripts/lib/synthetic-schema.tsgenerates the schemas;scripts/lib/synthetic-schema.test.tstests them.vitest.config.tsnow includesscripts/**/*.test.tsso the repo suite runs it.scripts/fake-model-server.ts: a stand-in for the OpenAI Responses endpoint, to try the Schema Change page without a key (the header says how to point the API at it). It does no thinking and says nothing about what a real model would answer.package.json: one line,eval:schema. The README's sentence that the evaluation "is not in this repository yet" is replaced by how to run it.How it was checked
npm run eval:schema -- --sizes 50 --per-size 14 --negative 6): 14 requests and 6 that must be refused; every table found 100% (14/14, scope median 4 of 50 tables, 6x smaller than sending every table), plans valid with the right tables 100%, invalid requests refused with the expected reason 6/6, lint 100%, every runbook ran on PostgreSQL 16 and left the schema as asked (14/14). That includes the cases the newer compiler handles differently (the column rename, the lengthened varchar, integer to bigint).Reviewer notes