feat(typegen): add bigint_as option for int8/numeric TypeScript generation#1083
feat(typegen): add bigint_as option for int8/numeric TypeScript generation#1083Maliik-B wants to merge 2 commits into
Conversation
avallete
left a comment
There was a problem hiding this comment.
Hi there ! Thank you for your contribution !
I would like to have this paired with an "e2e" test over the postgrest-js api, we already have bigint columns in the database for it:
Within it's own file similar to: https://github.com/supabase/supabase-js/blob/master/packages/core/postgrest-js/test/basic.test.ts
Idea would be to demonstrate that this type adjustment actually match the postgrest behavior at runtime, including how the bigint type is used.
Right now the way I see it, this type can only work if there is a casting over the columns from Postgrest. Also what happen on "update" ? Would be good to demonstrate that this type adjustment actually fix the original problem (precision loss) and match the runtime behaviour.
|
Thanks for the review, and for pointing at the existing bigint fixtures and basic.test.ts. You're right that on reads the generated type only lines up at runtime once the int8/numeric columns are cast (e.g. ::text), since PostgREST returns an un-cast int8 as a lossy JSON number. That read-side casting dependency is why the option is opt-in and defaults to number. Writes are cleaner: passed as a string, the value goes in losslessly with no cast. Happy to demonstrate both end to end rather than leave it at the type level. My plan for the e2e, as its own file under packages/core/postgrest-js/test/ against the existing bigint columns:
Does that line up with what you had in mind, or would you rather I scope it differently (for example just the supported cast path)? |
|
Sounds good, minor nitpick, I think the value should be set in postgres to test the "read" with both (with and without cast). |
|
Opened the e2e as supabase/supabase-js#2461 (postgrest-js). It seeds 9223372036854775807 (2^63 - 1) in Postgres and exercises read and write through the API:
So the runtime does handle these values: a ::text cast on read, and a string (or bigint) on write. That is what bigint_as makes the generated types reflect. It stays opt-in and defaults to number, matching the un-cast read, so nothing changes for anyone who has not opted in. bigint_as=string makes the Insert/Update column string (the lossless write form), and the Row column string for the cast-on-read path the consumer has arranged. The test also carries a @ts-expect-error on the string write: with the default number type the lossless string form is a type error today, which is the gap the option closes. Does that cover what you wanted to see on the runtime side? Happy to adjust the test or the option scope from here. |
|
Cross posting here: supabase/supabase-js#2461 (review) |
Per the read/write model worked out on supabase/supabase-js#2461: the Row stays number (an un-cast int8 is a lossy JSON number, and ::text already infers string on the casted column), and only Insert/Update widen to the bigint_as union. The option is now a pipe-split string accepting number and bigint; string is dropped, since an arbitrary non-numeric value would pass the type check and fail only at runtime in PostgREST.
Addresses #1078.
Problem
int8andnumericare generated as TypeScriptnumber, but values pastNumber.MAX_SAFE_INTEGER(2^53) are lossy once round-tripped through JSON. Consumers using snowflake/Instagram-style sharded IDs orxxhash64ETL output hit this in production: a row fetched by id gets a corrupted id back, and/items/<id>404s.Approach
Adds an opt-in
bigint_asoption for the TypeScript generator that widens the write types forint8/numericcolumns. It defaults tonumber, so existing consumers are unaffected.The key change from the first revision: the option is applied per direction, not as a single scalar.
number, always. PostgREST returns an un-castint8as a JSON number, so precision is already gone by the timeJSON.parseruns. Typing the Row as anything else would misrepresent what actually comes over the wire. Consumers who need the exact value cast to::textand get astringback, and the select-query parser already infersstringfor the casted column.bigint_asunion. This is the lossless input channel: a JSBigIntis serialized by postgrest-js to a JSON string, so values past 2^53 go in intact. Widening the input type is pure back-compat (it only accepts more).bigint_asis a pipe-delimited set passed as a single string, so the union is spelled inline, for examplebigint_as=number|bigint. Accepted tokens arenumberandbigint; the recommended value isnumber|bigint(bigintfor the lossless path,numberfor the ergonomic small-value case). Unknown tokens are ignored and an empty result falls back tonumber.stringis deliberately not accepted: an arbitrary non-numeric string would pass the type check and only fail at runtime in PostgREST. The documented pattern for the read-then-write case is to wrap the cast read value,update({ ... }).eq('id', BigInt(entity.id)).Surfaced two ways, matching the existing generator-option conventions:
PG_META_GENERATE_TYPES_BIGINT_AS(standalone generation path), alongsidePG_META_GENERATE_TYPES_INCLUDED_SCHEMAS,_DEFAULT_SCHEMA, etc.?bigint_as=onGET /generators/typescript, alongsidedetect_one_to_one_relationshipsandpostgrest_version.The mapping is scoped exactly as mandarini pointed out on the issue: only
int8andnumericare split out of thenumberbranch.int2,int4,float4, andfloat8all fit safely in a JSnumberand are left untouched.Runtime evidence
The read/write model above is demonstrated end to end in supabase/supabase-js#2461 (postgrest-js), which seeds
9223372036854775807(2^63 - 1) in Postgres and exercises the API:9223372036854776000,::textread returns the exact string"9223372036854775807",BigInt) round-trips losslessly,So the runtime already handles these values on both paths, and
bigint_asmakes the generated write types reflect the lossless one.Scope
TypeScript template only, and within it the table/view
Insert/Updatetypes. Row types and function returns/arguments are left asnumber. The sibling templates (go.ts,python.ts,swift.ts) that mandarini flagged as parallel work are intentionally left for a follow-up once the option shape is settled, to keep this PR reviewable.Tests
Three focused cases in
test/server/typegen.ts, using the existing fixtures (todos."user-id"isint8, thedays_since_eventcomputed field isnumeric):bigint_asdefaults tonumber: back-compat,"user-id": number(Row) and"user-id"?: number(Insert/Update).bigint_as=number|bigint:"user-id"?: number | biginton Insert/Update, while the Row stays"user-id": numberanddays_since_event: number | null.bigint_as=bigint:"user-id"?: biginton Insert/Update, Row unchanged.Each case asserts the read/write asymmetry directly (the Row assertions are the regression guard that reads are never widened). Used
toContain(as the schema-filter tests already do) rather than full snapshots, to keep the surgical behavior of the option legible. Happy to convert to inline snapshots if you would prefer consistency with the other typegen cases.Docs
No README change: the README documents only the DB connection vars, none of the existing generator options (
INCLUDED_SCHEMAS,DEFAULT_SCHEMA,SWIFT_ACCESS_CONTROL) live there, and the route uses inline types rather than a Fastify JSON schema, so there is no OpenAPI entry to update. Glad to add docs wherever you would want them.Thanks @mandarini for transferring the issue with the
typescript.ts:889pointer and the wire-format context, and @avallete for working through the read/write direction on #2461. That is what settled this into a contained, back-compatible change. Happy to adjust direction on any of the above.