feat(sourcedb-to-spanner): Add uniform splitting support for extended…#4028
feat(sourcedb-to-spanner): Add uniform splitting support for extended…#4028jsuhani-2026 wants to merge 2 commits into
Conversation
… PostgreSQL primary keys.
210dc2c to
2b231d2
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4028 +/- ##
============================================
+ Coverage 55.68% 63.60% +7.92%
+ Complexity 7236 2548 -4688
============================================
Files 1124 535 -589
Lines 68447 30900 -37547
Branches 7721 3450 -4271
============================================
- Hits 38117 19655 -18462
+ Misses 27856 10223 -17633
+ Partials 2474 1022 -1452
🚀 New features to boost your workflow:
|
|
Warning Gemini encountered an error creating the summary. You can try again by commenting |
|
/gemini summary |
Summary of ChangesThis pull request enhances the uniform splitting functionality for PostgreSQL migrations by extending support to a wider range of primary key data types. It introduces necessary logic for parsing and splitting these types, updates the dialect adapter to handle optimized boundary queries for non-standard types, and significantly improves test coverage to ensure reliability across various data types. Highlights
Activity
|
|
/gemini review |
1 similar comment
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for partitioning and splitting tables by LOCAL_TIME, OFFSET_TIME, and BIT column types in the sourcedb-to-spanner template. It implements boundary extraction, splitting logic, and prepared statement parameter mapping for these types, along with comprehensive integration tests. The review feedback suggests optimizing the parsing of PostgreSQL time and offset time bytes by checking for binary formats first, which avoids unnecessary UTF-8 string decoding and regex matching on binary payloads.
| // "t_time_pk", | ||
| // "t_time_without_time_zone_pk", | ||
| // "t_timetz_pk", | ||
| // "t_time_with_time_zone_pk", |
There was a problem hiding this comment.
why are these commented out?
There was a problem hiding this comment.
Right now, reading time and timetz types from the source isn't supported, so their values get extracted as null. Since these are primary key columns, the null values cause those rows to be routed straight to the DLQ instead of migrating. I've commented them out so the assertion for the tables in this unsupported list doesn't fail.
| System.setProperty("numWorkers", "20"); | ||
| Map<String, String> jobParameters = new HashMap<>(); | ||
| jobParameters.put("numPartitions", "100"); | ||
| jobParameters.put("numPartitions", "10"); |
There was a problem hiding this comment.
did we debug why higher partitions are causing issue ? Can we run a test using cloudsql instances with 100 partitions locally to see that this is an issue due to small mysql container?
1b70125 to
c681d4a
Compare
- Added 24:00:00 and non-zero timezone test cases for time/timetz types. - Centralized the 'bit' type literal into JdbcCommonConstants.BIT_TYPE. - Optimized time/timetz parsing by checking for binary payload format first.
c681d4a to
3b390f5
Compare
2 participants
Expands Uniform Splitting for PostgreSQL migrations by adding support for several new primary key data types: time, timetz, float, double, decimal, bit, and uuid.
Updated PostgreSQLDataTypesIT and PostgreSQLDataTypesPGDialectIT , to cover all supported primary key types (integers, strings, dates, timestamps, numerics, floats, etc.), addressing a previous gap in our test coverage.