Adding support for custom data generation using JAR#4024
Adding support for custom data generation using JAR#4024shreyakhajanchi wants to merge 3 commits into
Conversation
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (76.47%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #4024 +/- ##
============================================
+ Coverage 55.70% 55.75% +0.04%
+ Complexity 7225 6797 -428
============================================
Files 1125 1125
Lines 68425 68541 +116
Branches 7717 7729 +12
============================================
+ Hits 38119 38215 +96
- Misses 27835 27845 +10
- Partials 2471 2481 +10
🚀 New features to boost your workflow:
|
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request enables advanced data seeding by allowing users to override the default randomized data generation with custom, business-logic-driven implementations. By providing a GCS path to a JAR and a class name, users can now inject deterministic or stateful data into their pipelines without modifying the core template code, significantly increasing the flexibility of the CDC data generator. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for custom data generation in the CDC data generator template by adding customJarPath and customClassName parameters. It defines the CustomDataGenerator interface, implements a dynamic class-loading utility (CustomDataGeneratorFetcher), and integrates this custom generator across key data generation components and DoFns. The review feedback highlights two important improvements: first, explicitly specifying a parent classloader when instantiating URLClassLoader to prevent classloading issues in Apache Beam/Dataflow environments; and second, logging the error before throwing a RuntimeException during class loading to ensure visibility in Cloud Logging.
| key -> { | ||
| try { | ||
| URL[] urls = JarFileReader.saveFilesLocally(customJarPath); | ||
| URLClassLoader classLoader = URLClassLoader.newInstance(urls); |
There was a problem hiding this comment.
In Apache Beam/Dataflow environments, user code and template dependencies are often loaded using a custom worker classloader rather than the system classloader. Creating a URLClassLoader without specifying a parent classloader defaults to using the system classloader as parent. This can lead to NoClassDefFoundError or ClassCastException when the dynamically loaded class tries to reference or cast to the CustomDataGenerator interface.
To prevent this, explicitly pass the classloader of CustomDataGeneratorFetcher as the parent classloader.
| URLClassLoader classLoader = URLClassLoader.newInstance(urls); | |
| URLClassLoader classLoader = URLClassLoader.newInstance(urls, CustomDataGeneratorFetcher.class.getClassLoader()); |
| } catch (Exception e) { | ||
| throw new RuntimeException( | ||
| "Failed to load CustomDataGenerator from " | ||
| + customJarPath | ||
| + " class: " | ||
| + customClassName, | ||
| e); | ||
| } |
There was a problem hiding this comment.
When throwing exceptions in environments where the caller or global exception handler might not log them (such as certain Dataflow templates), log the error message before throwing the exception to ensure the failure is recorded in Cloud Logging.
} catch (Exception e) {
LOG.error(
"Failed to load CustomDataGenerator from {} class: {}",
customJarPath,
customClassName,
e);
throw new RuntimeException(
"Failed to load CustomDataGenerator from "
+ customJarPath
+ " class: "
+ customClassName,
e);
}References
- When throwing exceptions in environments where the caller or global exception handler might not log them (such as certain Dataflow templates), log the error message before throwing the exception to ensure the failure is recorded.
2 participants
Description
This PR introduces the ability to override standard randomized data generation with custom user-defined logic via external JARs.
Key Features
CustomDataGeneratorinterface that developers can implement to supply explicit values for specific tables and columns, allowing for complex, stateful, or deterministic data generation scenarios.Impact
This feature unlocks advanced testing and data seeding capabilities, enabling users to inject specific business-logic driven data into their databases without altering the core template code.