MLE-12345 Merge release/2.1.1 into main by rjrudin · Pull Request #642 · marklogic/flux

rjrudin · 2026-05-18T18:34:54Z

No description provided.

MLE-28021 Merge main to develop

Trying to minimize CVEs

Just pointing to releases page to avoid having to update the tag name in future releases.

…ication.

MLE-12345 - Clarifying the use of connectionString with Basic authentication.

…om-develop PDP-1182: Remove per-repo pr-workflow.yaml

No downside to this, and it allows for multiple concurrent jobs using S3 in the same Spark context.

Possibly a temporary addition, going to experiment with this in another project.

Also added support for publishing to GH Packages

…e option is specified (#630) * don't abort on batch write failures if flag is not specified for copy command * address PR comments. Improve test by attempting copy to other db ensuring it is empty prior to copy. * tweak role name * bump copyright header * add test assertion * extract var --------- Co-authored-by: Jason Teh <jteh@cas.org>

Eliminating CVEs

Treating this as a bug fix now and including in 2.1.1.

As noted, this shouldn't be necessary, trying to make Black Duck happy / not dumb.

Trying to un-confuse Black Duck.

Copilot

Pull request overview

This PR merges release 2.1.1 changes into main, updating project versions and adding API/CLI support around S3 anonymous access, basic-auth connection strings, write-option passthroughs, dependency CVE overrides, tests, and documentation.

Changes:

Updates Flux and connector versions to 2.1.1 / 3.1.1 and refreshes related docs/examples.
Adds S3 anonymous access and bucket-scoped S3 Hadoop configuration behavior.
Adds basic-auth connection-string API support, hidden writer passthrough options, dependency overrides, and expanded tests.

Reviewed changes

Copilot reviewed 29 out of 32 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
`gradle.properties`	Bumps Flux and connector versions.
`build.gradle`	Updates dependency resolution overrides.
`flux-cli/build.gradle`	Adds CVE-related dependency constraints and GitHub Packages publishing config.
`.github/workflows/pr-workflow.yaml`	Removes PR validation workflow.
`flux-cli/src/main/java/com/marklogic/flux/impl/S3Params.java`	Adds anonymous S3 auth and bucket-scoped Hadoop config.
`flux-cli/src/main/java/com/marklogic/flux/impl/AbstractCommand.java`	Applies S3 config based on file paths and adds basic connection-string API method.
`flux-cli/src/main/java/com/marklogic/flux/impl/importdata/WriteDocumentParams.java`	Adds hidden arbitrary writer options.
`flux-cli/src/main/java/com/marklogic/flux/impl/importdata/ReadFilesParams.java`	Adds API support for anonymous S3 reads.
`flux-cli/src/main/java/com/marklogic/flux/impl/export/WriteFilesParams.java`	Adds API support for anonymous S3 writes.
`flux-cli/src/main/java/com/marklogic/flux/impl/export/ExportFilesCommand.java`	Adds anonymous S3 support to generic file export API options.
`flux-cli/src/main/java/com/marklogic/flux/impl/custom/CustomImportCommand.java`	Adds anonymous S3 support to custom import read options.
`flux-cli/src/main/java/com/marklogic/flux/impl/custom/AbstractCustomExportCommand.java`	Adds anonymous S3 support to custom export write options.
`flux-cli/src/main/java/com/marklogic/flux/impl/copy/CopyCommand.java`	Adds hidden output writer options and explicit abort-on-failure setting.
`flux-cli/src/main/java/com/marklogic/flux/api/Executor.java`	Adds `connectionStringBasic` to the public API.
`flux-cli/src/main/java/com/marklogic/flux/api/ReadFilesOptions.java`	Adds anonymous S3 API method for readers.
`flux-cli/src/main/java/com/marklogic/flux/api/WriteFilesOptions.java`	Adds anonymous S3 API method for writers.
`flux-cli/src/main/java/com/marklogic/flux/api/GenericFilesExporter.java`	Adds anonymous S3 method to generic export options.
`flux-cli/src/main/java/com/marklogic/flux/api/CustomImporter.java`	Adds anonymous S3 method to custom import options.
`flux-cli/src/main/java/com/marklogic/flux/api/CustomExportWriteOptions.java`	Adds anonymous S3 method to custom export options.
`flux-cli/src/test/java/com/marklogic/flux/impl/S3ParamsTest.java`	Adds S3 anonymous, bucket scoping, and bucket extraction tests.
`flux-cli/src/test/java/com/marklogic/flux/impl/importdata/ImportFilesOptionsTest.java`	Covers hidden write-property passthrough.
`flux-cli/src/test/java/com/marklogic/flux/impl/copy/CopyTest.java`	Adds copy write-failure behavior tests and helper reuse.
`flux-cli/src/test/java/com/marklogic/flux/impl/copy/CopyOptionsTest.java`	Covers output write-property passthrough.
`flux-cli/src/test/java/com/marklogic/flux/api/ConnectionTest.java`	Tests `connectionStringBasic`.
`examples/client-project/build.gradle`	Updates example dependency versions.
`docs/spark-integration.md`	Updates spark-submit jar references.
`docs/import/import-files/selecting-files.md`	Documents S3 anonymous import access.
`docs/export/specifying-path.md`	Documents S3 anonymous export access.
`docs/import/embedder/embedder.md`	Updates default embedding field documentation.
`docs/common-options.md`	Clarifies connection-string auth behavior.
`docs/api.md`	Updates API dependency version and connection API docs.
`CONTRIBUTING.md`	Updates shadow jar examples.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 29 out of 32 changed files in this pull request and generated 10 comments.

+    if (project.hasProperty("ghToken")) {
+      maven {
+        name = "GitHubPackages"
+        url = uri("https://maven.pkg.github.com/rjrudin/flux")


+    if (project.hasProperty("ghToken")) {
+      maven {
+        name = "GitHubPackages"
+        url = uri("https://maven.pkg.github.com/rjrudin/flux")
+        credentials {
+          username = ghActor
+          password = ghToken


+            for (String path : paths) {
+                bucket = extractBucket(path);
+                if (bucket != null) {
+                    break;
+                }
+            }
+        }
+        addToHadoopConfiguration(config, bucket);


+        // Default implementation for backwards compatibility.
+        return connectionString(connectionString);
+    }
+


    ./gradlew shadowJar

-This will produce an assembly jar at `./flux-cli/build/libs/marklogic-flux-2.1-SNAPSHOT-all.jar`.
+This will produce an assembly jar at `./flux-cli/build/libs/marklogic-flux-2.2-SNAPSHOT-all.jar`.


 ```
 $SPARK_HOME/bin/spark-submit --class com.marklogic.flux.spark.Submit \
--master spark://NYWHYC3G0W:7077 flux-cli/build/libs/marklogic-flux-2.1-SNAPSHOT-all.jar \
+--master spark://NYWHYC3G0W:7077 flux-cli/build/libs/marklogic-flux-2.2-SNAPSHOT-all.jar \


 --packages org.apache.hadoop:hadoop-aws:3.3.4,org.apache.hadoop:hadoop-client:3.3.4 \
 --master spark://NYWHYC3G0W:7077 \
-flux-cli/build/libs/marklogic-flux-2.1-SNAPSHOT-all.jar \
+flux-cli/build/libs/marklogic-flux-2.2-SNAPSHOT-all.jar \


+        // useProfile and anonymousAccess have no bucket-scoped equivalent — always set globally.
+        if (anonymousAccess) {
+            config.set(S3A_CREDENTIALS_PROVIDER, "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider");
+        } else if (useProfile) {
            config.set(S3A_CREDENTIALS_PROVIDER, "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider");
        }

+        String prefix = (bucket != null && !bucket.isEmpty()) ? "fs.s3a.bucket." + bucket + "." : "fs.s3a.";
+


+    void addCredentialsWithBucketUsesBucketScopedKeys() {
+        params.setAddCredentials(true);
+        Configuration config = new Configuration();
+        params.addToHadoopConfiguration(config, "my-bucket");


+        Configuration config = new Configuration();
+        params.addToHadoopConfiguration(config, "my-bucket");
+        assertNull(config.get("fs.s3a.access.key"), "Global access key should not be set when bucket is provided");
+        assertNull(config.get("fs.s3a.secret.key"), "Global secret key should not be set when bucket is provided");
+        assertNotNull(config.get("fs.s3a.bucket.my-bucket.access.key"), "Bucket-scoped access key should be set");
+        assertNotNull(config.get("fs.s3a.bucket.my-bucket.secret.key"), "Bucket-scoped secret key should be set");
+    }
+
+    @Test
+    void addCredentialsWithoutBucketUsesGlobalKeys() {
+        params.setAddCredentials(true);
+        Configuration config = new Configuration();
+        params.addToHadoopConfiguration(config);
+        assertNotNull(config.get("fs.s3a.access.key"), "Global access key should be set when no bucket");
+        assertNotNull(config.get("fs.s3a.secret.key"), "Global secret key should be set when no bucket");


rjrudin and others added 16 commits March 18, 2026 14:11

MLE-28021 Merge main to develop, bump to 2.2-SNAPSHOT

b1a32af

Merge pull request #616 from marklogic/feature/temp-main

7b3511d

MLE-28021 Merge main to develop

MLE-28021 Bumping Netty to latest patch version

c241250

Trying to minimize CVEs

MLE-28021 Small docs fix

3463c60

Just pointing to releases page to avoid having to update the tag name in future releases.

MLE-12345 - Clarifying the use of connectionString with Basic authent…

429627b

…ication.

Merge pull request #622 from BillFarber/develop

b1aac14

MLE-12345 - Clarifying the use of connectionString with Basic authentication.

PDP-1182: Remove per-repo pr-workflow.yaml

038ebe0

Merge pull request #626 from marklogic/PDP-1182-remove-pr-workflow-fr…

7e8d48e

…om-develop PDP-1182: Remove per-repo pr-workflow.yaml

MLE-28702 Using bucket-scoped keys for S3

b5b6240

No downside to this, and it allows for multiple concurrent jobs using S3 in the same Spark context.

MLE-28702 Adding hidden prop for generic write options

6c02667

Possibly a temporary addition, going to experiment with this in another project.

MLE-28770 Added S3 anonymous access support

42396c4

Also added support for publishing to GH Packages

MLE-29542 Bumping netty, bouncycastle, and log4j dependencies

bbd6f74

Eliminating CVEs

MLE-28770 Adjusting version for S3 anonymous access support

203fa2b

Treating this as a bug fix now and including in 2.1.1.

MLE-29542 Specifying version for thrift

755d3a9

As noted, this shouldn't be necessary, trying to make Black Duck happy / not dumb.

MLE-29542 Using top-level config for thrift dependency

c0ffb4c

Trying to un-confuse Black Duck.

Copilot AI review requested due to automatic review settings May 18, 2026 18:34

rjrudin requested review from BillFarber and stevebio as code owners May 18, 2026 18:34

Copilot started reviewing on behalf of rjrudin May 18, 2026 18:35 View session

rjrudin changed the title ~~Merge release/2.1.1 into main~~ MLE-12345 Merge release/2.1.1 into main May 18, 2026

Copilot AI reviewed May 18, 2026

View reviewed changes

rjrudin added 2 commits May 18, 2026 15:02

MLE-29542 Added default implementations for new API methods

cff8783

Bumped to 3.1.1

70050b2

rjrudin force-pushed the release/2.1.1 branch from 98cb7b5 to 70050b2 Compare May 18, 2026 19:03

rjrudin requested a review from Copilot May 18, 2026 19:03

Copilot started reviewing on behalf of rjrudin May 18, 2026 19:03 View session

Merge branch 'main' into release/2.1.1

a3c68db

Copilot AI reviewed May 18, 2026

View reviewed changes

rjrudin merged commit 5691211 into main May 18, 2026
4 checks passed

rjrudin deleted the release/2.1.1 branch May 18, 2026 19:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MLE-12345 Merge release/2.1.1 into main#642

MLE-12345 Merge release/2.1.1 into main#642
rjrudin merged 19 commits into
mainfrom
release/2.1.1

rjrudin commented May 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

rjrudin commented May 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants