Repository navigation
docs: bring the README current and add CI, NOTICE and changelog entries - #5
Merged
Merged
Conversation
The README still described the connector as a Spark 3.2.1 fork, told readers to depend on com.roncemer.spark:spark-sql-kinesis_2.13:1.2.2_spark-3.2, and gave developer-setup steps that clone roncemer's repository and produce a 1.2.2 jar. It also carried a "Deploying to Maven Central" section describing OSSRH: applying for a Sonatype JIRA account, proving domain ownership via a DNS TXT record, and closing a staging repository at s01.oss.sonatype.org. That host is retired and the namespace is verified by GitHub account ownership, so following those steps could only fail. PUBLISHING.md is now the single source for releasing. Rewritten around what a reader needs first: which Spark and Java each connector version requires and why the Spark version is part of the artifact version, the io.github.technobasant coordinates for sbt, Maven and spark-submit, and the fact that the published jar is shaded and relocated. Added a Design notes section covering the synchronous-fetch read model and why DSv2 has not been adopted, deaggregation without KCL and the parent-sequence-number constraint that protects checkpoints, and where shard state lives. The option tables are kept and were checked against KinesisSourceProvider. The roadmap now names the real known gaps around resharding rather than listing DSv2 migration as simply pending. CI: .travis.yml is replaced with a GitHub Actions workflow. Travis was configured for JDK 8 on dist trusty and ran `mvn -DskipTests clean install`, so it neither built on a supported JDK nor executed a single test; effectively the repository had no CI. That is how a pom containing "--" inside an XML comment reached a commit. The workflow runs `-Prelease verify` rather than `test`, because the steps that break are the ones after the tests — shading, the sources and javadoc jars, and the published pom — and asserts the packaging invariants that unit tests cannot see: the DataSourceRegister entry resolves, no com.amazonaws classes are present, nothing bundled is left unrelocated, and the sources jar contains the Scala files and not only the Java ones. Each of those has been broken at some point during this port. NOTICE: every source file carries the ASF header referring to "the NOTICE file distributed with this work", but no such file existed. Added, crediting Qubole as the original author, Ron Cemer for the Spark 3.2 port, and the Amazon libraries whose deaggregation semantics and wire format AggregatorUtil reproduces. CHANGELOG: records 1.3.0_spark-4.2.0, and 1.2.6_spark-3.4.1 retrospectively, which was never logged because it only ever shipped as a hand-built jar.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README still described the connector as a Spark 3.2.1 fork, told readers to depend on com.roncemer.spark:spark-sql-kinesis_2.13:1.2.2_spark-3.2, and gave developer-setup steps that clone roncemer's repository and produce a 1.2.2 jar. It also carried a "Deploying to Maven Central" section describing OSSRH: applying for a Sonatype JIRA account, proving domain ownership via a DNS TXT record, and closing a staging repository at s01.oss.sonatype.org. That host is retired and the namespace is verified by GitHub account ownership, so following those steps could only fail. PUBLISHING.md is now the single source for releasing.
Rewritten around what a reader needs first: which Spark and Java each connector version requires and why the Spark version is part of the artifact version, the io.github.technobasant coordinates for sbt, Maven and spark-submit, and the fact that the published jar is shaded and relocated. Added a Design notes section covering the synchronous-fetch read model and why DSv2 has not been adopted, deaggregation without KCL and the parent-sequence-number constraint that protects checkpoints, and where shard state lives. The option tables are kept and were checked against KinesisSourceProvider. The roadmap now names the real known gaps around resharding rather than listing DSv2 migration as simply pending.
CI: .travis.yml is replaced with a GitHub Actions workflow. Travis was configured for JDK 8 on dist trusty and ran
mvn -DskipTests clean install, so it neither built on a supported JDK nor executed a single test; effectively the repository had no CI. That is how a pom containing "--" inside an XML comment reached a commit. The workflow runs-Prelease verifyrather thantest, because the steps that break are the ones after the tests — shading, the sources and javadoc jars, and the published pom — and asserts the packaging invariants that unit tests cannot see: the DataSourceRegister entry resolves, no com.amazonaws classes are present, nothing bundled is left unrelocated, and the sources jar contains the Scala files and not only the Java ones. Each of those has been broken at some point during this port.NOTICE: every source file carries the ASF header referring to "the NOTICE file distributed with this work", but no such file existed. Added, crediting Qubole as the original author, Ron Cemer for the Spark 3.2 port, and the Amazon libraries whose deaggregation semantics and wire format AggregatorUtil reproduces.
CHANGELOG: records 1.3.0_spark-4.2.0, and 1.2.6_spark-3.4.1 retrospectively, which was never logged because it only ever shipped as a hand-built jar.