This PR contains the the following documentation changes for the native docker image:
in the docker/README.md: How to build, release and promote the native docker image.
in the tests/README.md: How to run system tests by bringing up kafka in the native mode.
added docker/native/README.md
added html changes for the kafka-site
added native docker image support in the docker compose files examples.
Testing:
Tested all the docker compose files with both the docker images - jvm and native
Tested the html changes locally with the kafka-site
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Vedarth Sharma <vesharma@confluent.io>
Update release.py script:
- fix typo
- add more info about closing repository in stagingRepository
Reviewers: Mickael Maison <mickael.maison@gmail.com>
1. Permissions for mkdir set incorrectly, probably because it used incorrect python3 syntax (octals should use 0o prefix).
2. JAVA_HOME logic didn't seem to hold up to its promise. The original logic was broken when user input was empty. It was supposed to use JAVA_HOME system property to find java, but it wouldn't set jdk_java_home, so the following version check
java_version = cmd_output("%s/bin/java -version" % jdk_java_home, env=jdk_env)
would access /bin/java which does not exist on any system I know.
Reviewer: Bruno Cadonna <cadonna@apache.org>
ssh and rsync access has been removed from home.apache.org.
Removing the commands from release.py and replacing them with a note to make sure they are manually uploaded with an sftp client instead.
Reviewers: David Jacot <djacot@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Java 17 is at release candidate stage and it will be a LTS release once
it's out (previous LTS release was Java 11).
Details:
* Replace Java 16 with Java 17 in Jenkins and Readme.
* Replace `--illegal-access=permit` (which was removed from Java 17)
with `--add-opens` for the packages we require internal access to.
Filed KAFKA-13275 for updating the tests not to require `--add-opens`
(where possible).
* Update `release.py` to use JDK8. and JDK 17 (instead of JDK 8 and JDK 15).
* Removed all but one Streams test from `testsToExclude`. The
Connect test exclusion list remains the same.
* Add notable change to upgrade.html
* Upgrade to Gradle 7.2 as it's required for proper Java 17 support.
* Upgrade mockito to 3.12.4 for better Java 17 support.
* Adjusted `KafkaRaftClientTest` and `QuorumStateTest` not to require
private access to `jdk.internal.util.random`.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
This, upgrades JDK to version 15 for the docs generation, this way we
can circumvent bug https://bugs.openjdk.java.net/browse/JDK-8215291
present in JDK11
Reviewers: Ismael Juma <ismael@juma.me.uk>
Deprecates the following
1. StreamsConfig.EXACTLY_ONCE
2. StreamsConfig.EXACTLY_ONCE_BETA
3. Producer#sendOffsetsToTransaction(Map offsets, String consumerGroupId)
And introduces a new StreamsConfig.EXACTLY_ONCE_V2 config. Additionally, this PR replaces usages of the term "eos-beta" throughout the code with the term "eos-v2"
Reviewers: Matthias J. Sax <mjsax@confluent.io>
* Run all JDK/Scala version combinations for trunk/release branch builds.
* Only retry failures in PR builds for now (we can remove this distinction if/when
we report flaky failures as described in KAFKA-12216).
* Disable concurrent builds
* Send email to dev list on build failure
* Use triple double quotes in `doValidation` since we use string interpolation
for `SCALA_VERSION`.
* Update release.py to output new `Unit/integration tests` Jenkins link
Reviewers: Gwen Shapira <cshapi@gmail.com>, David Arthur <mumrah@gmail.com>
Java 11 generates html5 pages with search support, which provides a better user experience.
Fixed `get_jdk` bugs found during testing and updated `release.py` blurb to indicate that
both JDK 8 and JDK 11 are required to perform a release.
Tested by running `python2 release.py stage-docs`, which triggers the `aggregateJavadoc`
path without some of the undesired (for testing) release steps.
Reviewers: John Roesler <vvcephei@apache.org>
Gradle 7.0 is required for Java 16 compatibility and it removes a number of
deprecated APIs. Fix most issues preventing the upgrade to Gradle 7.0.
The remaining ones are more complicated and should be handled
in a separate PR. Details of the changes:
* Release tarball no longer includes includes test, sources, javadoc and test sources jars (these
are still published to the Maven Central repository).
* Replace `compile` with `api` or `implementation` - note that `implementation`
dependencies appear with `runtime` scope in the pom file so this is a (positive)
change in behavior
* Add missing dependencies that were uncovered by the usage of `implementation`
* Replace `testCompile` with `testImplementation`
* Replace `runtime` with `runtimeOnly` and `testRuntime` with `testRuntimeOnly`
* Replace `configurations.runtime` with `configurations.runtimeClasspath`
* Replace `configurations.testRuntime` with `configurations.testRuntimeClasspath` (except for
the usage in the `streams` project as that causes a cyclic dependency error)
* Use `java-library` plugin instead of `java`
* Use `maven-publish` plugin instead of deprecated `maven` plugin - this changes the
commands used to publish and to install locally, but task aliases for `install` and
`uploadArchives` were added for backwards compatibility
* Removed `-x signArchives` line from the readme since it was wrong (it was a
no-op before and it fails now, however)
* Replaces `artifacts` block with an approach that works with the `maven-publish` plugin
* Don't publish `jmh-benchmark` module - the shadow jar is pretty large and not
particularly useful (before this PR, we would publish the non shadow jars)
* Replace `version` with `archiveVersion`, `baseName` with `archiveBaseName` and
`classifier` with `archiveClassifier`
* Update Gradle and plugins to the latest stable version (7.0 is not stable yet)
* Use `plugin` DSL to configure plugins
* Updated notable changes for 3.0
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Randall Hauch <rhauch@gmail.com>
Scala 2.13.2 introduces support for suppressing warnings,
which makes it possible to enable fatal warnings. This is
useful enough from a development perspective to justify
this change.
In addition, Scala 2.13.2 also has a Vector implementation
with significant performance improvements and encoding
of String matches to switches.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
* Introduce `gradlewAll` script to replace `*All` tasks since the approach
used by the latter doesn't work since Gradle 6.0 and it's unclear when,
if ever, it will work again ( see https://github.com/gradle/gradle/issues/11301).
* Update release script and README given the above.
* Update zinc to 1.3.5.
* Update gradle-versions-plugin to 0.28.0.
The major improvements in Gradle 6.0 to 6.3 are:
- Improved incremental compilation for Scala
- Support for Java 14 (although some Gradle plugins
like spotBugs may need to be updated or disabled,
will do that separately)
- Improved scalac reporting, warnings are clearly
marked as such, which is very helpful.
Tested `gradlewAll` manually for the commands listed in the README
and release script. For `uploadArchive`, I tested it with a local Maven
repository.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
When building a release candidate with release.py, if it's not the first RC, we need to drop the previous RC's artifacts from the staging repository before closing the new ones. This adds a log message to remind the release manager of this
Author: Manikumar Reddy <manikumar.reddy@gmail.com>
Reviewers: Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>
Closes#8072 from omkreddy/release-script
Rather than using sed, use built-in Python regular expressions to strip
the SNAPSHOT expression from the pom.xml files. Sed has different flags
on different platforms, such as Linux. Using Python directly here is
more compatible, as well as being more efficient, and not requiring an
rm command afterwards.
When running release_notes.py, use the current Python interpreter.
This is needed to prevent attempting to run release_notes.py with
Python 3 on some systems. release_notes.py will not (yet) work with
Python 3.
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Magnus Edenhill <magnus@edenhill.se>, David Arthur <mumrah@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
Closes#6198 from cmccabe/release_py
Scala 2.12 has better support for newer Java versions and includes additional
compiler warnings that are helpful during development. In addition, Scala 2.11
hasn't been supported by the Scala community for a long time, the soon to be
released Spark 2.4.0 will finally support Scala 2.12 (this was the main reason
preventing many from upgrading to Scala 2.12) and Scala 2.13 is at the RC stage.
It's time to start recommending the Scala 2.12 build as we prepare support for
Scala 2.13 and start thinking about removing support for Scala 2.11.
In the meantime, Jenkins will continue to build all supported Scala versions (including
Scala 2.11) so the PR and trunk jobs will fail if people accidentally use methods
introduced in Scala 2.12.
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>
Just a doc change
Author: John Eismeier <john.eismeier@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#4573 from jeis2497052/trunk
* Set --source, --target and --release to 1.8.
* Build Scala 2.12 by default.
* Remove some conditionals in the build file now that Java 8
is the minimum version.
* Bump the version of Jetty, Jersey and Checkstyle (the newer
versions require Java 8).
* Fixed issues uncovered by the new version if Checkstyle.
* A couple of minor updates to handle an incompatible source
change in the new version of Jetty.
* Add dependency to jersey-hk2 to fix failing tests caused
by Jersey upgrade.
* Update release script to use Java 8 and to take into account
that Scala 2.12 is now built by default.
* While we're at it, bump the version of Gradle, Gradle plugins,
ScalaLogging, JMH and apache directory api.
* Minor documentation updates including the readme and upgrade
notes. A number of Streams Java 7 examples can be removed
subsequently.
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>
Closes#3917 from ewencp/stage-docs
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Damian Guy <damian.guy@gmail.com>, Ismael Juma <github@juma.me.uk>
Closes#4528 from ewencp/update-release-script
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#4054 from guozhangwang/KMinor-pre-1.0-release
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#3846 from dguy/minor-release-script
Author: Ewen Cheslack-Postava <me@ewencp.org>
Reviewers: Gwen Shapira
Closes#2795 from ewencp/release-script and squashes the following commits:
f1d0590 [Ewen Cheslack-Postava] Don't expose promotion to the user since it is not implemented yet.
1a6947a [Ewen Cheslack-Postava] Handle cleanup if there's a failure during generation of release notes.
fa58401 [Ewen Cheslack-Postava] Fix hard-coded uses of trunk
639bcca [Ewen Cheslack-Postava] Try to cleanup after most failures.
a3a7245 [Ewen Cheslack-Postava] Fix SCRIPT_DIR to be an absolute path so git clones against the REPO work when it is also your cwd
de54c97 [Ewen Cheslack-Postava] Load/save preferences in a .release-settings.json file so you don't have to keep entering the same info repeatedly
b559a61 [Ewen Cheslack-Postava] Check that the user doesn't have any oustanding diffs before starting the rest of the script
ff0b330 [Ewen Cheslack-Postava] Store original starting branch to switch back to instead of using a default.
b793562 [Ewen Cheslack-Postava] Use 2.12 instead of specific Scala version so we use the default 2.12 version.
382b7f9 [Ewen Cheslack-Postava] MINOR: Add a release script that helps generate release candidates.
Krishna Agarwal
Manikumar Reddy
Matthias Berndt
Owen Leung
Vedarth Sharma
iit2009060
Gantigmaa Selenge
Divij Vaidya
Tanay Karmarkar
Yi-Sheng Lien
Luke Chen
Lucas Brutschy
David Jacot
Mickael Maison
Konstantine Karantasis
Ismael Juma
Josep Prat
A. Sophie Blee-Goldman
Bill Bejeck
Randall Hauch
John Roesler
David Arthur
Matthias J. Sax
Colin P. Mccabe
Dong Lin
Bibin Sebastian
John Eismeier
Rajini Sivaram
Ewen Cheslack-Postava
Guozhang Wang
Damian Guy