This commit implements KIP-899: Allow producer and consumer clients to rebootstrap. It introduces the new setting `metadata.recovery.strategy`, applicable to all the types of clients.
Reviewers: Greg Harris <gharris1727@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Add the support for DescribeTopicPartitions API to AdminClient. For this initial implementation, we are simply loading all of the results into memory on the client side.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Artem Livshits <alivshits@confluent.io>, David Arthur <mumrah@gmail.com>
This is one of the steps required for kafka to compile with Java 21.
For each case, one of the following fixes were applied:
1. Suppress warning if fixing would potentially result in an incompatible change (for public classes)
2. Add final to one or more methods so that the escape is not possible
3. Replace method calls with direct field access.
In addition, we also fix a couple of compiler warnings related to deprecated references in the `core` module.
See the following for more details regarding the new lint warning:
https://www.oracle.com/java/technologies/javase/21-relnote-issues.html#JDK-8015831
Reviewers: Divij Vaidya <diviv@amazon.com>, Satish Duggana <satishd@apache.org>, Chris Egerton <chrise@aiven.io>
The GRACEFUL_SHUTDOWN_TIMEOUT_MS for the Trogdor JsonRestServer is 100ms.
In heavily loaded CI environments, this timeout can be exceeded. When this happens,
it causes the jettyServer.stop() and jettyServer.destroy() calls to throw exceptions, which
prevents shutdownExecutor.shutdown() from running. This has the effect of causing the JsonRestServer::waitForShutdown method to block for 1 day, which exceeds the 120s
timeout on the CoordinatorTest (and any other test relying on MiniTrogdorCluster).
This change makes it such that the graceful shutdown timeout is less likely to be exceeded,
and when it is, the timeout does not cause the waitForShutdown method to block for much
longer than the graceful shutdown timeout.
Reviewers: Ismael Juma <ismael@juma.me.uk>
MiniTrogdorCluster spins up agents from a different thread when scheduling them, but does not use volatiles in these objects. It's not clear that the updated fields are visible.
Reviewers: Colin P. McCabe <cmccabe@apache.org>, Kvicii <Karonazaba@gmail.com>, David Jacot <djacot@confluent.io>
The trogdor ConsumeBenchWorker allows several consumption tasks to be run in parallel, the number is configurable using the threadsPerWorker config. If one of the consumption tasks completes executing successfully due to maxMessages being consumed, then, the consumption task prematurely notifies the doneFuture causing the entire ConsumeBenchWorker to halt. This becomes a problem when more than 1 consumption task is running in parallel, because the successful completion of 1 of the tasks shuts down the entire worker while the other tasks are still running. When the worker is shut down, it kills all the active consumption tasks, though they have not consumed maxMessages yet. This commit defers notification of the doneFuture to the CloseStatusUpdater thread, which is already responsible for tracking the status of the tasks and updating their status when all of the tasks complete.
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
* Changes the new Throughput Generators to track messages per window
instead of making per-second calculations which can have rounding errors.
Also, one of these had a calculation error which prompted this change in
the first place.
* Fixes a couple typos.
* Fixes an error where certain JSON fields were not exposed, causing the
workloads to not behave as intended.
* Fixes a bug where we use wait not in a loop, which exits too quickly.
* Adds additional constant payload generators.
* Fixes problems with an example spec.
* Fixes several off-by-one comparisons.
Reviewers: Colin P. McCabe <cmccabe@apache.org>
Move Trogdor out of tools and into its own gradle module. This allows us to minimize
the dependencies of the tools module. We still keep Trogdor in the CLASSPATH
created by kafka-run-class.sh.
Reviewers: Colin P. McCabe <cmccabe@apache.org>
Ivan Yurchenko
Chia Chuan Yu
Calvin Liu
Mickael Maison
David Mao
Divij Vaidya
Greg Harris
shuoer86
Ismael Juma
hudeqi
Christo Lolov
Lucas Bradstreet
Jorge Esteban Quilcate Otoya
Kevin Zhang
dengziming
Kowshik Prakasam
Scott Hendricks
Shay Elkin