This avoids `gradle dependencyCheckAggregate` from reporting on
advisories in build-time dependencies (e.g. CVE-2023-46122) which
typically should not affect us.
I checked that this does not prevent advisories in 'regular'
dependencies from being reported (but there currently are none).
Reviewers: Josep Prat <josep.prat@aiven.io>
The format of the 'describe topic' output was changed as part of KAFKA-15585 which required an update in the parsing logic used by system tests.
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
Some services are not closed, so our CI print following error.
org.opentest4j.AssertionFailedError: Found 16 unexpected threads during @BeforeAll: `kafka-admin-client-thread | adminclient-287,kafka-admin-client-thread | adminclient-276,kafka-admin-client-thread | adminclient-271,kafka-admin-client-thread | adminclient-293,kafka-admin-client-thread | adminclient-281,kafka-admin-client-thread | adminclient-302,kafka-admin-client-thread | adminclient-334,kafka-admin-client-thread | adminclient-323,kafka-admin-client-thread | adminclient-257,kafka-admin-client-thread | adminclient-336,kafka-admin-client-thread | adminclient-308,kafka-admin-client-thread | adminclient-263,kafka-admin-client-thread | adminclient-273,kafka-admin-client-thread | adminclient-278,kafka-admin-client-thread | adminclient-283,kafka-admin-client-thread | adminclient-317` ==> expected: <true> but was: <false>
#15679 use AfterEach to release service. However, the test cases having multi consumerConfigs will create a lot of services in testing. Hence, the intermediate servers are not closed.
Reviewers: Luke Chen <showuon@gmail.com>
Added the check before the reassignment occurs and we start bouncing brokers.
Reviewers: David Mao <dmao@confluent.io>, David Jacot <djacot@confluent.io>
Even if the log start offset is updated, the log deletion might still not completed. Making the test more robust.
Reviewers: Luke Chen <showuon@gmail.com>
If there are some logs to be deleted during the log dir movement, we'll send for a scheduler to do the deletion later.
However, when the log dir movement completed, the future log is renamed, the async log deletion will fail with no file existed error.
Signed-off-by: PoAn Yang <payang@apache.org>
Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>, SoontaekLim <soontaek.lim@neya.kr>, Johnny Hsu <johnnyhsu@fb.com>
The system test was failing because the VerifiableConsumer failed with a NullPointerException during startup. The reason for the NPE was an attempt to put a null as the value of --group-remote-assignor in the Consumer's configuration.
Reviewers: Lianet Magrans <lianetmr@gmail.com>, Lucas Brutschy <lbrutschy@confluent.io>
The generic error handler of QuorumController didn't log the exception message for non-fault errors, which includes very useful debugging info.
Reviewers: Igor Soarez <soarez@apple.com>
Implementation of KIP-773 deprecated iotime-total and io-waittime-total metrics. It wasn't expected to mark io-ratio and io-wait-ratio deprecated. However, now they have *Deprecated* in their description. Here is the reason:
register io-ratio (desc: *Deprecated* The fraction of time ...) -> registered
register iotime-total (desc: *Deprecated* The total time ...) -> registered
register io-ratio (desc: The fraction of time ...) -> skipped, the same name already exists in registry
register io-time-ns-total (desc: The total time ...) -> registered
As a result, io-ratio has incorrect description. The same for io-wait-ratio. This PR fixes these descriptions..
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
The javadoc for KafkaConsumer.commitSync says:
Note that asynchronous offset commits sent previously with the {@link #commitAsync(OffsetCommitCallback)}
(or similar) are guaranteed to have their callbacks invoked prior to completion of this method.
This is not always true in the async consumer, where there is no code at all to make sure that the callback is executed before commitSync returns.
Similarly, the async consumer is also missing logic to await callback execution in close. While the javadoc doesn't explicitly promise callback execution, it promises "completing commits", which one would reasonably expect to include callback execution. Also, the legacy consumer contains some code to execute callbacks before closing.
This change proposed a number of fixes to clean up the callback execution guarantees in the async consumer:
We keep track of the incomplete async commit
futures and wait for them to complete before returning from
commitSync or close (if there is time).
Since we need to block to make sure that our previous commits are
completed, we allow the consumer to wake up.
Some similar gaps are addressed in the legacy consumer, see #15693
Testing
Two new integration tests and a couple of unit tests.
Reviewers: Bruno Cadonna <cadonna@apache.org>, Kirk True <ktrue@confluent.io>, Lianet Magrans <lianetmr@gmail.com>
Due to difference in packages present when jsa files were generated and when docker image is generated, there is a log on starting docker image.
[0.001s][warning][cds] The shared archive file has a bad magic number: 0
There is no functionality impact, only startup time is higher because of this issue.
This PR fixes the warning and improves startup performance of the docker image
Reviewers: Manikumar Reddy <anikumar.reddy@gmail.co
It is observed that for scenario (3), i.e. a broker crashes while it
waits for the future replica to catch up for the second time and the
`dir1` is unavailable when the broker is restarted, the
broker tries to create the partition in `dir2` according to the metadata
in the controller. However, ReplicaManager also tries to resume the
stale future replica which was abandoned when the broker crashed. This
results in the renaming of the future replica to fail eventually because
the directory for the topic partition already exists in `dir2` and the
broker then marks `dir2` as offline.
This PR attempts to fix this behaviour by ignoring any future replicas
which are in the same directory as where the log exists. It further
marks the stale future replica for deletion.
Reviewers: Omnia Ibrahim <o.g.h.ibrahim@gmail.com>, Igor Soarez <soarez@apple.com>, Proven Provenzano <pprovenzano@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
The PR leverages the changes defined in KIP-1019. Does the cleanup for accessing KafkaMetric field by reflection and uses method exposed by KIP-1019 for metric measurability.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Summary of the changes:
Parameterizes the tests to use new coordinator and pass in consumer group protocol. This would be applicable to sink connectors only.
Enhances the sink connector creation code in system tests to accept a new optional parameter for consumer group protocol to be used.
Sets the consumer group protocol via consumer.override. override config when the new group coordinator is enabled.
Note about testing: There are 288 tests that need to be run and running on my local takes a lot of time. I will try to post the test results once I have a full run.
Reviewers: Kirk True <ktrue@confluent.io>, Lucas Brutschy <lbrutschy@confluent.io>, Philip Nee <pnee@confluent.io>
Consumer Rolling Upgrade is meant to test the protocol upgrade for the old protocol. Therefore, I am removing old changes.
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
Updating consumer system test that was failing with the new protocol, related to static membership behaviour. The behaviour regarding static consumers that join with conflicting group instance id is slightly different between the classic and new consumer protocol, so the expectations in the tests needed to be updated.
If static members join with same instance id:
Classic protocol: all members join the group with the same group instance id, and then the first one will eventually fail (receives a HB error with FencedInstanceIdException)
Consumer protocol: new member with an instance Id already in use is not able to join, and first member remains active (new member with same instance Id receives an UnreleasedInstanceIdException in the response to the HB to join the group)
This PR is keeping the single parametrized test that existed before, given that what's being tested and part of the test itself apply to all protocols. This is just updating the expectations that are different, based on the protocol parameter.
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>, Kirk True <ktrue@confluent.io>
RemoteLogMetadataSerde references RemoteLogMetadataTransform in a Raw
form. Given that the class is parametrized we should make use of it.
Signed-off-by: Josep Prat <josep.prat@aiven.io>
Reviewers: Matthew de Detrich <matthew.dedetrich@aiven.io>, Mickael Maison <mickael.maison@gmail.com>
Kafka Streams DSL operators use internal wall-clock based throttling
parameters for performance reasons. These configs make the usage of TTD
difficult: users need to advance the mocked wall-clock time in
their test code, or set these internal configs to zero.
To simplify testing, TDD should disable both configs automatically.
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
Implements KIP-1019, which exposes method to check if metric is of type Measurable.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Add the support for DescribeTopicPartitions API to AdminClient. For this initial implementation, we are simply loading all of the results into memory on the client side.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Artem Livshits <alivshits@confluent.io>, David Arthur <mumrah@gmail.com>
This patch introduces the conversion from a classic group to a consumer group when a member joins with the new consumer group protocol (epoch is 0) but only if the conversion is enabled.
Reviewers: David Jacot <djacot@confluent.io>
Arnout Engelen
Kirk True
Chia-Ping Tsai
TingIāu "Ting" Kì
Justine Olshan
PoAn Yang
vamossagar12
Kamal Chandraprakash
Omnia Ibrahim
Cheng-Kai, Zhang
Gyeongwon, Do
ilyazr
Eugene Mitskevich
Ken Huang
charliecheng630
Yu-Chen Lai
Andras Katona
Lucas Brutschy
Vedarth Sharma
Kuan-Po (Cooper) Tseng
hudeqi
Gaurav Narula
Apoorv Mittal
Philip Nee
Lianet Magrans
Josep Prat
Matthias J. Sax
Calvin Liu
Mickael Maison
Ayoub Omari
Dongnuo Lyu
Nikolay
Michael Westerby
Johnny Hsu