The PR makes the persister write RPC async. Also handles the errors from persister as per the review comment here:
Addressing review comment for PR: #16397 (comment)
Reviewers: Andrew Schofield <aschofield@confluent.io>, Abhinav Dixit <adixit@confluent.io>, Jun Rao <junrao@gmail.com>
According to GitHub rule:
If a repository contains more than one README file, then the file shown is chosen from locations in the following order: the .github directory, then the repository's root directory, and finally the docs directory.
the file .github/readme will override root/readme. Hence, we should move .github/readme to ./github/workflows/readme
Reviewers: David Arthur <mumrah@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Add the version check to server side for the specific timestamp:
- the version must be >=8 if timestamp=-4L
- the version must be >=9 if timestamp=-5L
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
In AsyncKafkaConsumer, FindCoordinatorRequest is sent by background thread. In MockClient#prepareResponseFrom, it only matches the response to a future request. If there is some race condition, FindCoordinatorResponse may not match to a FindCoordinatorRequest. It's better to put MockClient#prepareResponseFrom before the request to avoid flaky test.
Reviewers: TaiJuWu <tjwu1217@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
This patch adds a "deflake" github action which can be used to run a single JUnit test or suites. It works by parameterizing the --tests Gradle option. If the test extends ClusterTest, the "deflake" workflow can repeat number of times by setting the kafka.cluster.test.repeat system property.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Per KIP-289, the use of an empty value for group.id configuration was deprecated back in 2.2.0.
In 3.7, the AsyncKafkaConsumer implementation will throw an error (see KAFKA-14438).
This task is to update the LegacyKafkaConsumer implementation to throw an error in 4.0.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Because of KIP-902 (Upgrade Zookeeper version to 3.8.2), it is not possible to upgrade from a Kafka version
earlier than 2.4 to a version later than 2.4. Therefore, we should not test these upgrade scenarios
in upgrade_test.py. They do happen to work sometimes, but only in the trivial case where we don't
create topics or make changes during the upgrade (which would reveal the ZK incompatibility).
Instead, we should test only supported scenarios.
Reviewers: Reviewers: José Armando García Sancio <jsancio@gmail.com>
The docker image for Native Apache Kafka was introduced with KIP-974 and was first release with 3.8 AK release.
The docker image for Native Apache Kafka is currently intended for local development and testing purposes.
This PR intends to add a logline indicating the same during docker image startup.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
This patch re-introduces the `group.version` feature flag and gates the new consumer rebalance protocol with it. The `group.version` feature flag is attached to the metadata version `4.0-IV0` and it is marked as production ready. This allows system tests to pick it up directly by default without requiring to set `unstable.feature.versions.enable` in all of them. This is fine because we don't plan to do any incompatible changes before 4.0.
Reviewers: Justine Olshan <jolshan@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
The COPY_SEGMENT_STARTED state segments are counted when calculating remote retention size. This causes unexpected retention size breached segment deletion. This PR fixes it by
1. only counting COPY_SEGMENT_FINISHED and DELETE_SEGMENT_STARTED state segments when calculating remote log size.
2. During copy Segment, if we encounter errors, we will delete the segment immediately.
3. Tests added.
Co-authored-by: Guillaume Mallet <>
Reviewers: Kamal Chandraprakash<kamal.chandraprakash@gmail.com>, Satish Duggana <satishd@apache.org>, Guillaume Mallet <>
Previously in KRaft mode, we could request an unclean leader election for a specific topic using
the electLeaders API. This PR adds an additional way to trigger unclean leader election when in
KRaft mode via the static controller configuration and various dynamic configurations.
In order to support all possible configuration methods, we have to do a multi-step configuration
lookup process:
1. check the dynamic topic configuration for the topic.
2. check the dynamic node configuration.
3. check the dynamic cluster configuration.
4. check the controller's static configuration.
Fortunately, we already have the logic to do this multi-step lookup in KafkaConfigSchema.java.
This PR reuses that logic. It also makes setting a configuration schema in
ConfigurationControlManager mandatory. Previously, it was optional for unit tests.
Of course, the dynamic configuration can change over time, or the active controller can change
to a different one with a different configuration. These changes can make unclean leader
elections possible for partitions that they were not previously possible for. In order to address
this, I added a periodic background task which scans leaderless partitions to check if they are
eligible for an unclean leader election.
Finally, this PR adds the UncleanLeaderElectionsPerSec metric.
Co-authored-by: Luke Chen showuon@gmail.com
Reviewers: Igor Soarez <soarez@apple.com>, Luke Chen <showuon@gmail.com>
AccessTokenRetrieverFactory uses the value of sasl.oauthbearer.header.urlencode provided by the user, or null if no value was provided for that configuration. When the HttpAccessTokenRetriever is created the JVM attempts to unbox the value into a boolean, a NullPointerException is thrown.
The fix is to explicitly check the Boolean, and if it's null, use Boolean.FALSE.
Reviewers: bachmanity1 <81428651+bachmanity1@users.noreply.github.com>, Chia-Ping Tsai <chia7712@gmail.com>
We currently use a `CountDownLatch` to signal when a thread has
completed shutdown to the blocking `shutdown` method. However, this
latch triggers _before_ the thread has fully exited.
Dependent on the OS thread scheduling, it's possible that this thread
will still be "alive" after the latch has unblocked the `shutdown`
method.
In practice, this is mostly a problem for `StreamThreadTest`, which now
checks that there are no `TaskExecutor` or `StateUpdater` threads
immediately after shutting them down.
Sometimes, after shutdown returns, we find that these threads are still
"alive", usually completing execution of the "thread shutdown" log
message, or even the `Thread#exit` JVM method that's invoked to clean up
threads just before they exit. This causes sporadic test failures, even
though these threads did indeed shutdown correctly.
Instead of using a `CountDownLatch`, let's just await the thread to exit
directly, using `Thread#join`. Just as before, we set a timeout, and if
the Thread is still alive after the timeout, we throw a
`StreamsException`, maintaining the contract of the `shutdown` method.
There should be no measurable impact on production code here. This will
mostly just improve the reliability of tests that require these threads
have fully exited after calling `shutdown`.
Reviewers: Matthias J. Sax <matthias@confluent.io>, Lucas Brutschy <lbrutschy@confluent.io>, Bruno Cadonna <cadonna@apache.org>
The given test took 5 seconds as the logic was waiting completely for 5 seconds for the expiration task to be completed. Adding copySegmentLatch countdown after expiration task is over
Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Fix to ensure that the HB request to leave the group is generated when closing the HBRequestManager if the state is LEAVING. This is needed because we could end up closing the network thread without giving a chance to the HBManager to generate the request. This flow on consumer.close with short timeout:
1. app thread triggers releaseAssignmentAndLeaveGroup
2. background thread transitions to LEAVING
2.1 the next run of the background thread should poll the HB manager and generate a request
3. app thread releaseAssignmentAndLeaveGroup times out and moves on to close the network thread (stops polling managers. Calls pollOnClose to gather the final requests and send them along with the unsent)
If 3 happens in the app thread before 2.1 happens in the background, the HB manager won't have a chance to generate the request to leave. This PR implements the pollOnClose to generate the final request if needed.
Reviewers: Kirk True <kirk@kirktrue.pro>, TaiJuWu <tjwu1217@gmail.com>, TengYao Chi <kitingiao@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Implement the remaining ForwardingManager metrics from KIP-938: Add more metrics for measuring KRaft performance:
kafka.server:type=ForwardingManager,name=QueueTimeMs.p99
kafka.server:type=ForwardingManager,name=QueueTimeMs.p999
kafka.server:type=ForwardingManager,name=QueueLength
kafka.server:type=ForwardingManager,name=RemoteTimeMs.p99
kafka.server:type=ForwardingManager,name=RemoteTimeMs.p999
Reviewers: Colin P. McCabe <cmccabe@apache.org>
The topic partitions were not being released as the final epoch removes the share session for the client at the start.
The PR streamlines the release process by invoking an explicit releaseSession API which removes the session and releases acquired records. Also newContext and acknowledgeSession method previously used to evict the share session but streamlined methods to just do the needful without altering the share session eviction.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Chirag Wadhwa <122860692+chirag-wadhwa5@users.noreply.github.com>
The patch adds support of alter/describe configs for group in kafka-configs.sh.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>, David Jacot <djacot@confluent.io>
This is a follow-up after KIP-919, extending support for BOOTSTRAP_CONTROLLERS_CONFIG to both Admin#alterPartitionReassignments and Admin#listPartitionReassignments.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
TaiJuWu
Apoorv Mittal
Matthias J. Sax
Dmitry Werner
Kuan-Po Tseng
TengYao Chi
PoAn Yang
Viktor Somogyi-Vass
Arnav Dadarya
Chia Chuan Yu
David Arthur
Mickael Maison
Eric Chang
abhi-ksolves
JohnHuang
Nancy
Colin Patrick McCabe
Logan Zhu
Ken Huang
Krishna Agarwal
David Jacot
Luke Chen
xijiu
José Armando García Sancio
Kirk True
Nick Telford
Arpit Goyal
Lianet Magrans
kevin-wu24
DL1231