- Fixed the RemoteIndexCacheTest that fails with caffeine > 3.1.1
Reviewers: Luke Chen <showuon@gmail.com>, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>
This commit adds the Streams group heartbeat request manager
to the async consumer. The Streams group heartbeat request
manager is responsible to send heartbeat requests and to
process their responses.
This commit implements:
- sending of full heartbeat request (independent of any state)
- processing successful response
Reviewers: Bill Bejeck <bill@confluent.io>, Lucas Brutschy <lbrutschy@confluent.io>
Before the patch:
```
% python3 ./committer-tools/verify_license.py
...
All libs from ./libs are present in the LICENSE file.
The following entries are in the LICENSE file but not present in ./libs. These should be removed from the LICENSE-binary file:
- audience-annotations-0.12.0
- jackson-jaxrs-base-2.16.2
- jackson-jaxrs-json-provider-2.16.2
- jackson-module-jaxb-annotations-2.16.2
- jakarta.inject-2.6.1
- javax.servlet-api-3.1.0
- jetty-continuation-9.4.56.v20240826
- jetty-servlet-9.4.56.v20240826
- jetty-servlets-9.4.56.v20240826
- jetty-util-ajax-9.4.56.v20240826
- jsr305-3.0.2
- log4j-core-test-2.24.1
```
After the patch:
```
% python3 ./committer-tools/verify_license.py
...
All libs from ./libs are present in the LICENSE file.
No extra dependencies in the LICENSE file.
```
Reviewers: Mickael Maison <mickael.maison@gmail.com>
This patch adds the verify_license.py tool. It compares the libraries shipped within the tarball to the LICENSE file, and vice versa, to ensure that they are aligned. It also slightly update the format of the LICENSE file to make it easier to parse it.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Mickael Maison <mickael.maison@gmail.com>
The stale/invalid files that ends-with ".deleted" and ".tmp" should be cleaned when the broker gets restarted.
- fix the remote-index-cache test to use the logDir instead of topicDir
- fix the flaky test
Reviewers: Luke Chen <showuon@gmail.com>
When group.coordinator.threads is greater than 1, we lose track of thread idle time because of integer arithmetic. Use doubles instead.
Reviewers: David Jacot <djacot@confluent.io>
This patch removes dropwizard metrics in the dependency list as it is not used any more. It was introduced in 4f5b4c868e because it was required by Zookeeper. Zookeeper is no longer there so we can remove it too.
Reviewers: Ismael Juma <ismael@juma.me.uk>
Introduced two new exception classes to the Kafka error handling framework:
ApplicationRecoverableException: This exception signals that the error is recoverable, but the producer needs to be restarted. It helps in scenarios where recovery actions (like re-balancing or restoring from checkpoints) are needed.
RefreshRetriableException: This exception occurs when metadata is outdated or invalid and needs to be refreshed before retrying the request. It helps handle retries that depend on updated metadata.
Both classes are abstract and in upcoming PRs they will be extended by relevant classes as mentioned in KIP-1050:Exception Table.
Reviewers: Justine Olshan <jolshan@confluent.io>, Sanskar Jhajharia <jhajharia.sanskar@gmail.com>
Adds streams group to the GroupMetadataManager, and implements loading
the records from the offset topic into state. The state also contains
two timers (rebalance timeout and session timeout) that are started
after the group coordinator has been loaded.
Reviewers: Bruno Cadonna <bruno@confluent.io>, Bill Bejeck <bill@confluent.io>
In PR #18267, we removed old message format for cases in ConsumerWithLegacyMessageFormatIntegrationTest. Although test cases can pass, they don't fulfill original purpose. We can't send old message format since 4.0, so I change cases to append old records by ReplicaManager directly.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
- The share.coordinator.state.topic.min.isr config defined in ShareCoordinatorConfig was not being used in the AutoTopicCreationManager.
- The AutoTopicCreationManager calls the ShareCoordinatorService.shareGroupStateTopicConfigs to configs for the topic to create.
- The method ShareCoordinatorService.shareGroupStateTopicConfigs was not setting the supplied config value for share.coordinator.state.topic.min.isr to min.insync.replicas.
- In this PR, we remedy the situation by setting the value
- A test has been added to ShareCoordinatorServiceTest so that this is not repeated for any configs.
Reviewers: poorv Mittal <apoorvmittal10@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Frequently updating the trust store can cause unexpected termination of the AsyncConsumer background thread.
1. To resolve this issue, reuse the same AdminClient instead of recreating it.
2. Add error logging when fail to initialize resources for the consumer network thread.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
KAFKA-16720 aims to add the support for the AlterShareGroupOffsets AdminClient. Key Changes in the PR:
1. Added handing of alterShareGroupOffsets() in KafkaAdminClient and introduce AlterShareGroupOffsetRequest/AlterShareGroupOffsetResponse/AlterShareGroupOffsetsOptions classes.
2. Corresponding test in KafkaAdminClientTest.
3. Added ALTER_SHARE_GROUP_OFFSETS API (will finish it in next PR and the share coordinator pieces)
Reviewers: poorv Mittal <apoorvmittal10@gmail.com>, Andrew Schofield <aschofield@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
This patch marks IBP_4_0_IV3 as production ready for the Apache Kafka 4.0 release. It also introduced IBP_4_1_IV0 as the next development version.
Reviewers: Justine Olshan <jolshan@confluent.io>
This patch cleans up the places that should not use MV to determine ELR is enabled marks 4.0IV1 stable.
Reviewers: Alyssa Huang <ahuang@confluent.io>, Colin P. McCabe <cmccabe@apache.org>
KIP-744 introduced the StreamsMetadata class as part of the implementation. In the KIP, the javadoc for the standbyTopicPartitions states that the method returns the set of source TopicPartition that it represents as a standby. The current javadoc states that it represents the changelog TopicPartition(s). While the partitions of the source and changelog topics will match, the javadoc needs to be updated to reflect the correct behavior.
Note that the deprecated o.a.k.streams.state.StreamsMetadata#standbyTopicPartitions method also describes the set of TopicPartition being source TopicPartition.
Reviewers: Matthias Sax<mjsax@apache.org>
During the transaction commit phase, it is normal to hit CONCURRENT_TRANSACTION error before the transaction markers are fully propagated. Instead of letting the client to retry the produce request, it is better to retry on the server side.
Reviewers: Artem Livshits <alivshits@confluent.io>, Justine Olshan <jolshan@confluent.io>
TaiJuWu
Parker Chang
Mickael Maison
Andrew Schofield
Chirag Wadhwa
Bruno Cadonna
David Jacot
Luke Chen
xijiu
Kamal Chandraprakash
Sean Quah
Matthias J. Sax
Kaushik Raina
Apoorv Mittal
Jimmy Wang
Lucas Brutschy
PoAn Yang
Ken Huang
TengYao Chi
Sushant Mahajan
Jhen-Yung Hsu
David Arthur
Ming-Yen Chung
Justine Olshan
Calvin Liu
陳昱霖(Yu-Lin Chen)
Bill Bejeck
Kirk True
Lianet Magrans