Introduce ShareMemberDescription and ShareMemberAssignment as distinct classes for share groups. Although the correspondence with consumer groups is fairly close, the concepts are likely to diverge over time and separating these concepts now makes sense.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
Convert testUncleanLeaderElectionEnable and testMetricsReporterUpdate to KRaft.
Remove testAdvertisedListenerUpdate, testAddRemoveSslListener, testAddRemoveSaslListeners since we no longer support dynamically adding or removing network listeners when in KRaft mode.
Make TestUtils.ensureConsistentKRaftMetadata robust against brokers that don't have sharedServer.loader initialized yet (or have shut down).
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
This PR is part of the implementation for KIP-1112 (KAFKA-18026). In order to have DSL operators be properly wrapped by the interface suggestion in 1112, we need to make sure they all use the ConnectedStoreProvider#stores method to connect stores instead of manually calling addStateStore.
This is a refactor only, there is no new behaviors.
Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
The KStreamRepartitionIntegrationTest.shouldThrowAnExceptionWhenNumberOfPartitionsOfRepartitionOperationDoNotMatchSourceTopicWhenJoining test was taking two minutes due not reaching an expected condition. By updating to have the StreamsUncaughtExceptionHandler return a response of SHUTDOWN_CLIENT the expected ERROR state is now reached. The root cause was using the Thread.UncaughtExceptionHandler to handle the exception.
Without this fix, the test takes 2 minutes to run, and now it's 1 second.
Reviewers: Matthias Sax <mjsax@apache.org>
https://github.com/apache/kafka/pull/15881 changed our tests to utilize `using` blocks. But these blocks don't throw any errors, so if there is a failed assertion within the block, the test will still pass.
We should either check the failure using a corresponding `match` block with Success(_) and Failure(e), use `using.resource`, or use try/finally blocks to clean up resources.
See https://www.scala-lang.org/api/3.0.2/scala/util/Using$.html
Co-authored-by: frankvicky <kitingiao@gmail.com>
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
This patch introduces the asynchronous resolution of regular expressions. Let me unpack a few details about the implementations:
1) I have decided to finally update all the regular expressions within a consumer group together. My assumption is that the number of regular expressions in a group will be generally small but the number of topics in a cluster is large. Hence grouping has two benefits. Firstly, it allows to go through the list of topics once for all the regular expressions. Secondly, it reduces the number of potential rebalances because all the regular expressions are updated at the same time.
2) An update is triggered when the group is subscribed to at least one regular expressions.
3) An update is triggered when there is no ongoing update.
4) An update is triggered only of the previous one is older than 10s.
5) An update is triggered when the group has unresolved regular expressions.
6) An update is triggered when the metadata image has new topics.
Reviewers: Jeff Kim <jeff.kim@confluent.io>
There was a bug in handling the ShareAcknowledgeResponse for commitAsync(). Currently after we receive a response, we send out a background event to the application thread to update the acknowledgement commit callbacks for EVERY TopicIdPartition.
The map that was sent was not cleared after sending the event. This meant we ended up sending responses for partitions that were already sent in the previous event. So there will be duplicate calls to the callback.
The PR fixes the bug and adds a unit test for the same.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>
ShareConsumerTest.testShareAutoOffsetResetDefaultValue has been tightened up by making sure that records produced have been flushed before starting consumption. A possible but unlikely race condition seems the source of the flakiness and this should now be eliminated in the previous PR to this test case.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
When a KafkaShareConsumer is constructed in AK 4.0, a log message is written warning about the early access nature of the feature.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
This patch adds some clarity about the environment variables usage and limitations.
It also removes the invalid example and adds few style fixes.
Signed-off-by: Federico Valeri <fedevaleri@gmail.com>
Reviewers: Vedarth Sharma <vesharma@confluent.io>, Luke Chen <showuon@gmail.com>
With KAFKA-14562, we implemented epoch bump on both the client and the server. Mentioned below are the different epoch bump scenarios we have on hand after enabled tv2
Non-Transactional Producers
• Epoch bumping is always allowed.
• Different code paths are used to handle epoch bumping.
Transactional Producers
No Epoch Bump Allowed
• coordinatorSupportsBumpingEpoch = false when initPIDVersion < 3 or initPIDVersion = null.
Client-Triggered Epoch Bump Allowed
• coordinatorSupportsBumpingEpoch = true when initPIDVersion >= 3.
• TransactionVersion2Enabled = false when endTxnVersion < 5.
Only Server-Triggered Epoch Bump Allowed
• TransactionVersion2Enabled = true and endTxnVersion >= 5.
We want to refine the code and make it more structured to correctly handle epoch bumping in the above mentioned cases.
The changes made in this patch are:
Rename epochBumpRequired to epochBumpTriggerRequired to symbolize a manual epoch bump request from the client
Modify canEpochBump method according to the above mentioned scenarios
Reviewers: Artem Livshits <alivshits@confluent.io>, Calvin Liu <caliu@confluent.io>, Justine Olshan <jolshan@confluent.io>
When disconnecting channels before rebootstrapping due to the rebootstrap conditions introduced in KIP-1102, we should ensure that inflight requests are aborted similar to other disconnections like request timeout in clients. With the earlier rebootstrapping from KIP-899, we only rebootstrapped when there were no connections, so no disconnections are required.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
Noticed that RangeQueryIntegrationTest is taking ~approx 20 - 30min to run
Upon deep dive in logs, noticed that there were error for consumer rebalancing and test was stuck in loop
Seems like due to same application.id across tests, Kafka Streams application is failing to track its state correctly across rebalances.
Reviewers: Bill Bejeck <bbejeck@apache.org>
KIP-1043 introduced Admin.listGroups as the way to list all types of groups. As a result, Admin.listShareGroups has been removed. This PR is the final step of the removal.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
- Deprecates OffsetResetStrategy enum
- Adds new internal class AutoOffsetResetStrategy
- Replaces all OffsetResetStrategy enum usages with AutoOffsetResetStrategy
- Deprecate old/Add new constructors to MockConsumer
Reviewers: Andrew Schofield <aschofield@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Andrew Schofield
Colin Patrick McCabe
HYUNSANG HAN (한현상, Travis)
PoAn Yang
TaiJuWu
Almog Gavra
TengYao Chi
Peter Lee
Bill Bejeck
Chia-Chuan Yu
Lianet Magrans
David Jacot
Ken Huang
Yung
Joao Pedro Fonseca Dantas
Calvin Liu
Matthias J. Sax
ShivsundarR
Kuan-Po Tseng
Federico Valeri
Ritika Reddy
Mickael Maison
Rajini Sivaram
Kaushik Raina
ClarkChen
Manikumar Reddy
Bruno Cadonna