Declare the delayed share fetch purgatory inside ReplicaManager along with the existing purgatories.
Check the share fetch purgatory when a replica becomes the follower or a replica is deleted from a broker through ReplicaManager.
Perform a checkAndComplete for share fetch when HWM is updated.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Apoorv Mittal <apoorvmittal10@gmail.com>, Jun Rao <junrao@gmail.com>
Integration tests should run with either at-least-once or exactly-once.
There is no need to run them twice.
This PR removes the corresponding test parameters and picks either one of both.
Reviewers: Matthias J. Sax <matthias@confluent.io>
The ClientsMetricManager keeps the client instance cache at a limit of 16384. The active connections can be low, but connections can be created and destroyed in short span. Which hits the cache limit and results in cache miss for new connections.
The client instance cache keeps the instances for 3 * push interval ms. Hence when the cache gets full it creates new instances for every request while evicting others from cache. This gives some bumps to the GC for the broker and eventually CPU. Though with evicting early it will still be garbage collected but the long running active connections will not be removed from the cache.
This PR adds capability to evict client instances from cache when the connection from client is dropped.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Andrew Schofield <aschofield@confluent.io>, Jun Rao <junrao@gmail.com>
Rework the uniform heterogeneous assignor to improve performance, while
preserving the high level ideas and structure from the existing
implementation:
* The assignor works in 3 stages: importing the previous assignment for
stickiness, assigning unassigned partitions and iteratively
reassigning partitions to improve balance.
* Unassigned partitions are assigned to the subscribers with the least
number of partitions. This maximizes balance within a single topic.
* During the iterative rebalancing phase, partitions are reassigned to
their previous owner if it improves balance (stickiness restoration).
* During the iterative rebalancing phase, partitions are reassigned to
the subscriber with the least number of partitions to improve
balance.
A non-exhaustive list of changes is:
* The assignment of unassigned partitions and iterative reassignment
stages now works through partitions topic by topic. Previously
partitions from topics with the same number of partitions per
subscriber would be interleaved. Since we iterate topic by topic, we
can reuse data about topic subscribers.
* Instead of maintaining TreeSets to find the least loaded subscribers,
we sort an ArrayList of subscribers once per topic and start filling
up subscribers, least loaded first. In testing, this approach was
found to be faster than maintaining PriorityQueues.
* Implement stickiness restoration by creating a mapping of partitions
to previous owner and checking against that mapping, instead of
tracking partition movements during iterative reassignment.
* Track member partition counts using a plain int array, to avoid
overhead from boxing and HashMap lookups. Member partition counts are
accessed very frequently and this needs to be fast. As a consequence,
we have to number members 0 to M - 1.
* Bound the iterative reassignment stage to a fixed number of
iterations. Under some uncommon subscription patterns, the iterative
reassignment stage converges slowly. In these cases, the iterative
reassignment stage terminates without producing an optimally balanced
assignment anyway (see javadoc for balanceTopics).
* Re-use Maps from the previous assignment where possible,
ie. introduce a copy-on-write mechanism while computing the new
assignment.
Reviewers: David Jacot <djacot@confluent.io>
Since we are removing support for Scala 2.12, CoreUtils#atomicGetOrUpdate should also be removed, as it is a workaround for the different behavior between ConcurrentMap.getOrElse and JConcurrentMapWrapper.
JMH results show no significant performance regression when using `getOrElseUpdate`:
Benchmark (mapSize) (writePercentage) Mode Cnt Score Error Units
ConcurrentMapBenchmark.testConcurrentHashMapComputeIfAbsentReadOnly 100 0.1 avgt 5 4.182 ± 0.006 ns/op
ConcurrentMapBenchmark.testConcurrentHashMapGetReadOnly 100 0.1 avgt 5 3.372 ± 0.024 ns/op
JMH benchmarks done
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Some tests didn't catch output messages correctly, for example, the output of a test should be "Completed updating config for user admin." but we didn't got it. Copied a method from `ToolsTestUtils.java` which can be removed after we move ConfigCommand to tools module.
Change the output stream from `scala.Console.println` to `System.out.println` to make test easier.
As of now, there isn't any way that resources can be cleaned in Kafka when client disconnects the connections. Generally time bound clean up happens which keep the resources occupied.
This PR provides capability to attach listeners in Socket Server which can react to client connection disconnections.
Reviewers: Jun Rao <junrao@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Deprecate for 4.0. This feature will be phased out, with new functionality generally not supporting named topologies, so users are encouraged to migrate off of named topologies as soon as possible.
Though who are using this experimental feature are encouraged to reach out by filing a jira ticket so we can better understand your use case and how to support it going forward.
Reviewers: Matthias Sax <mjsax@apache.org>
This PR disables metrics push for the AdminClient as the default. KafkaStreams enables metrics push for its internal AdminClient.
Tests are included that assert an error if a user disables either the main consumer or admin client metrics push but Kafka Streams metrics push config is enabled.
Reviewers: Matthias Sax <mjsax@apache.org>
The tests fails sometime while reading for timed out records. Moved the timeout from Thread.sleep to waituntilcondition and increased it so records can be read in cases where it took more than expected.
Reviewers: ShivsundarR <rshiva1405@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
Unit test shouldRestoreSingleActiveStatefulTask() in DefaultStateUpdaterTest is flaky.
The flakiness comes from the fact that the state updater thread could call the first time changelogReader.allChangelogsCompleted() before it calls the first time changelogReader.completedChangelogs(). That happens, if runOnce() is run before the state updater thread reads a task from the input queue.
This commit fixes the flakiness, by making stubbing changelogReader.allChangelogsCompleted() depend on stubbing changelogReader.completedChangelogs().
Reviewers: Lucas Brutschy <lbrutschy@confluent.io>, Matthias J. Sax <matthias@confluent.io>
In Kafka 4.0, by default the consumer uses the updated consumer group protocol defined in KIP-848. When the consumer is used with a cluster that does not support (or is not configured to use) the new protocol, the consumer will get an unfriendly error about unavailable APIs. Since this error could be the user's first impression when attempting to upgrade to 4.0, we need to make sure that the error is very clear about the remediation steps (set the group.protocol to CLASSIC on the client or upgrade and enable the new protocol on the cluster).
Reviewers: Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>
This PR adds the 2 configs share.heartbeat.interval.ms and share.session.timeout.ms in GroupConfig. These can be dynamically set for a share group without restarting the server
Reviewers: Andrew Schofield <aschofield@confluent.io>, Apoorv Mittal <apoorvmittal10@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
As part of KIP-932, a new internal topic __share_group_state was introduced. There are 2 types of records which are currently being added in this topic - ShareSnapshotKey/Value and ShareUpdateKey/Value
In light of this, we must make the existing tooling like kafka-console-consumer and kafka-dump-log aware of these records for debugging and introspection purposes.
This PR introduces ShareGroupStateMessageFormatter to be used used with kafka-console-consumer and adds an internal class ShareGroupStateMessageParser in DumpLogSegments.scala.
Unit tests have been added to DumpLogSegmentsTest.scala
Reviewers: Andrew Schofield <aschofield@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>
KIP-932 used ignorable annotations in some RPC schemas when they are not applicable in situations where there is only a single version of the RPC in existence. This PR removes the unnecessary annotations. The KIP has been updated accordingly.
Reviewers: Jun Rao <junrao@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Abhinav Dixit
Zhengke Zhou
Apoorv Mittal
Kuan-Po Tseng
kartik-3513
Ken Huang
Andrew Schofield
Sean Quah
Josep Prat
TengYao Chi
David Arthur
dengziming
Chung, Ming-Yen
A. Sophie Blee-Goldman
Bill Bejeck
Bruno Cadonna
Yung
Chia-Chuan Yu
PoAn Yang
Kamal Chandraprakash
kwonyonghyun
Chirag Wadhwa
陳昱霖(Yu-Lin Chen)
Sushant Mahajan
Linsiyuan9
Mickael Maison
Logan Zhu
ClarkChen
Chengyan
mingdaoy
Anshul Goyal
Gaurav Narula