root
/
kafka
mirror of https://github.com/apache/kafka.git
1
0
Fork 0
Code Issues Actions 2 Packages Projects Releases Wiki Activity
kafka/checkstyle/import-control-metadata.xml

200 lines
8.9 KiB
XML
Raw Normal View History

KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<!DOCTYPE import-control PUBLIC
"-//Puppy Crawl//DTD Import Control 1.1//EN"
"http://www.puppycrawl.com/dtds/import_control_1_1.dtd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<import-control pkg="org.apache.kafka">
<!-- THINK HARD ABOUT THE LAYERING OF THE PROJECT BEFORE CHANGING THIS FILE -->
<!-- common library dependencies -->
<allow pkg="java" />
<allow pkg="javax.management" />
<allow pkg="org.slf4j" />
<allow pkg="org.junit" />
<allow pkg="org.mockito" />
<allow pkg="java.security" />
<allow pkg="javax.net.ssl" />
<allow pkg="javax.security" />
KAFKA-15219: KRaft support for DelegationTokens (#14083) Reviewers: David Arthur <mumrah@gmail.com>, Ron Dagostino <rndgstn@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Viktor Somogyi <viktor.somogyi@cloudera.com>
2023-08-20 02:01:08 +08:00
<allow pkg="javax.crypto" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<allow pkg="net.jqwik.api" />
<!-- no one depends on the server -->
<disallow pkg="kafka" />
<!-- anyone can use public classes -->
<allow pkg="org.apache.kafka.common" exact-match="true" />
<allow pkg="org.apache.kafka.common.security" />
<allow pkg="org.apache.kafka.common.serialization" />
<allow pkg="org.apache.kafka.common.utils" />
<allow pkg="org.apache.kafka.common.errors" exact-match="true" />
<allow pkg="org.apache.kafka.common.memory" />
KAFKA-17587 Refactor test infrastructure (#18602) This patch reorganizes our test infrastructure into three Gradle modules: ":test-common:test-common-internal-api" is now a minimal dependency which exposes interfaces and annotations only. It has one project dependency on server-common to expose commonly used data classes (MetadataVersion, Feature, etc). Since this pulls in server-common, this module is Java 17+. It cannot be used by ":clients" or other Java 11 modules. ":test-common:test-common-util" includes the auto-quarantined JUnit extension. The @Flaky annotation has been moved here. Since this module has no project dependencies, we can add it to the Java 11 list so that ":clients" and others can utilize the @Flaky annotation ":test-common:test-common-runtime" now includes all of the test infrastructure code (TestKitNodes, etc). This module carries heavy dependencies (core, etc) and so it should not normally be included as a compile-time dependency. In addition to this reorganization, this patch leverages JUnit SPI service discovery so that modules can utilize the integration test framework without depending on ":core". This will allow us to start moving integration tests out of core and into the appropriate sub-module. This is done by adding ":test-common:test-common-runtime" as a testRuntimeOnly dependency rather than as a testImplementation dependency. A trivial example was added to QuorumControllerTest to illustrate this. Reviewers: Ismael Juma <ismael@juma.me.uk>, Chia-Ping Tsai <chia7712@gmail.com>
2025-01-24 22:03:43 +08:00
<allow pkg="org.apache.kafka.common.test.api" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<!-- persistent collection factories/non-library-specific wrappers -->
<allow pkg="org.apache.kafka.server.immutable" exact-match="true" />
<subpackage name="common">
<subpackage name="metadata">
<allow pkg="com.fasterxml.jackson" />
<allow pkg="org.apache.kafka.common.protocol" />
<allow pkg="org.apache.kafka.common.protocol.types" />
<allow pkg="org.apache.kafka.common.message" />
<allow pkg="org.apache.kafka.common.metadata" />
</subpackage>
KAFKA-15355: Message schema changes (#14290) Reviewers: Christo Lolov <lolovc@amazon.com>, Colin P. McCabe <cmccabe@apache.org>, Proven Provenzano <pprovenzano@confluent.io>, Ron Dagostino <rdagostino@confluent.io>
2023-11-02 21:46:05 +08:00
<subpackage name="errors">
<allow pkg="org.apache.kafka.common.metadata" />
</subpackage>
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
</subpackage>
<subpackage name="controller">
<allow pkg="org.apache.kafka.clients" />
<allow pkg="org.apache.kafka.clients.admin" />
<allow pkg="org.apache.kafka.common.acl" />
<allow pkg="org.apache.kafka.common.annotation" />
<allow pkg="org.apache.kafka.common.config" />
<allow pkg="org.apache.kafka.common.feature" />
<allow pkg="org.apache.kafka.common.internals" />
<allow pkg="org.apache.kafka.common.message" />
<allow pkg="org.apache.kafka.common.metadata" />
<allow pkg="org.apache.kafka.common.network" />
<allow pkg="org.apache.kafka.common.protocol" />
<allow pkg="org.apache.kafka.common.quota" />
<allow pkg="org.apache.kafka.common.requests" />
<allow pkg="org.apache.kafka.common.resource" />
<allow pkg="org.apache.kafka.controller" />
<allow pkg="org.apache.kafka.image" />
<allow pkg="org.apache.kafka.image.writer" />
<allow pkg="org.apache.kafka.metadata" />
<allow pkg="org.apache.kafka.metadata.authorizer" />
<allow pkg="org.apache.kafka.metadata.migration" />
MINOR: Move `ControllerPurgatory` to `server-common` (#13555) This patch renames from `ControllerPurgatory` to `DeferredEventQueue` and moves it from the `metadata` module to `server-common` module. Reviewers: Alexandre Dupriez <alexandre.dupriez@gmail.com>, Ziming Deng <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@apache.org>
2023-04-21 17:19:04 +08:00
<allow pkg="org.apache.kafka.deferred" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<allow pkg="org.apache.kafka.queue" />
<allow pkg="org.apache.kafka.raft" />
<allow pkg="org.apache.kafka.server.authorizer" />
<allow pkg="org.apache.kafka.server.common" />
<allow pkg="org.apache.kafka.server.config" />
<allow pkg="org.apache.kafka.server.fault" />
KAFKA-14996: Handle overly large user operations on the kcontroller (#13742) Previously, if a user tried to perform an overly large batch operation on the KRaft controller (such as creating a million topics), we would create a very large number of records in memory. Our attempt to write these records to the Raft layer would fail, because there were too many to fit in an atomic batch. This failure, in turn, would trigger a controller failover. (Note: I am assuming here that no topic creation policy was in place that would prevent the creation of a million topics. I am also assuming that the user operation must be done atomically, which is true for all current user operations, since we have not implemented KIP-868 yet.) With this PR, we fail immediately when the number of records we have generated exceeds the threshold that we can apply. This failure does not generate a controller failover. We also now fail with a PolicyViolationException rather than an UnknownServerException. In order to implement this in a simple way, this PR adds the BoundedList class, which wraps any list and adds a maximum length. Attempts to grow the list beyond this length cause an exception to be thrown. Reviewers: David Arthur <mumrah@gmail.com>, Ismael Juma <ijuma@apache.org>, Divij Vaidya <diviv@amazon.com>
2023-05-27 04:16:17 +08:00
<allow pkg="org.apache.kafka.server.mutable" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<allow pkg="org.apache.kafka.server.policy"/>
<allow pkg="org.apache.kafka.server.util"/>
<allow pkg="org.apache.kafka.snapshot" />
<allow pkg="org.apache.kafka.test" />
<allow pkg="org.apache.kafka.timeline" />
KAFKA-15183: Add more controller, loader, snapshot emitter metrics (#14010) Implement some of the metrics from KIP-938: Add more metrics for measuring KRaft performance. Add these metrics to QuorumControllerMetrics: kafka.controller:type=KafkaController,name=TimedOutBrokerHeartbeatCount kafka.controller:type=KafkaController,name=EventQueueOperationsStartedCount kafka.controller:type=KafkaController,name=EventQueueOperationsTimedOutCount kafka.controller:type=KafkaController,name=NewActiveControllersCount Create LoaderMetrics with these new metrics: kafka.server:type=MetadataLoader,name=CurrentMetadataVersion kafka.server:type=MetadataLoader,name=HandleLoadSnapshotCount Create SnapshotEmitterMetrics with these new metrics: kafka.server:type=SnapshotEmitter,name=LatestSnapshotGeneratedBytes kafka.server:type=SnapshotEmitter,name=LatestSnapshotGeneratedAgeMs Reviewers: Ron Dagostino <rndgstn@gmail.com>
2023-07-25 12:13:58 +08:00
<subpackage name="metrics">
<allow pkg="com.yammer.metrics"/>
<allow pkg="org.apache.kafka.common.metrics" />
<allow pkg="org.apache.kafka.server.metrics" />
</subpackage>
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
</subpackage>
<subpackage name="image">
<allow pkg="org.apache.kafka.common.config" />
<allow pkg="org.apache.kafka.common.message" />
<allow pkg="org.apache.kafka.common.metadata" />
<allow pkg="org.apache.kafka.common.protocol" />
<allow pkg="org.apache.kafka.common.quota" />
<allow pkg="org.apache.kafka.common.record" />
<allow pkg="org.apache.kafka.common.requests" />
<allow pkg="org.apache.kafka.common.resource" />
<allow pkg="org.apache.kafka.image" />
<allow pkg="org.apache.kafka.image.writer" />
<allow pkg="org.apache.kafka.metadata" />
<allow pkg="org.apache.kafka.queue" />
<allow pkg="org.apache.kafka.clients.admin" />
<allow pkg="org.apache.kafka.raft" />
<allow pkg="org.apache.kafka.server.common" />
KAFKA-15853 Move quota configs into server-common package (#15774) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-04-24 13:05:18 +08:00
<allow pkg="org.apache.kafka.server.config" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<allow pkg="org.apache.kafka.server.fault" />
<allow pkg="org.apache.kafka.server.util" />
<allow pkg="org.apache.kafka.snapshot" />
<allow pkg="org.apache.kafka.test" />
KAFKA-15183: Add more controller, loader, snapshot emitter metrics (#14010) Implement some of the metrics from KIP-938: Add more metrics for measuring KRaft performance. Add these metrics to QuorumControllerMetrics: kafka.controller:type=KafkaController,name=TimedOutBrokerHeartbeatCount kafka.controller:type=KafkaController,name=EventQueueOperationsStartedCount kafka.controller:type=KafkaController,name=EventQueueOperationsTimedOutCount kafka.controller:type=KafkaController,name=NewActiveControllersCount Create LoaderMetrics with these new metrics: kafka.server:type=MetadataLoader,name=CurrentMetadataVersion kafka.server:type=MetadataLoader,name=HandleLoadSnapshotCount Create SnapshotEmitterMetrics with these new metrics: kafka.server:type=SnapshotEmitter,name=LatestSnapshotGeneratedBytes kafka.server:type=SnapshotEmitter,name=LatestSnapshotGeneratedAgeMs Reviewers: Ron Dagostino <rndgstn@gmail.com>
2023-07-25 12:13:58 +08:00
<subpackage name="loader">
<subpackage name="metrics">
<allow pkg="com.yammer.metrics"/>
<allow pkg="org.apache.kafka.common.metrics" />
<allow pkg="org.apache.kafka.controller.metrics" />
<allow pkg="org.apache.kafka.server.metrics" />
</subpackage>
</subpackage>
<subpackage name="publisher">
<subpackage name="metrics">
<allow pkg="com.yammer.metrics"/>
<allow pkg="org.apache.kafka.common.metrics" />
<allow pkg="org.apache.kafka.controller.metrics" />
<allow pkg="org.apache.kafka.server.metrics" />
</subpackage>
</subpackage>
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
</subpackage>
<subpackage name="metadata">
KAFKA-17565 Move MetadataCache interface to metadata module (#18801) ### Changes * Move MetadataCache interface to metadata module and change Scala function to Java. * Remove functions `getTopicPartitions`, `getAliveBrokers`, `topicNamesToIds`, `topicIdInfo`, and `getClusterMetadata` from MetadataCache interface, because these functions are only used in test code. ### Performance * ReplicaFetcherThreadBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.fetcher.ReplicaFetcherThreadBenchmark ``` * trunk ``` Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 2 4775.490 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 2 25730.790 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 2 55334.206 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 2 488427.547 ns/op ``` * branch ``` Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 2 4825.219 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 2 25985.662 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 2 56056.005 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 2 497138.573 ns/op ``` * KRaftMetadataRequestBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.metadata.KRaftMetadataRequestBenchmark ``` * trunk ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 500 avgt 2 884933.558 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 1000 avgt 2 1910054.621 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 5000 avgt 2 21778869.337 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 500 avgt 2 1537550.670 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 1000 avgt 2 3168237.805 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 5000 avgt 2 29699652.466 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 500 avgt 2 3501483.852 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 1000 avgt 2 7405481.182 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 5000 avgt 2 55839670.124 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 500 avgt 2 333.667 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 1000 avgt 2 339.685 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 5000 avgt 2 334.293 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 500 avgt 2 329.899 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 1000 avgt 2 347.537 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 5000 avgt 2 332.781 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 500 avgt 2 327.085 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 1000 avgt 2 325.206 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 5000 avgt 2 316.758 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 500 avgt 2 7.569 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 1000 avgt 2 7.565 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 5000 avgt 2 7.574 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 500 avgt 2 7.568 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 1000 avgt 2 7.557 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 5000 avgt 2 7.585 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 500 avgt 2 7.560 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 1000 avgt 2 7.554 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 5000 avgt 2 7.574 ns/op ``` * branch ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 500 avgt 2 910337.770 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 1000 avgt 2 1902351.360 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 5000 avgt 2 22215893.338 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 500 avgt 2 1572683.875 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 1000 avgt 2 3188560.081 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 5000 avgt 2 29984751.632 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 500 avgt 2 3413567.549 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 1000 avgt 2 7303174.254 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 5000 avgt 2 54293721.640 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 500 avgt 2 318.335 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 1000 avgt 2 331.386 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 5000 avgt 2 332.944 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 500 avgt 2 340.322 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 1000 avgt 2 330.294 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 5000 avgt 2 342.154 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 500 avgt 2 341.053 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 1000 avgt 2 335.458 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 5000 avgt 2 322.050 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 500 avgt 2 7.538 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 1000 avgt 2 7.548 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 5000 avgt 2 7.545 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 500 avgt 2 7.597 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 1000 avgt 2 7.567 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 5000 avgt 2 7.558 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 500 avgt 2 7.559 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 1000 avgt 2 7.615 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 5000 avgt 2 7.562 ns/op ``` * PartitionMakeFollowerBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.partition.PartitionMakeFollowerBenchmark ``` * trunk ``` Benchmark Mode Cnt Score Error Units PartitionMakeFollowerBenchmark.testMakeFollower avgt 2 158.816 ns/op ``` * branch ``` Benchmark Mode Cnt Score Error Units PartitionMakeFollowerBenchmark.testMakeFollower avgt 2 160.533 ns/op ``` * UpdateFollowerFetchStateBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.partition.UpdateFollowerFetchStateBenchmark ``` * trunk ``` Benchmark Mode Cnt Score Error Units UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBench avgt 2 4975.261 ns/op UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBenchNoChange avgt 2 4880.880 ns/op ``` * branch ``` Benchmark Mode Cnt Score Error Units UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBench avgt 2 5020.722 ns/op UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBenchNoChange avgt 2 4878.855 ns/op ``` * CheckpointBench ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.server.CheckpointBench ``` * trunk ``` Benchmark (numPartitions) (numTopics) Mode Cnt Score Error Units CheckpointBench.measureCheckpointHighWatermarks 3 100 thrpt 2 0.997 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 1000 thrpt 2 0.703 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 2000 thrpt 2 0.486 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 2 1.038 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 2 0.734 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 2 0.637 ops/ms ``` * branch ``` Benchmark (numPartitions) (numTopics) Mode Cnt Score Error Units CheckpointBench.measureCheckpointHighWatermarks 3 100 thrpt 2 0.990 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 1000 thrpt 2 0.659 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 2000 thrpt 2 0.508 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 2 0.923 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 2 0.736 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 2 0.637 ops/ms ``` * PartitionCreationBench ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.server.PartitionCreationBench ``` * trunk ``` Benchmark (numPartitions) (useTopicIds) Mode Cnt Score Error Units PartitionCreationBench.makeFollower 20 false avgt 2 5.997 ms/op PartitionCreationBench.makeFollower 20 true avgt 2 6.961 ms/op ``` * branch ``` Benchmark (numPartitions) (useTopicIds) Mode Cnt Score Error Units PartitionCreationBench.makeFollower 20 false avgt 2 6.212 ms/op PartitionCreationBench.makeFollower 20 true avgt 2 7.005 ms/op ``` Reviewers: Ismael Juma <ismael@juma.me.uk>, David Arthur <mumrah@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2025-03-17 23:59:11 +08:00
<allow pkg="org.apache.kafka.admin" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<allow pkg="org.apache.kafka.clients" />
<allow pkg="org.apache.kafka.common.acl" />
<allow pkg="org.apache.kafka.common.annotation" />
<allow pkg="org.apache.kafka.common.config" />
KAFKA-17565 Move MetadataCache interface to metadata module (#18801) ### Changes * Move MetadataCache interface to metadata module and change Scala function to Java. * Remove functions `getTopicPartitions`, `getAliveBrokers`, `topicNamesToIds`, `topicIdInfo`, and `getClusterMetadata` from MetadataCache interface, because these functions are only used in test code. ### Performance * ReplicaFetcherThreadBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.fetcher.ReplicaFetcherThreadBenchmark ``` * trunk ``` Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 2 4775.490 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 2 25730.790 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 2 55334.206 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 2 488427.547 ns/op ``` * branch ``` Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 2 4825.219 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 2 25985.662 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 2 56056.005 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 2 497138.573 ns/op ``` * KRaftMetadataRequestBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.metadata.KRaftMetadataRequestBenchmark ``` * trunk ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 500 avgt 2 884933.558 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 1000 avgt 2 1910054.621 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 5000 avgt 2 21778869.337 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 500 avgt 2 1537550.670 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 1000 avgt 2 3168237.805 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 5000 avgt 2 29699652.466 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 500 avgt 2 3501483.852 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 1000 avgt 2 7405481.182 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 5000 avgt 2 55839670.124 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 500 avgt 2 333.667 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 1000 avgt 2 339.685 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 5000 avgt 2 334.293 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 500 avgt 2 329.899 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 1000 avgt 2 347.537 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 5000 avgt 2 332.781 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 500 avgt 2 327.085 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 1000 avgt 2 325.206 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 5000 avgt 2 316.758 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 500 avgt 2 7.569 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 1000 avgt 2 7.565 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 5000 avgt 2 7.574 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 500 avgt 2 7.568 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 1000 avgt 2 7.557 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 5000 avgt 2 7.585 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 500 avgt 2 7.560 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 1000 avgt 2 7.554 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 5000 avgt 2 7.574 ns/op ``` * branch ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 500 avgt 2 910337.770 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 1000 avgt 2 1902351.360 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 5000 avgt 2 22215893.338 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 500 avgt 2 1572683.875 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 1000 avgt 2 3188560.081 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 5000 avgt 2 29984751.632 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 500 avgt 2 3413567.549 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 1000 avgt 2 7303174.254 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 5000 avgt 2 54293721.640 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 500 avgt 2 318.335 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 1000 avgt 2 331.386 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 5000 avgt 2 332.944 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 500 avgt 2 340.322 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 1000 avgt 2 330.294 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 5000 avgt 2 342.154 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 500 avgt 2 341.053 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 1000 avgt 2 335.458 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 5000 avgt 2 322.050 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 500 avgt 2 7.538 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 1000 avgt 2 7.548 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 5000 avgt 2 7.545 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 500 avgt 2 7.597 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 1000 avgt 2 7.567 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 5000 avgt 2 7.558 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 500 avgt 2 7.559 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 1000 avgt 2 7.615 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 5000 avgt 2 7.562 ns/op ``` * PartitionMakeFollowerBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.partition.PartitionMakeFollowerBenchmark ``` * trunk ``` Benchmark Mode Cnt Score Error Units PartitionMakeFollowerBenchmark.testMakeFollower avgt 2 158.816 ns/op ``` * branch ``` Benchmark Mode Cnt Score Error Units PartitionMakeFollowerBenchmark.testMakeFollower avgt 2 160.533 ns/op ``` * UpdateFollowerFetchStateBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.partition.UpdateFollowerFetchStateBenchmark ``` * trunk ``` Benchmark Mode Cnt Score Error Units UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBench avgt 2 4975.261 ns/op UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBenchNoChange avgt 2 4880.880 ns/op ``` * branch ``` Benchmark Mode Cnt Score Error Units UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBench avgt 2 5020.722 ns/op UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBenchNoChange avgt 2 4878.855 ns/op ``` * CheckpointBench ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.server.CheckpointBench ``` * trunk ``` Benchmark (numPartitions) (numTopics) Mode Cnt Score Error Units CheckpointBench.measureCheckpointHighWatermarks 3 100 thrpt 2 0.997 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 1000 thrpt 2 0.703 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 2000 thrpt 2 0.486 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 2 1.038 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 2 0.734 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 2 0.637 ops/ms ``` * branch ``` Benchmark (numPartitions) (numTopics) Mode Cnt Score Error Units CheckpointBench.measureCheckpointHighWatermarks 3 100 thrpt 2 0.990 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 1000 thrpt 2 0.659 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 2000 thrpt 2 0.508 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 2 0.923 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 2 0.736 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 2 0.637 ops/ms ``` * PartitionCreationBench ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.server.PartitionCreationBench ``` * trunk ``` Benchmark (numPartitions) (useTopicIds) Mode Cnt Score Error Units PartitionCreationBench.makeFollower 20 false avgt 2 5.997 ms/op PartitionCreationBench.makeFollower 20 true avgt 2 6.961 ms/op ``` * branch ``` Benchmark (numPartitions) (useTopicIds) Mode Cnt Score Error Units PartitionCreationBench.makeFollower 20 false avgt 2 6.212 ms/op PartitionCreationBench.makeFollower 20 true avgt 2 7.005 ms/op ``` Reviewers: Ismael Juma <ismael@juma.me.uk>, David Arthur <mumrah@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2025-03-17 23:59:11 +08:00
<allow pkg="org.apache.kafka.common.internals" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<allow pkg="org.apache.kafka.common.message" />
<allow pkg="org.apache.kafka.common.metadata" />
KAFKA-17565 Move MetadataCache interface to metadata module (#18801) ### Changes * Move MetadataCache interface to metadata module and change Scala function to Java. * Remove functions `getTopicPartitions`, `getAliveBrokers`, `topicNamesToIds`, `topicIdInfo`, and `getClusterMetadata` from MetadataCache interface, because these functions are only used in test code. ### Performance * ReplicaFetcherThreadBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.fetcher.ReplicaFetcherThreadBenchmark ``` * trunk ``` Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 2 4775.490 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 2 25730.790 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 2 55334.206 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 2 488427.547 ns/op ``` * branch ``` Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 2 4825.219 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 2 25985.662 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 2 56056.005 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 2 497138.573 ns/op ``` * KRaftMetadataRequestBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.metadata.KRaftMetadataRequestBenchmark ``` * trunk ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 500 avgt 2 884933.558 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 1000 avgt 2 1910054.621 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 5000 avgt 2 21778869.337 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 500 avgt 2 1537550.670 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 1000 avgt 2 3168237.805 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 5000 avgt 2 29699652.466 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 500 avgt 2 3501483.852 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 1000 avgt 2 7405481.182 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 5000 avgt 2 55839670.124 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 500 avgt 2 333.667 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 1000 avgt 2 339.685 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 5000 avgt 2 334.293 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 500 avgt 2 329.899 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 1000 avgt 2 347.537 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 5000 avgt 2 332.781 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 500 avgt 2 327.085 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 1000 avgt 2 325.206 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 5000 avgt 2 316.758 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 500 avgt 2 7.569 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 1000 avgt 2 7.565 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 5000 avgt 2 7.574 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 500 avgt 2 7.568 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 1000 avgt 2 7.557 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 5000 avgt 2 7.585 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 500 avgt 2 7.560 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 1000 avgt 2 7.554 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 5000 avgt 2 7.574 ns/op ``` * branch ``` Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 500 avgt 2 910337.770 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 1000 avgt 2 1902351.360 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 5000 avgt 2 22215893.338 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 500 avgt 2 1572683.875 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 1000 avgt 2 3188560.081 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 5000 avgt 2 29984751.632 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 500 avgt 2 3413567.549 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 1000 avgt 2 7303174.254 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 5000 avgt 2 54293721.640 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 500 avgt 2 318.335 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 1000 avgt 2 331.386 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 5000 avgt 2 332.944 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 500 avgt 2 340.322 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 1000 avgt 2 330.294 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 5000 avgt 2 342.154 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 500 avgt 2 341.053 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 1000 avgt 2 335.458 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 5000 avgt 2 322.050 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 500 avgt 2 7.538 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 1000 avgt 2 7.548 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 10 5000 avgt 2 7.545 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 500 avgt 2 7.597 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 1000 avgt 2 7.567 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 20 5000 avgt 2 7.558 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 500 avgt 2 7.559 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 1000 avgt 2 7.615 ns/op KRaftMetadataRequestBenchmark.testTopicIdInfo 50 5000 avgt 2 7.562 ns/op ``` * PartitionMakeFollowerBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.partition.PartitionMakeFollowerBenchmark ``` * trunk ``` Benchmark Mode Cnt Score Error Units PartitionMakeFollowerBenchmark.testMakeFollower avgt 2 158.816 ns/op ``` * branch ``` Benchmark Mode Cnt Score Error Units PartitionMakeFollowerBenchmark.testMakeFollower avgt 2 160.533 ns/op ``` * UpdateFollowerFetchStateBenchmark ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.partition.UpdateFollowerFetchStateBenchmark ``` * trunk ``` Benchmark Mode Cnt Score Error Units UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBench avgt 2 4975.261 ns/op UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBenchNoChange avgt 2 4880.880 ns/op ``` * branch ``` Benchmark Mode Cnt Score Error Units UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBench avgt 2 5020.722 ns/op UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBenchNoChange avgt 2 4878.855 ns/op ``` * CheckpointBench ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.server.CheckpointBench ``` * trunk ``` Benchmark (numPartitions) (numTopics) Mode Cnt Score Error Units CheckpointBench.measureCheckpointHighWatermarks 3 100 thrpt 2 0.997 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 1000 thrpt 2 0.703 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 2000 thrpt 2 0.486 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 2 1.038 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 2 0.734 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 2 0.637 ops/ms ``` * branch ``` Benchmark (numPartitions) (numTopics) Mode Cnt Score Error Units CheckpointBench.measureCheckpointHighWatermarks 3 100 thrpt 2 0.990 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 1000 thrpt 2 0.659 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 2000 thrpt 2 0.508 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 2 0.923 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 2 0.736 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 2 0.637 ops/ms ``` * PartitionCreationBench ``` ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.server.PartitionCreationBench ``` * trunk ``` Benchmark (numPartitions) (useTopicIds) Mode Cnt Score Error Units PartitionCreationBench.makeFollower 20 false avgt 2 5.997 ms/op PartitionCreationBench.makeFollower 20 true avgt 2 6.961 ms/op ``` * branch ``` Benchmark (numPartitions) (useTopicIds) Mode Cnt Score Error Units PartitionCreationBench.makeFollower 20 false avgt 2 6.212 ms/op PartitionCreationBench.makeFollower 20 true avgt 2 7.005 ms/op ``` Reviewers: Ismael Juma <ismael@juma.me.uk>, David Arthur <mumrah@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2025-03-17 23:59:11 +08:00
<allow pkg="org.apache.kafka.common.network" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<allow pkg="org.apache.kafka.common.protocol" />
KAFKA-14859: SCRAM ZK to KRaft migration with dual write (#13628) Handle migrating SCRAM records in ZK when migrating from ZK to KRaft. This includes handling writing back SCRAM records to ZK while in dual write mode where metadata updates are written to both the KRaft metadata log and to ZK. This allows for rollback of migration to include SCRAM metadata changes. Reviewers: David Arthur <mumrah@gmail.com>
2023-05-01 21:56:04 +08:00
<allow pkg="org.apache.kafka.common.quota" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
<allow pkg="org.apache.kafka.common.record" />
<allow pkg="org.apache.kafka.common.resource" />
<allow pkg="org.apache.kafka.common.requests" />
<allow pkg="org.apache.kafka.image" />
<allow pkg="org.apache.kafka.metadata" />
<allow pkg="org.apache.kafka.queue" />
<allow pkg="org.apache.kafka.raft" />
<allow pkg="org.apache.kafka.server.authorizer" />
<allow pkg="org.apache.kafka.server.common" />
<allow pkg="org.apache.kafka.server.fault" />
<allow pkg="org.apache.kafka.server.config" />
<allow pkg="org.apache.kafka.server.util"/>
<allow pkg="org.apache.kafka.test" />
<subpackage name="authorizer">
<allow pkg="org.apache.kafka.common.acl" />
<allow pkg="org.apache.kafka.common.requests" />
<allow pkg="org.apache.kafka.common.resource" />
<allow pkg="org.apache.kafka.controller" />
<allow pkg="org.apache.kafka.metadata" />
<allow pkg="org.apache.kafka.common.internals" />
KAFKA-18888: Add KIP-877 support to Authorizer (#19050) This also adds metrics to StandardAuthorizer Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Ken Huang <s7133700@gmail.com>, Jhen-Yung Hsu <jhenyunghsu@gmail.com>, TaiJuWu <tjwu1217@gmail.com>
2025-04-16 01:40:24 +08:00
<allow pkg="org.apache.kafka.common.metrics" />
<allow pkg="org.apache.kafka.common.metrics.internals" />
<allow pkg="org.apache.kafka.common.metrics.stats" />
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
</subpackage>
<subpackage name="bootstrap">
<allow pkg="org.apache.kafka.snapshot" />
</subpackage>
<subpackage name="fault">
<allow pkg="org.apache.kafka.server.fault" />
</subpackage>
KAFKA-7632: Support Compression Levels (KIP-390) (#15516) Reviewers: Jun Rao <jun@confluent.io>, Luke Chen <showuon@gmail.com> Co-authored-by: Lee Dongjin <dongjin@apache.org>
2024-05-21 23:58:49 +08:00
<subpackage name="migration">
<allow pkg="org.apache.kafka.controller" />
</subpackage>
KAFKA-16518: Implement KIP-853 flags for storage-tool.sh (#16669) As part of KIP-853, storage-tool.sh now has two new flags: --standalone, and --initial-voters. This PR implements these two flags in storage-tool.sh. There are currently two valid ways to format a cluster: The pre-KIP-853 way, where you use a statically configured controller quorum. In this case, neither --standalone nor --initial-voters may be specified, and kraft.version must be set to 0. The KIP-853 way, where one of --standalone and --initial-voters must be specified with the initial value of the dynamic controller quorum. In this case, kraft.version must be set to 1. This PR moves the formatting logic out of StorageTool.scala and into Formatter.java. The tool file was never intended to get so huge, or to implement complex logic like generating metadata records. Those things should be done by code in the metadata or raft gradle modules. This is also useful for junit tests, which often need to do formatting. (The 'info' and 'random-uuid' commands remain in StorageTool.scala, for now.) Reviewers: José Armando García Sancio <jsancio@apache.org>
2024-08-03 06:47:45 +08:00
<subpackage name="storage">
<allow pkg="org.apache.kafka.common.internals" />
<allow pkg="org.apache.kafka.snapshot" />
</subpackage>
KAFKA-7632: Support Compression Levels (KIP-390) (#15516) Reviewers: Jun Rao <jun@confluent.io>, Luke Chen <showuon@gmail.com> Co-authored-by: Lee Dongjin <dongjin@apache.org>
2024-05-21 23:58:49 +08:00
<subpackage name="util">
<allow class="org.apache.kafka.common.compress.Compression" exact-match="true" />
</subpackage>
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280) topic counts. Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-18 05:52:28 +08:00
</subpackage>
</import-control>