2022-02-17 14:35:36 +08:00
|
|
|
<?xml version="1.0"?>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<!DOCTYPE suppressions PUBLIC
|
|
|
|
|
"-//Puppy Crawl//DTD Suppressions 1.1//EN"
|
|
|
|
|
"http://www.puppycrawl.com/dtds/suppressions_1_1.dtd">
|
2022-02-17 14:35:36 +08:00
|
|
|
<!--
|
|
|
|
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
|
|
|
|
contributor license agreements. See the NOTICE file distributed with
|
|
|
|
|
this work for additional information regarding copyright ownership.
|
|
|
|
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
|
|
|
|
(the "License"); you may not use this file except in compliance with
|
|
|
|
|
the License. You may obtain a copy of the License at
|
|
|
|
|
|
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2022-02-17 14:35:36 +08:00
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
|
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
|
See the License for the specific language governing permissions and
|
|
|
|
|
limitations under the License.
|
|
|
|
|
-->
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppressions>
|
|
|
|
|
|
2017-06-28 08:47:00 +08:00
|
|
|
<!-- Note that [/\\] must be used as the path separator for cross-platform support -->
|
|
|
|
|
|
2019-09-25 23:58:54 +08:00
|
|
|
<!-- Generator -->
|
|
|
|
|
<suppress checks="CyclomaticComplexity|BooleanExpressionComplexity"
|
2024-10-04 03:13:38 +08:00
|
|
|
files="(SchemaGenerator|MessageDataGenerator|FieldSpec|FieldSpecPairIterator|FieldType).java"/>
|
2019-09-25 23:58:54 +08:00
|
|
|
<suppress checks="NPathComplexity"
|
2024-10-04 03:13:38 +08:00
|
|
|
files="(FieldSpecPairIterator|MessageDataGenerator|FieldSpec|WorkerSinkTask).java"/>
|
2019-09-25 23:58:54 +08:00
|
|
|
<suppress checks="JavaNCSS"
|
2021-08-31 06:39:25 +08:00
|
|
|
files="(ApiMessageType|FieldSpec|MessageDataGenerator|KafkaConsumerTest).java"/>
|
2019-11-21 08:40:18 +08:00
|
|
|
<suppress checks="MethodLength"
|
2021-01-12 06:17:52 +08:00
|
|
|
files="(FieldSpec|MessageDataGenerator).java"/>
|
2020-08-06 04:52:50 +08:00
|
|
|
<suppress id="dontUseSystemExit"
|
|
|
|
|
files="MessageGenerator.java"/>
|
2019-09-25 23:58:54 +08:00
|
|
|
|
2020-11-18 14:49:04 +08:00
|
|
|
<!-- core -->
|
2024-09-12 02:47:33 +08:00
|
|
|
<suppress checks="NPathComplexity" files="(ClusterTestExtensions|KafkaApisBuilder|SharePartition|SharePartitionManager).java"/>
|
KAFKA-14124: improve quorum controller fault handling (#12447)
Before trying to commit a batch of records to the __cluster_metadata log, the active controller
should try to apply them to its current in-memory state. If this application process fails, the
active controller process should exit, allowing another node to take leadership. This will prevent
most bad metadata records from ending up in the log and help to surface errors during testing.
Similarly, if the active controller attempts to renounce leadership, and the renunciation process
itself fails, the process should exit. This will help avoid bugs where the active controller
continues in an undefined state.
In contrast, standby controllers that experience metadata application errors should continue on, in
order to avoid a scenario where a bad record brings down the whole controller cluster. The
intended effect of these changes is to make it harder to commit a bad record to the metadata log,
but to continue to ride out the bad record as well as possible if such a record does get committed.
This PR introduces the FaultHandler interface to implement these concepts. In junit tests, we use a
FaultHandler implementation which does not exit the process. This allows us to avoid terminating
the gradle test runner, which would be very disruptive. It also allows us to ensure that the test
surfaces these exceptions, which we previously were not doing (the mock fault handler stores the
exception).
In addition to the above, this PR fixes a bug where RaftClient#resign was not being called from the
renounce() function. This bug could have resulted in the raft layer not being informed of an active
controller resigning.
Reviewers: David Arthur <mumrah@gmail.com>
2022-08-05 13:49:45 +08:00
|
|
|
<suppress checks="MethodLength"
|
|
|
|
|
files="(KafkaClusterTestKit).java"/>
|
2024-10-25 01:48:25 +08:00
|
|
|
<suppress checks="NPathComplexity" files="TestKitNodes.java"/>
|
2024-05-31 00:06:49 +08:00
|
|
|
<suppress checks="JavaNCSS"
|
2025-04-17 17:05:14 +08:00
|
|
|
files="(SharePartitionManagerTest|SharePartitionTest).java"/>
|
2024-09-12 02:47:33 +08:00
|
|
|
<suppress checks="ClassDataAbstractionCoupling|ClassFanOutComplexity" files="SharePartitionManagerTest"/>
|
2024-12-09 03:02:10 +08:00
|
|
|
<suppress checks="CyclomaticComplexity" files="SharePartition.java"/>
|
2020-11-18 14:49:04 +08:00
|
|
|
|
2024-01-25 04:16:09 +08:00
|
|
|
<!-- server tests -->
|
|
|
|
|
<suppress checks="MethodLength|JavaNCSS|NPath" files="DescribeTopicPartitionsRequestHandlerTest.java"/>
|
2024-02-29 16:38:42 +08:00
|
|
|
<suppress checks="CyclomaticComplexity" files="ListConsumerGroupTest.java"/>
|
2024-09-27 02:19:47 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity|CyclomaticComplexity|MethodLength|ParameterNumber|JavaNCSS|ImportControl" files="RequestConvertToJson.java"/>
|
2024-01-25 04:16:09 +08:00
|
|
|
|
2017-03-01 06:55:46 +08:00
|
|
|
<!-- Clients -->
|
2020-08-06 04:52:50 +08:00
|
|
|
<suppress id="dontUseSystemExit"
|
|
|
|
|
files="Exit.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
KAFKA-17642: PreVote response handling and ProspectiveState (#18240)
This PR implements the second part of KIP-996 and KAFKA-16164 (tasks KAFKA-16607, KAFKA-17642, KAFKA-17643, KAFKA-17675) which encompass the response handling of PreVotes, addition of new ProspectiveState, update to metrics, and addition of Raft simulation tests.
Voters now transition to ProspectiveState first before CandidateState to prevent unnecessary epoch bumps. Voters in ProspectiveState send PreVotes requests which are Vote requests with PreVote set to true.
Follower grants PreVotes if it has not yet fetched successfully from leader. Leader denies all PreVotes. Unattached, Prospective, Candidate, and Resigned will grant PreVotes if the requesting replica's log is at least as long as theirs. Granted PreVotes are not persisted like standard votes. It is possible for a voter to grant several PreVotes in the same epoch.
The only state which is allowed to transition directly to CandidateState is ProspectiveState. This happens on reception of majority of granted PreVotes or if at least one voter doesn't support PreVote requests.
Prospective will transition to Follower after election loss/timeout if it was already aware of last known leader and the leader's endpoint, or at any point if it discovers the leader.
Prospective will transition to Unattached after election loss/timeout if it does not know the leader endpoints.
After electionTimeout, Resigned now always transitions to Unattached and increases the epoch.
Prospective grants standard votes if it has not already granted a standard vote (no votedKey), has no leaderId, and the recipient's log is current enough
Candidate no longer backs off after election timeout. Candidate still backs off after election loss.
Reviewers: José Armando García Sancio <jsancio@apache.org>
2025-01-17 22:38:03 +08:00
|
|
|
files="(AbstractFetch|Sender|SenderTest|ConsumerCoordinator|KafkaConsumer|KafkaProducer|Utils|TransactionManager|TransactionManagerTest|KafkaAdminClient|NetworkClient|Admin|RaftClientTestContext|TestingMetricsInterceptingAdminClient).java"/>
|
2017-09-16 00:16:29 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
|
|
|
|
files="(SaslServerAuthenticator|SaslAuthenticatorTest).java"/>
|
2022-05-16 21:55:02 +08:00
|
|
|
<suppress checks="NPath"
|
2024-06-14 16:47:34 +08:00
|
|
|
files="(SaslServerAuthenticator).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2017-06-28 08:47:00 +08:00
|
|
|
files="Errors.java"/>
|
2017-03-25 03:38:36 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2017-06-28 08:47:00 +08:00
|
|
|
files="Utils.java"/>
|
2017-04-07 18:22:09 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2017-06-28 08:47:00 +08:00
|
|
|
files="AbstractRequest.java"/>
|
2017-04-07 18:22:09 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2017-06-28 08:47:00 +08:00
|
|
|
files="AbstractResponse.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="MethodLength"
|
2023-10-19 23:55:21 +08:00
|
|
|
files="(KerberosLogin|RequestResponseTest|ConnectMetricsRegistry|KafkaConsumer|AbstractStickyAssignor|AbstractRequest|AbstractResponse).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="ParameterNumber"
|
KAFKA-17642: PreVote response handling and ProspectiveState (#18240)
This PR implements the second part of KIP-996 and KAFKA-16164 (tasks KAFKA-16607, KAFKA-17642, KAFKA-17643, KAFKA-17675) which encompass the response handling of PreVotes, addition of new ProspectiveState, update to metrics, and addition of Raft simulation tests.
Voters now transition to ProspectiveState first before CandidateState to prevent unnecessary epoch bumps. Voters in ProspectiveState send PreVotes requests which are Vote requests with PreVote set to true.
Follower grants PreVotes if it has not yet fetched successfully from leader. Leader denies all PreVotes. Unattached, Prospective, Candidate, and Resigned will grant PreVotes if the requesting replica's log is at least as long as theirs. Granted PreVotes are not persisted like standard votes. It is possible for a voter to grant several PreVotes in the same epoch.
The only state which is allowed to transition directly to CandidateState is ProspectiveState. This happens on reception of majority of granted PreVotes or if at least one voter doesn't support PreVote requests.
Prospective will transition to Follower after election loss/timeout if it was already aware of last known leader and the leader's endpoint, or at any point if it discovers the leader.
Prospective will transition to Unattached after election loss/timeout if it does not know the leader endpoints.
After electionTimeout, Resigned now always transitions to Unattached and increases the epoch.
Prospective grants standard votes if it has not already granted a standard vote (no votedKey), has no leaderId, and the recipient's log is current enough
Candidate no longer backs off after election timeout. Candidate still backs off after election loss.
Reviewers: José Armando García Sancio <jsancio@apache.org>
2025-01-17 22:38:03 +08:00
|
|
|
files="(NetworkClient|FieldSpec|KafkaProducer).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
2023-12-01 21:37:21 +08:00
|
|
|
files="(KafkaConsumer|ConsumerCoordinator).java"/>
|
2017-09-06 08:36:53 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
2023-09-06 02:57:51 +08:00
|
|
|
files="(RecordAccumulator|Sender).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
|
|
|
|
files="ConfigDef.java"/>
|
2017-03-25 03:38:36 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
|
|
|
|
files="DefaultRecordBatch.java"/>
|
2017-08-26 01:38:15 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
2021-09-17 00:17:15 +08:00
|
|
|
files="MemoryRecordsBuilder.java"/>
|
2023-12-06 01:50:33 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
|
|
|
|
files="ClientUtils.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
KAFKA-17642: PreVote response handling and ProspectiveState (#18240)
This PR implements the second part of KIP-996 and KAFKA-16164 (tasks KAFKA-16607, KAFKA-17642, KAFKA-17643, KAFKA-17675) which encompass the response handling of PreVotes, addition of new ProspectiveState, update to metrics, and addition of Raft simulation tests.
Voters now transition to ProspectiveState first before CandidateState to prevent unnecessary epoch bumps. Voters in ProspectiveState send PreVotes requests which are Vote requests with PreVote set to true.
Follower grants PreVotes if it has not yet fetched successfully from leader. Leader denies all PreVotes. Unattached, Prospective, Candidate, and Resigned will grant PreVotes if the requesting replica's log is at least as long as theirs. Granted PreVotes are not persisted like standard votes. It is possible for a voter to grant several PreVotes in the same epoch.
The only state which is allowed to transition directly to CandidateState is ProspectiveState. This happens on reception of majority of granted PreVotes or if at least one voter doesn't support PreVote requests.
Prospective will transition to Follower after election loss/timeout if it was already aware of last known leader and the leader's endpoint, or at any point if it discovers the leader.
Prospective will transition to Unattached after election loss/timeout if it does not know the leader endpoints.
After electionTimeout, Resigned now always transitions to Unattached and increases the epoch.
Prospective grants standard votes if it has not already granted a standard vote (no votedKey), has no leaderId, and the recipient's log is current enough
Candidate no longer backs off after election timeout. Candidate still backs off after election loss.
Reviewers: José Armando García Sancio <jsancio@apache.org>
2025-01-17 22:38:03 +08:00
|
|
|
files="(KafkaConsumer|ConsumerCoordinator|AbstractFetch|KafkaProducer|AbstractRequest|AbstractResponse|TransactionManager|Admin|KafkaAdminClient|MockAdminClient|KafkaNetworkChannelTest).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
2023-12-06 01:50:33 +08:00
|
|
|
files="(Errors|SaslAuthenticatorTest|AgentTest|CoordinatorTest|NetworkClientTest).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="BooleanExpressionComplexity"
|
2024-05-21 23:58:49 +08:00
|
|
|
files="(Utils|Topic|Lz4BlockOutputStream|AclData|JoinGroupRequest).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="CyclomaticComplexity"
|
2024-03-09 00:43:44 +08:00
|
|
|
files="(AbstractFetch|ClientTelemetryReporter|ConsumerCoordinator|CommitRequestManager|FetchCollector|OffsetFetcherUtils|KafkaProducer|Sender|ConfigDef|KerberosLogin|AbstractRequest|AbstractResponse|Selector|SslFactory|SslTransportLayer|SaslClientAuthenticator|SaslClientCallbackHandler|SaslServerAuthenticator|AbstractCoordinator|TransactionManager|AbstractStickyAssignor|DefaultSslEngineFactory|Authorizer|RecordAccumulator|MemoryRecords|FetchSessionHandler|MockAdminClient).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="JavaNCSS"
|
2025-04-25 00:41:06 +08:00
|
|
|
files="(AbstractRequest|AbstractResponse|KerberosLogin|WorkerSinkTaskTest|TransactionManagerTest|SenderTest|KafkaAdminClient|ConsumerCoordinatorTest|KafkaAdminClientTest|KafkaProducerTest).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="NPathComplexity"
|
2025-04-10 04:17:38 +08:00
|
|
|
files="(AbstractMembershipManager|ConsumerCoordinator|BufferPool|MetricName|Node|ConfigDef|RecordBatch|SslFactory|SslTransportLayer|MetadataResponse|KerberosLogin|Selector|Sender|Serdes|TokenInformation|Agent|PluginUtils|MiniTrogdorCluster|TasksRequest|KafkaProducer|AbstractStickyAssignor|Authorizer|FetchSessionHandler|RecordAccumulator|Shell|MockConsumer).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2019-01-25 01:26:51 +08:00
|
|
|
<suppress checks="(JavaNCSS|CyclomaticComplexity|MethodLength)"
|
|
|
|
|
files="CoordinatorClient.java"/>
|
2019-10-04 06:59:31 +08:00
|
|
|
<suppress checks="(UnnecessaryParentheses|BooleanExpressionComplexity|CyclomaticComplexity|WhitespaceAfter|LocalVariableName)"
|
|
|
|
|
files="Murmur3.java"/>
|
2019-01-25 01:26:51 +08:00
|
|
|
|
2019-09-25 23:58:54 +08:00
|
|
|
<suppress checks="NPathComplexity"
|
2021-07-07 18:55:00 +08:00
|
|
|
files="MessageTest.java|OffsetFetchRequest.java"/>
|
2019-09-25 23:58:54 +08:00
|
|
|
|
2020-08-06 04:52:50 +08:00
|
|
|
<!-- Clients tests -->
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
2023-12-01 21:37:21 +08:00
|
|
|
files="(Sender|Fetcher|FetchRequestManager|OffsetFetcher|KafkaConsumer|Metrics|RequestResponse|TransactionManager|KafkaAdminClient|Message|KafkaProducer)Test.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2024-11-13 21:01:08 +08:00
|
|
|
files="(ConsumerCoordinator|KafkaConsumer|RequestResponse|Fetcher|FetchRequestManager|KafkaAdminClient|Message|KafkaProducer|NetworkClient)Test.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2017-12-08 08:16:54 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
|
|
|
|
files="MockAdminClient.java"/>
|
|
|
|
|
|
2021-12-11 01:54:52 +08:00
|
|
|
<suppress checks="CyclomaticComplexity"
|
2023-03-25 05:33:13 +08:00
|
|
|
files="(OffsetFetcher|RequestResponse)Test.java"/>
|
2021-12-11 01:54:52 +08:00
|
|
|
|
2017-04-03 10:41:44 +08:00
|
|
|
<suppress checks="JavaNCSS"
|
2025-04-06 13:02:45 +08:00
|
|
|
files="RequestResponseTest.java|FetcherTest.java|FetchRequestManagerTest.java|KafkaAdminClientTest.java|ConsumerMembershipManagerTest.java"/>
|
2017-04-03 10:41:44 +08:00
|
|
|
|
2018-05-22 14:17:42 +08:00
|
|
|
<suppress checks="NPathComplexity"
|
2021-03-02 22:20:47 +08:00
|
|
|
files="MemoryRecordsTest|MetricsTest|RequestResponseTest|TestSslUtils|AclAuthorizerBenchmark"/>
|
2018-05-22 14:17:42 +08:00
|
|
|
|
2019-10-04 06:59:31 +08:00
|
|
|
<suppress checks="(WhitespaceAround|LocalVariableName|ImportControl|AvoidStarImport)"
|
|
|
|
|
files="Murmur3Test.java"/>
|
|
|
|
|
|
2017-03-01 06:55:46 +08:00
|
|
|
<!-- Connect -->
|
|
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2021-07-01 12:13:07 +08:00
|
|
|
files="(AbstractHerder|DistributedHerder|Worker).java"/>
|
2020-05-28 09:18:36 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
|
|
|
|
files="Worker(|Test).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="MethodLength"
|
2024-05-24 04:23:18 +08:00
|
|
|
files="(DistributedHerder|DistributedConfig|KafkaConfigBackingStore|IncrementalCooperativeAssignor).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
2024-01-23 08:56:52 +08:00
|
|
|
files="Worker(SinkTask|SourceTask|Coordinator)?.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
2023-02-09 23:50:07 +08:00
|
|
|
files="(ConfigKeyInfo|DistributedHerder).java"/>
|
2023-10-20 21:52:37 +08:00
|
|
|
<suppress checks="DefaultComesLast"
|
|
|
|
|
files="LoggingResource.java" />
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
2024-02-02 00:33:04 +08:00
|
|
|
files="(RestServer|AbstractHerder|DistributedHerder|Worker(Test)?).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="BooleanExpressionComplexity"
|
|
|
|
|
files="JsonConverter.java"/>
|
|
|
|
|
|
|
|
|
|
<suppress checks="CyclomaticComplexity"
|
2020-05-27 13:07:34 +08:00
|
|
|
files="(FileStreamSourceTask|DistributedHerder|KafkaConfigBackingStore).java"/>
|
KAFKA-5142: Add Connect support for message headers (KIP-145)
**[KIP-145](https://cwiki.apache.org/confluence/display/KAFKA/KIP-145+-+Expose+Record+Headers+in+Kafka+Connect) has been accepted, and this PR implements KIP-145 except without the SMTs.**
Changed the Connect API and runtime to support message headers as described in [KIP-145](https://cwiki.apache.org/confluence/display/KAFKA/KIP-145+-+Expose+Record+Headers+in+Kafka+Connect).
The new `Header` interface defines an immutable representation of a Kafka header (key-value pair) with support for the Connect value types and schemas. This interface provides methods for easily converting between many of the built-in primitive, structured, and logical data types.
The new `Headers` interface defines an ordered collection of headers and is used to track all headers associated with a `ConnectRecord` (and thus `SourceRecord` and `SinkRecord`). This does allow multiple headers with the same key. The `Headers` contains methods for adding, removing, finding, and modifying headers. Convenience methods allow connectors and transforms to easily use and modify the headers for a record.
A new `HeaderConverter` interface is also defined to enable the Connect runtime framework to be able to serialize and deserialize headers between the in-memory representation and Kafka’s byte[] representation. A new `SimpleHeaderConverter` implementation has been added, and this serializes to strings and deserializes by inferring the schemas (`Struct` header values are serialized without the schemas, so they can only be deserialized as `Map` instances without a schema.) The `StringConverter`, `JsonConverter`, and `ByteArrayConverter` have all been extended to also be `HeaderConverter` implementations. Each connector can be configured with a different header converter, although by default the `SimpleHeaderConverter` is used to serialize header values as strings without schemas.
Unit and integration tests are added for `ConnectHeader` and `ConnectHeaders`, the two implementation classes for headers. Additional test methods are added for the methods added to the `Converter` implementations. Finally, the `ConnectRecord` object is already used heavily, so only limited tests need to be added while quite a few of the existing tests already cover the changes.
Author: Randall Hauch <rhauch@gmail.com>
Reviewers: Arjun Satish <arjun@confluent.io>, Ted Yu <yuzhihong@gmail.com>, Magesh Nandakumar <magesh.n.kumar@gmail.com>, Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes #4319 from rhauch/kafka-5142-b
2018-02-01 02:40:24 +08:00
|
|
|
<suppress checks="CyclomaticComplexity"
|
2024-05-24 04:23:18 +08:00
|
|
|
files="(JsonConverter|ConnectHeaders).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="JavaNCSS"
|
2024-05-24 04:23:18 +08:00
|
|
|
files="(KafkaConfigBackingStore|ConnectMetricsRegistry).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2020-05-24 21:56:27 +08:00
|
|
|
<suppress checks="NPathComplexity"
|
2023-12-12 05:48:16 +08:00
|
|
|
files="(DistributedHerder|AbstractHerder|RestClient|RestServer|JsonConverter|KafkaConfigBackingStore|FileStreamSourceTask|WorkerSourceTask|TopicAdmin).java"/>
|
2020-05-24 21:56:27 +08:00
|
|
|
|
2025-04-10 16:00:03 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
|
|
|
|
files="ShareConsumerTest.java"/>
|
|
|
|
|
|
2017-03-01 06:55:46 +08:00
|
|
|
<!-- connect tests-->
|
|
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
2021-03-19 22:03:36 +08:00
|
|
|
files="(DistributedHerder|KafkaBasedLog|WorkerSourceTaskWithTopicCreation|WorkerSourceTask)Test.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2019-09-26 00:23:03 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2024-03-07 02:31:33 +08:00
|
|
|
files="(WorkerSink|WorkerSource|ErrorHandling)Task(|WithTopicCreation|Mockito)Test.java"/>
|
2020-05-27 13:07:34 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
|
|
|
|
files="DistributedHerderTest.java"/>
|
|
|
|
|
|
|
|
|
|
<suppress checks="MethodLength"
|
2024-03-07 02:31:33 +08:00
|
|
|
files="(RequestResponse|WorkerSinkTask|WorkerSinkTaskMockito)Test.java"/>
|
2019-09-26 00:23:03 +08:00
|
|
|
|
2021-05-06 05:11:15 +08:00
|
|
|
<suppress checks="JavaNCSS"
|
2023-06-24 01:27:46 +08:00
|
|
|
files="(DistributedHerder|Worker)Test.java"/>
|
2021-05-06 05:11:15 +08:00
|
|
|
|
2025-02-12 01:27:15 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
|
|
|
|
files="WorkerSinkTaskTest.java"/>
|
|
|
|
|
|
2022-05-21 06:23:12 +08:00
|
|
|
<!-- Raft -->
|
|
|
|
|
<suppress checks="NPathComplexity"
|
KAFKA-16518: Implement KIP-853 flags for storage-tool.sh (#16669)
As part of KIP-853, storage-tool.sh now has two new flags: --standalone, and --initial-voters. This PR implements these two flags in storage-tool.sh.
There are currently two valid ways to format a cluster:
The pre-KIP-853 way, where you use a statically configured controller quorum. In this case, neither --standalone nor --initial-voters may be specified, and kraft.version must be set to 0.
The KIP-853 way, where one of --standalone and --initial-voters must be specified with the initial value of the dynamic controller quorum. In this case, kraft.version must be set to 1.
This PR moves the formatting logic out of StorageTool.scala and into Formatter.java. The tool file was never intended to get so huge, or to implement complex logic like generating metadata records. Those things should be done by code in the metadata or raft gradle modules. This is also useful for junit tests, which often need to do formatting. (The 'info' and 'random-uuid' commands remain in StorageTool.scala, for now.)
Reviewers: José Armando García Sancio <jsancio@apache.org>
2024-08-03 06:47:45 +08:00
|
|
|
files="(DynamicVoter|RecordsIterator).java"/>
|
2022-05-21 06:23:12 +08:00
|
|
|
|
KAFKA-17642: PreVote response handling and ProspectiveState (#18240)
This PR implements the second part of KIP-996 and KAFKA-16164 (tasks KAFKA-16607, KAFKA-17642, KAFKA-17643, KAFKA-17675) which encompass the response handling of PreVotes, addition of new ProspectiveState, update to metrics, and addition of Raft simulation tests.
Voters now transition to ProspectiveState first before CandidateState to prevent unnecessary epoch bumps. Voters in ProspectiveState send PreVotes requests which are Vote requests with PreVote set to true.
Follower grants PreVotes if it has not yet fetched successfully from leader. Leader denies all PreVotes. Unattached, Prospective, Candidate, and Resigned will grant PreVotes if the requesting replica's log is at least as long as theirs. Granted PreVotes are not persisted like standard votes. It is possible for a voter to grant several PreVotes in the same epoch.
The only state which is allowed to transition directly to CandidateState is ProspectiveState. This happens on reception of majority of granted PreVotes or if at least one voter doesn't support PreVote requests.
Prospective will transition to Follower after election loss/timeout if it was already aware of last known leader and the leader's endpoint, or at any point if it discovers the leader.
Prospective will transition to Unattached after election loss/timeout if it does not know the leader endpoints.
After electionTimeout, Resigned now always transitions to Unattached and increases the epoch.
Prospective grants standard votes if it has not already granted a standard vote (no votedKey), has no leaderId, and the recipient's log is current enough
Candidate no longer backs off after election timeout. Candidate still backs off after election loss.
Reviewers: José Armando García Sancio <jsancio@apache.org>
2025-01-17 22:38:03 +08:00
|
|
|
<suppress checks="JavaNCSS"
|
|
|
|
|
files="(KafkaRaftClientTest).java"/>
|
|
|
|
|
|
2017-03-01 06:55:46 +08:00
|
|
|
<!-- Streams -->
|
|
|
|
|
<suppress checks="ClassFanOutComplexity"
|
KAFKA-17411: Create local state Standbys on start (#16922)
Instead of waiting until Tasks are assigned to us, we pre-emptively
create a StandbyTask for each non-empty Task directory found on-disk.
We do this before starting any StreamThreads, and on our first
assignment (after joining the consumer group), we recycle any of these
StandbyTasks that were assigned to us, either as an Active or a
Standby.
We can't just use these "initial Standbys" as-is, because they were
constructed outside the context of a StreamThread, so we first have to
update them with the context (log context, ChangelogReader, and source
topics) of the thread that it has been assigned to.
The motivation for this is to (in a later commit) read StateStore
offsets for unowned Tasks from the StateStore itself, rather than the
.checkpoint file, which we plan to deprecate and remove.
There are a few additional benefits:
Initializing these Tasks on start-up, instead of on-assignment, will
reduce the time between a member joining the consumer group and beginning
processing. This is especially important when active tasks are being moved
over, for example, as part of a rolling restart.
If a Task has corrupt data on-disk, it will be discovered on startup and
wiped under EOS. This is preferable to wiping the state after being
assigned the Task, because another instance may have non-corrupt data and
would not need to restore (as much).
There is a potential performance impact: we open all on-disk Task
StateStores, and keep them all open until we have our first assignment.
This could require large amounts of memory, in particular when there are
a large number of local state stores on-disk.
However, since old local state for Tasks we don't own is automatically
cleaned up after a period of time, in practice, we will almost always
only be dealing with the state that was last assigned to the local
instance.
Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Bruno Cadonna <cadonna@apache.org>, Matthias Sax <mjsax@apache.org>
2024-10-30 03:59:25 +08:00
|
|
|
files="(KafkaStreams|KStreamImpl|KTableImpl|InternalTopologyBuilder|StreamsPartitionAssignor|StreamThread|IQv2StoreIntegrationTest|KStreamImplTest|RocksDBStore|StreamTask|TaskManager).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="MethodLength"
|
2020-06-18 08:04:43 +08:00
|
|
|
files="KTableImpl.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2020-02-05 13:06:39 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
2024-09-27 08:07:00 +08:00
|
|
|
files="(StreamThread|ActiveTaskCreator).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
2024-06-12 03:24:44 +08:00
|
|
|
files="(InternalTopologyBuilder|KafkaStreams|KStreamImpl|KTableImpl|StreamsPartitionAssignor).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="CyclomaticComplexity"
|
2022-04-16 02:28:43 +08:00
|
|
|
files="(KafkaStreams|StreamsPartitionAssignor|StreamThread|TaskManager|PartitionGroup|SubscriptionWrapperSerde|AssignorConfiguration).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2020-05-28 04:55:29 +08:00
|
|
|
<suppress checks="StaticVariableName"
|
|
|
|
|
files="StreamsMetricsImpl.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="NPathComplexity"
|
2024-06-09 10:48:43 +08:00
|
|
|
files="(KafkaStreams|StreamsPartitionAssignor|StreamThread|TaskManager|TaskAssignmentUtils|GlobalStateManagerImpl|KStreamImplJoin|TopologyConfig|KTableKTableOuterJoin).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2019-09-20 07:38:18 +08:00
|
|
|
<suppress checks="(FinalLocalVariable|UnnecessaryParentheses|BooleanExpressionComplexity|CyclomaticComplexity|WhitespaceAfter|LocalVariableName)"
|
|
|
|
|
files="Murmur3.java"/>
|
|
|
|
|
|
2020-09-10 03:02:19 +08:00
|
|
|
<suppress checks="(NPathComplexity|CyclomaticComplexity)"
|
2023-07-27 07:02:52 +08:00
|
|
|
files="(KStreamSlidingWindowAggregate|RackAwareTaskAssignor).java"/>
|
2020-09-10 03:02:19 +08:00
|
|
|
|
2018-08-04 04:19:46 +08:00
|
|
|
<!-- suppress FinalLocalVariable outside of the streams package. -->
|
|
|
|
|
<suppress checks="FinalLocalVariable"
|
|
|
|
|
files="^(?!.*[\\/]org[\\/]apache[\\/]kafka[\\/]streams[\\/].*$)"/>
|
|
|
|
|
|
2020-08-06 04:52:50 +08:00
|
|
|
<!-- Generated code -->
|
2024-08-26 17:24:27 +08:00
|
|
|
<suppress checks="[a-zA-Z0-9]*" files="[\\/]generated[\\/]"/>
|
KAFKA-8868: Generate SubscriptionInfo protocol message (#7248)
Rather than maintain hand coded protocol serialization code, Streams could use the same code-generation framework as Clients/Core.
There isn't a perfect match, since the code generation framework includes an assumption that you're generating "protocol messages", rather than just arbitrary blobs, but I think it's close enough to justify using it, and improving it over time.
Using the code generation allows us to drop a lot of detail-oriented, brittle, and hard-to-maintain serialization logic in favor of a schema spec.
Reviewers: Colin P. McCabe <cmccabe@apache.org>, Boyang Chen <boyang@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2019-11-02 01:03:55 +08:00
|
|
|
|
2020-07-31 01:29:39 +08:00
|
|
|
<suppress checks="ImportControl" files="FetchResponseData.java"/>
|
2020-08-14 00:52:23 +08:00
|
|
|
<suppress checks="ImportControl" files="RecordsSerdeTest.java"/>
|
2024-08-21 03:34:10 +08:00
|
|
|
<suppress checks="ImportControl" files="ClientTelemetryTest.java"/>
|
2024-10-03 00:47:29 +08:00
|
|
|
<suppress checks="ImportControl" files="AdminFenceProducersTest.java"/>
|
2020-07-31 01:29:39 +08:00
|
|
|
|
2018-01-30 09:21:48 +08:00
|
|
|
<!-- Streams tests -->
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2024-03-09 02:57:56 +08:00
|
|
|
files="(RecordCollectorTest|StreamsPartitionAssignorTest|StreamThreadTest|StreamTaskTest|TaskManagerTest|TopologyTestDriverTest|KafkaStreamsTest|EosIntegrationTest|RestoreIntegrationTest).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="MethodLength"
|
2023-04-12 11:42:55 +08:00
|
|
|
files="(EosIntegrationTest|EosV2UpgradeIntegrationTest|KStreamKStreamJoinTest|RocksDBWindowStoreTest|StreamStreamJoinIntegrationTest).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
2017-06-28 08:47:00 +08:00
|
|
|
files=".*[/\\]streams[/\\].*test[/\\].*.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="CyclomaticComplexity"
|
2025-02-19 08:18:29 +08:00
|
|
|
files="(EosV2UpgradeIntegrationTest|KStreamKStreamJoinTest|KTableKTableForeignKeyJoinIntegrationTest|RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest|RelationalSmokeTest|MockProcessorContextStateStoreTest|IQv2StoreIntegrationTest|StreamsConfigTest).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="JavaNCSS"
|
2023-10-02 21:41:21 +08:00
|
|
|
files="(EosV2UpgradeIntegrationTest|KStreamKStreamJoinTest|StreamThreadTest|TaskManagerTest|StreamTaskTest).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
|
|
|
|
<suppress checks="NPathComplexity"
|
2024-06-20 09:24:44 +08:00
|
|
|
files="(EosV2UpgradeIntegrationTest|EosTestDriver|KStreamKStreamJoinTest|KTableKTableForeignKeyJoinIntegrationTest|RelationalSmokeTest|MockProcessorContextStateStoreTest|TopologyTestDriverTest|IQv2StoreIntegrationTest).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2019-09-20 07:38:18 +08:00
|
|
|
<suppress checks="(FinalLocalVariable|WhitespaceAround|LocalVariableName|ImportControl|AvoidStarImport)"
|
|
|
|
|
files="Murmur3Test.java"/>
|
|
|
|
|
|
2020-09-12 05:38:17 +08:00
|
|
|
<suppress checks="MethodLength"
|
2024-06-20 09:24:44 +08:00
|
|
|
files="(KStreamSlidingWindowAggregateTest|KStreamKStreamLeftJoinTest|KStreamKStreamOuterJoinTest|KTableKTableForeignKeyJoinIntegrationTest).java"/>
|
2020-09-12 05:38:17 +08:00
|
|
|
|
2020-10-01 11:07:23 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
|
|
|
|
files="StreamTaskTest.java"/>
|
|
|
|
|
|
2020-08-06 04:52:50 +08:00
|
|
|
<!-- Streams test-utils -->
|
2018-01-30 09:21:48 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
|
|
|
|
files="TopologyTestDriver.java"/>
|
|
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
|
|
|
|
files="TopologyTestDriver.java"/>
|
|
|
|
|
|
2020-08-06 04:52:50 +08:00
|
|
|
<!-- Streams examples -->
|
|
|
|
|
<suppress id="dontUseSystemExit"
|
|
|
|
|
files="PageViewTypedDemo.java|PipeDemo.java|TemperatureDemo.java|WordCountDemo.java|WordCountProcessorDemo.java|WordCountTransformerDemo.java"/>
|
|
|
|
|
|
2017-03-01 06:55:46 +08:00
|
|
|
<!-- Tools -->
|
|
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
|
|
|
|
files="VerifiableConsumer.java"/>
|
|
|
|
|
<suppress checks="CyclomaticComplexity"
|
2024-12-15 03:05:46 +08:00
|
|
|
files="(AclCommand|ConsoleConsumer|DefaultMessageFormatter|StreamsResetter|ProducerPerformance|Agent).java"/>
|
2019-10-04 06:59:31 +08:00
|
|
|
<suppress checks="BooleanExpressionComplexity"
|
2024-08-26 17:30:57 +08:00
|
|
|
files="(StreamsResetter|DefaultMessageFormatter).java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
<suppress checks="NPathComplexity"
|
2024-12-15 03:05:46 +08:00
|
|
|
files="(AclCommand|DefaultMessageFormatter|ProducerPerformance|StreamsResetter|Agent|TransactionalMessageCopier|ReplicaVerificationTool|LineMessageReader).java"/>
|
2017-08-26 03:29:40 +08:00
|
|
|
<suppress checks="ImportControl"
|
|
|
|
|
files="SignalLogger.java"/>
|
|
|
|
|
<suppress checks="IllegalImport"
|
|
|
|
|
files="SignalLogger.java"/>
|
2018-03-20 21:51:45 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
|
|
|
|
files="ProduceBenchSpec.java"/>
|
2020-12-19 05:03:59 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
|
|
|
|
files="ConsumeBenchSpec.java"/>
|
2019-09-09 10:49:13 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
|
|
|
|
files="SustainedConnectionSpec.java"/>
|
2020-08-06 04:52:50 +08:00
|
|
|
<suppress id="dontUseSystemExit"
|
|
|
|
|
files="VerifiableConsumer.java"/>
|
|
|
|
|
<suppress id="dontUseSystemExit"
|
|
|
|
|
files="VerifiableProducer.java"/>
|
2025-01-17 20:03:32 +08:00
|
|
|
<suppress id="dontUseSystemExit"
|
|
|
|
|
files="VerifiableShareConsumer.java"/>
|
2017-03-01 06:55:46 +08:00
|
|
|
|
2021-02-10 06:11:35 +08:00
|
|
|
<!-- Shell -->
|
|
|
|
|
<suppress checks="CyclomaticComplexity"
|
2021-10-27 04:23:20 +08:00
|
|
|
files="(GlobComponent|MetadataNodeManager).java"/>
|
2022-08-25 14:09:01 +08:00
|
|
|
<suppress checks="MethodLength"
|
|
|
|
|
files="(MetadataNodeManager).java"/>
|
|
|
|
|
<suppress checks="JavaNCSS"
|
|
|
|
|
files="(MetadataNodeManager).java"/>
|
2021-02-10 06:11:35 +08:00
|
|
|
|
2021-01-15 01:58:52 +08:00
|
|
|
<!-- metadata -->
|
KAFKA-12276: Add the quorum controller code (#10070)
The quorum controller stores metadata in the KIP-500 metadata log, not in Apache
ZooKeeper. Each controller node is a voter in the metadata quorum. The leader of the
quorum is the active controller, which processes write requests. The followers are standby
controllers, which replay the operations written to the log. If the active controller goes away,
a standby controller can take its place.
Like the ZooKeeper-based controller, the quorum controller is based on an event queue
backed by a single-threaded executor. However, unlike the ZK-based controller, the quorum
controller can have multiple operations in flight-- it does not need to wait for one operation
to be finished before starting another. Therefore, calls into the QuorumController return
CompleteableFuture objects which are completed with either a result or an error when the
operation is done. The QuorumController will also time out operations that have been
sitting on the queue too long without being processed. In this case, the future is completed
with a TimeoutException.
The controller uses timeline data structures to store multiple "versions" of its in-memory
state simultaneously. "Read operations" read only committed state, which is slightly older
than the most up-to-date in-memory state. "Write operations" read and write the latest
in-memory state. However, we can not return a successful result for a write operation until
its state has been committed to the log. Therefore, if a client receives an RPC response, it
knows that the requested operation has been performed, and can not be undone by a
controller failover.
Reviewers: Jun Rao <junrao@gmail.com>, Ron Dagostino <rdagostino@confluent.io>
2021-02-20 10:03:23 +08:00
|
|
|
<suppress checks="ClassDataAbstractionCoupling"
|
2023-05-05 16:35:26 +08:00
|
|
|
files="(QuorumController|QuorumControllerTest|ReplicationControlManager|ReplicationControlManagerTest|ClusterControlManagerTest|KRaftMigrationDriverTest).java"/>
|
KAFKA-12276: Add the quorum controller code (#10070)
The quorum controller stores metadata in the KIP-500 metadata log, not in Apache
ZooKeeper. Each controller node is a voter in the metadata quorum. The leader of the
quorum is the active controller, which processes write requests. The followers are standby
controllers, which replay the operations written to the log. If the active controller goes away,
a standby controller can take its place.
Like the ZooKeeper-based controller, the quorum controller is based on an event queue
backed by a single-threaded executor. However, unlike the ZK-based controller, the quorum
controller can have multiple operations in flight-- it does not need to wait for one operation
to be finished before starting another. Therefore, calls into the QuorumController return
CompleteableFuture objects which are completed with either a result or an error when the
operation is done. The QuorumController will also time out operations that have been
sitting on the queue too long without being processed. In this case, the future is completed
with a TimeoutException.
The controller uses timeline data structures to store multiple "versions" of its in-memory
state simultaneously. "Read operations" read only committed state, which is slightly older
than the most up-to-date in-memory state. "Write operations" read and write the latest
in-memory state. However, we can not return a successful result for a write operation until
its state has been committed to the log. Therefore, if a client receives an RPC response, it
knows that the requested operation has been performed, and can not be undone by a
controller failover.
Reviewers: Jun Rao <junrao@gmail.com>, Ron Dagostino <rdagostino@confluent.io>
2021-02-20 10:03:23 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2022-05-19 03:08:36 +08:00
|
|
|
files="(QuorumController|QuorumControllerTest|ReplicationControlManager|ReplicationControlManagerTest).java"/>
|
2022-04-15 01:04:32 +08:00
|
|
|
<suppress checks="(ParameterNumber|ClassDataAbstractionCoupling)"
|
KAFKA-12467: Implement QuorumController snapshot generation (#10366)
Implement controller-side snapshot generation.Implement QuorumController snapshot
generation. Note that this PR does not handle KRaft integration, just the internal
snapshot record generation and consumption logic.
Reading a snapshot is relatively straightforward. When the QuorumController
starts up, it loads the most recent snapshot. This is just a series of records
that we replay, plus a log offset ("snapshot epoch") that we advance to.
Writing a snapshot is more complex. There are several components:
the SnapshotWriter which persists the snapshot, the SnapshotGenerator
which manages writing each batch of records, and the SnapshotGeneratorManager
which interfaces the preceding two classes with the event queue.
Controller snapshots are done incrementally. In order to avoid blocking the
controller thread for a long time, we pull a few record batches at a time from
our record batch iterators. These iterators are implemented by controller
manager classes such as ReplicationControlManager, ClusterControlManager, etc.
Finally, this PR adds ControllerTestUtils#deepSortRecords and
ControllerTestUtils#assertBatchIteratorContains, which make it easier to write
unit tests. Since records are often constructed from unsorted data structures,
it is often useful to sort them before comparing them.
Reviewers: David Arthur <mumrah@gmail.com>
2021-04-07 01:18:06 +08:00
|
|
|
files="(QuorumController).java"/>
|
2023-11-07 06:21:51 +08:00
|
|
|
<suppress checks="(CyclomaticComplexity|NPathComplexity)"
|
KAFKA-16540: enforce min.insync.replicas config invariants for ELR (#17952)
If ELR is enabled, we need to set a cluster-level min.insync.replicas, and remove all broker-level overrides. The reason for this is that if brokers disagree about which partitions are under min ISR, it breaks the KIP-966 replication invariants. In order to enforce this, when the eligible.leader.replicas.version feature is turned on, we automatically remove all broker-level min.insync.replicas overrides, and create the required cluster-level override if needed. Similarly, if the cluster was created with eligible.leader.replicas.version enabled, we create a similar cluster-level record. In both cases, we don't allow setting overrides for individual brokers afterwards, or removing the cluster-level override.
Split ActivationRecordsGeneratorTest up into multiple test cases rather than having it be one giant test case.
Fix a bug in QuorumControllerTestEnv where we would replay records manually on objects, racing with the active controller thread. Instead, we should simply ensure that the initial bootstrap records contains what we want.
Reviewers: Colin P. McCabe <cmccabe@apache.org>
2025-01-09 05:42:25 +08:00
|
|
|
files="(ConfigurationControlManager|PartitionRegistration|PartitionChangeBuilder|ScramParser).java"/>
|
KAFKA-12276: Add the quorum controller code (#10070)
The quorum controller stores metadata in the KIP-500 metadata log, not in Apache
ZooKeeper. Each controller node is a voter in the metadata quorum. The leader of the
quorum is the active controller, which processes write requests. The followers are standby
controllers, which replay the operations written to the log. If the active controller goes away,
a standby controller can take its place.
Like the ZooKeeper-based controller, the quorum controller is based on an event queue
backed by a single-threaded executor. However, unlike the ZK-based controller, the quorum
controller can have multiple operations in flight-- it does not need to wait for one operation
to be finished before starting another. Therefore, calls into the QuorumController return
CompleteableFuture objects which are completed with either a result or an error when the
operation is done. The QuorumController will also time out operations that have been
sitting on the queue too long without being processed. In this case, the future is completed
with a TimeoutException.
The controller uses timeline data structures to store multiple "versions" of its in-memory
state simultaneously. "Read operations" read only committed state, which is slightly older
than the most up-to-date in-memory state. "Write operations" read and write the latest
in-memory state. However, we can not return a successful result for a write operation until
its state has been committed to the log. Therefore, if a client receives an RPC response, it
knows that the requested operation has been performed, and can not be undone by a
controller failover.
Reviewers: Jun Rao <junrao@gmail.com>, Ron Dagostino <rdagostino@confluent.io>
2021-02-20 10:03:23 +08:00
|
|
|
<suppress checks="CyclomaticComplexity"
|
MINOR: Rewrite the meta.properties handling code in Java and fix some issues #14628 (#14628)
meta.properties files are used by Kafka to identify log directories within the filesystem.
Previously, the code for handling them was in BrokerMetadataCheckpoint.scala. This PR rewrites the
code for handling them as Java and moves it to the apache.kafka.metadata.properties namespace. It
also gets rid of the separate types for v0 and v1 meta.properties objects. Having separate types
wasn't so bad back when we had a strict rule that zk clusters used v0 and kraft clusters used v1.
But ZK migration has blurred the lines. Now, a zk cluster may have either v0 or v1, if it is
migrating, and a kraft cluster may have either v0 or v1, at any time.
The new code distinguishes between an individual meta.properties file, which is represented by
MetaProperties, and a collection of meta.properties files, which is represented by
MetaPropertiesEnsemble. It is useful to have this distinction, because in JBOD mode, even if some
log directories are inaccessible, we can still use the ensemble to extract needed information like
the cluster ID. (Of course, even when not in JBOD mode, KRaft servers have always been able to
configure a metadata log directory separate from the main log directory.)
Since we recently added a unique directory.id to each meta.properties file, the previous convention
of passing a "canonical" MetaProperties object for the cluster around to various places in the code
needs to be revisited. After all, we can no longer assume all of the meta.properties files are the
same. This PR fixes these parts of the code. For example, it fixes the constructors of
ControllerApis and RaftManager to just take a cluster ID, rather than a MetaProperties object. It
fixes some other parts of the code, like the constructor of SharedServer, to take a
MetaPropertiesEnsemble object.
Another goal of this PR was to centralize meta.properties validation a bit more and make it
unit-testable. For this purpose, the PR adds MetaPropertiesEnsemble.verify, and a few other
verification methods. These enforce invariants like "the metadata directory must be readable," and
so on.
Reviewers: Igor Soarez <soarez@apple.com>, David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Proven Provenzano <pprovenzano@confluent.io>
2023-11-10 01:32:35 +08:00
|
|
|
files="(ClientQuotasImage|KafkaEventQueue|MetadataDelta|QuorumController|ReplicationControlManager|KRaftMigrationDriver|ClusterControlManager|MetaPropertiesEnsemble).java"/>
|
2021-02-05 06:46:57 +08:00
|
|
|
<suppress checks="NPathComplexity"
|
MINOR: Rewrite the meta.properties handling code in Java and fix some issues #14628 (#14628)
meta.properties files are used by Kafka to identify log directories within the filesystem.
Previously, the code for handling them was in BrokerMetadataCheckpoint.scala. This PR rewrites the
code for handling them as Java and moves it to the apache.kafka.metadata.properties namespace. It
also gets rid of the separate types for v0 and v1 meta.properties objects. Having separate types
wasn't so bad back when we had a strict rule that zk clusters used v0 and kraft clusters used v1.
But ZK migration has blurred the lines. Now, a zk cluster may have either v0 or v1, if it is
migrating, and a kraft cluster may have either v0 or v1, at any time.
The new code distinguishes between an individual meta.properties file, which is represented by
MetaProperties, and a collection of meta.properties files, which is represented by
MetaPropertiesEnsemble. It is useful to have this distinction, because in JBOD mode, even if some
log directories are inaccessible, we can still use the ensemble to extract needed information like
the cluster ID. (Of course, even when not in JBOD mode, KRaft servers have always been able to
configure a metadata log directory separate from the main log directory.)
Since we recently added a unique directory.id to each meta.properties file, the previous convention
of passing a "canonical" MetaProperties object for the cluster around to various places in the code
needs to be revisited. After all, we can no longer assume all of the meta.properties files are the
same. This PR fixes these parts of the code. For example, it fixes the constructors of
ControllerApis and RaftManager to just take a cluster ID, rather than a MetaProperties object. It
fixes some other parts of the code, like the constructor of SharedServer, to take a
MetaPropertiesEnsemble object.
Another goal of this PR was to centralize meta.properties validation a bit more and make it
unit-testable. For this purpose, the PR adds MetaPropertiesEnsemble.verify, and a few other
verification methods. These enforce invariants like "the metadata directory must be readable," and
so on.
Reviewers: Igor Soarez <soarez@apple.com>, David Arthur <mumrah@gmail.com>, Divij Vaidya <diviv@amazon.com>, Proven Provenzano <pprovenzano@confluent.io>
2023-11-10 01:32:35 +08:00
|
|
|
files="(ClientQuotasImage|KafkaEventQueue|ReplicationControlManager|FeatureControlManager|KRaftMigrationDriver|ScramControlManager|ClusterControlManager|MetadataDelta|MetaPropertiesEnsemble).java"/>
|
KAFKA-13646; Implement KIP-801: KRaft authorizer (#11649)
Currently, when using KRaft mode, users still have to have an Apache ZooKeeper instance if they want to use AclAuthorizer. We should have a built-in Authorizer for KRaft mode that does not depend on ZooKeeper. This PR introduces such an authorizer, called StandardAuthorizer. See KIP-801 for a full description of the new Authorizer design.
Authorizer.java: add aclCount API as described in KIP-801. StandardAuthorizer is currently the only authorizer that implements it, but eventually we may implement it for AclAuthorizer and others as well.
ControllerApis.scala: fix a bug where createPartitions was authorized using CREATE on the topic resource rather than ALTER on the topic resource as it should have been.
QuorumTestHarness: rename the controller endpoint to CONTROLLER for consistency (the brokers already called it that). This is relevant in AuthorizerIntegrationTest where we are examining endpoint names. Also add the controllerServers call.
TestUtils.scala: adapt the ACL functions to be usable from KRaft, by ensuring that they use the Authorizer from the current active controller.
BrokerMetadataPublisher.scala: add broker-side ACL application logic.
Controller.java: add ACL APIs. Also add a findAllTopicIds API in order to make junit tests that use KafkaServerTestHarness#getTopicNames and KafkaServerTestHarness#getTopicIds work smoothly.
AuthorizerIntegrationTest.scala: convert over testAuthorizationWithTopicExisting (more to come soon)
QuorumController.java: add logic for replaying ACL-based records. This means storing them in the new AclControlManager object, and integrating them into controller snapshots. It also means applying the changes in the Authorizer, if one is configured. In renounce, when reverting to a snapshot, also set newBytesSinceLastSnapshot to 0.
Reviewers: YeonCheol Jang <YeonCheolGit@users.noreply.github.com>, Jason Gustafson <jason@confluent.io>
2022-02-10 02:38:52 +08:00
|
|
|
<suppress checks="BooleanExpressionComplexity"
|
|
|
|
|
files="(MetadataImage).java"/>
|
2022-05-03 07:27:52 +08:00
|
|
|
<suppress checks="ImportControl"
|
|
|
|
|
files="ApiVersionsResponse.java"/>
|
2022-05-19 03:08:36 +08:00
|
|
|
<suppress checks="AvoidStarImport"
|
|
|
|
|
files="MetadataVersionTest.java"/>
|
2021-06-19 00:38:42 +08:00
|
|
|
|
2023-04-26 00:50:51 +08:00
|
|
|
<!-- group coordinator -->
|
|
|
|
|
<suppress checks="CyclomaticComplexity"
|
2025-02-17 23:13:21 +08:00
|
|
|
files="(ConsumerGroupMember|GroupMetadataManager|GeneralUniformAssignmentBuilder|GroupCoordinatorRecordSerde|GroupMetadataManagerTestContext).java"/>
|
2023-07-19 15:15:13 +08:00
|
|
|
<suppress checks="(NPathComplexity|MethodLength)"
|
2025-02-17 23:13:21 +08:00
|
|
|
files="(GroupMetadataManager|ConsumerGroupTest|ShareGroupTest|GroupMetadataManagerTest|GroupMetadataManagerTestContext|GeneralUniformAssignmentBuilder|GroupCoordinatorShard).java"/>
|
2023-07-19 15:15:13 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
2024-02-14 15:29:29 +08:00
|
|
|
files="(GroupMetadataManager|GroupMetadataManagerTest|GroupMetadataManagerTestContext|GroupCoordinatorService|GroupCoordinatorServiceTest).java"/>
|
2023-04-26 00:50:51 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
2023-10-12 14:45:13 +08:00
|
|
|
files="(ConsumerGroupMember|GroupMetadataManager|GroupCoordinatorConfig).java"/>
|
2023-05-23 16:42:13 +08:00
|
|
|
<suppress checks="ClassDataAbstractionCouplingCheck"
|
2025-02-26 23:33:26 +08:00
|
|
|
files="(RecordHelpersTest|GroupCoordinatorRecordHelpers|GroupMetadataManager|GroupCoordinatorService|GroupMetadataManagerTest|OffsetMetadataManagerTest|GroupCoordinatorServiceTest|GroupCoordinatorShardTest|GroupCoordinatorRecordSerde|StreamsGroupTest).java"/>
|
2023-07-19 15:15:13 +08:00
|
|
|
<suppress checks="JavaNCSS"
|
2024-05-29 14:21:30 +08:00
|
|
|
files="(GroupMetadataManager|GroupMetadataManagerTest).java"/>
|
2023-04-26 00:50:51 +08:00
|
|
|
|
2024-08-18 21:48:44 +08:00
|
|
|
<!-- coordinator-common -->
|
|
|
|
|
<suppress checks="NPathComplexity"
|
|
|
|
|
files="CoordinatorRuntime.java"/>
|
|
|
|
|
|
2025-01-20 15:47:14 +08:00
|
|
|
<!-- share coordinator and persister-->
|
2024-10-02 23:30:51 +08:00
|
|
|
<suppress checks="NPathComplexity"
|
|
|
|
|
files="ShareCoordinatorShard.java"/>
|
2025-01-20 15:47:14 +08:00
|
|
|
<suppress checks="ClassDataAbstractionCouplingCheck"
|
|
|
|
|
files="(ShareCoordinatorServiceTest|DefaultStatePersisterTest|PersisterStateManagerTest).java"/>
|
2024-10-02 23:30:51 +08:00
|
|
|
<suppress checks="CyclomaticComplexity"
|
|
|
|
|
files="ShareCoordinatorShard.java"/>
|
2025-02-05 22:51:19 +08:00
|
|
|
<suppress checks="ClassFanOutComplexity"
|
|
|
|
|
files="(PersisterStateManagerTest|DefaultStatePersisterTest).java"/>
|
2024-10-02 23:30:51 +08:00
|
|
|
|
2022-12-22 08:57:02 +08:00
|
|
|
<!-- storage -->
|
|
|
|
|
<suppress checks="CyclomaticComplexity"
|
2025-03-13 17:49:55 +08:00
|
|
|
files="(LogLoader|LogValidator|RemoteLogManagerConfig|RemoteLogManager|UnifiedLog).java"/>
|
2022-12-22 08:57:02 +08:00
|
|
|
<suppress checks="NPathComplexity"
|
2025-03-13 17:49:55 +08:00
|
|
|
files="(LocalLog|LogLoader|LogValidator|RemoteLogManager|RemoteIndexCache|UnifiedLog).java"/>
|
2022-12-22 08:57:02 +08:00
|
|
|
<suppress checks="ParameterNumber"
|
2025-03-13 17:49:55 +08:00
|
|
|
files="(LogAppendInfo|LogLoader|RemoteLogManagerConfig|UnifiedLog).java"/>
|
|
|
|
|
<suppress checks="(ClassDataAbstractionCoupling|ClassFanOutComplexity)"
|
2025-04-17 17:05:14 +08:00
|
|
|
files="(UnifiedLog|RemoteLogManager|RemoteLogManagerTest).java"/>
|
|
|
|
|
<suppress checks="MethodLength" files="(RemoteLogManager|RemoteLogManagerConfig).java"/>
|
|
|
|
|
<suppress checks="JavaNCSS" files="RemoteLogManagerTest.java"/>
|
2021-06-19 00:38:42 +08:00
|
|
|
|
2021-07-08 07:02:37 +08:00
|
|
|
<!-- benchmarks -->
|
2022-06-04 00:12:06 +08:00
|
|
|
<suppress checks="(ClassDataAbstractionCoupling|ClassFanOutComplexity)"
|
2021-07-08 07:02:37 +08:00
|
|
|
files="(ReplicaFetcherThreadBenchmark).java"/>
|
|
|
|
|
|
2024-08-14 02:11:33 +08:00
|
|
|
<!-- Add the new suppression rule for JaasTestUtils.java -->
|
2024-11-03 21:36:11 +08:00
|
|
|
<suppress checks="ImportControl" files="(JaasTestUtils).java" />
|
2017-03-01 06:55:46 +08:00
|
|
|
</suppressions>
|
