Create 3 new metadata versions:
- 3.8-IV0, for the upcoming 3.8 release.
- 3.9-IV0, to add support for KIP-1005.
- 3.9-IV1, as the new release vehicle for KIP-966.
Create ListOffsetRequest v9, which will be used in 3.9-IV0 to support KIP-1005. v9 is currently an unstable API version.
Reviewers: Jun Rao <junrao@gmail.com>, Justine Olshan <jolshan@confluent.io>
Abstracted code for 2 classes `ConsumerGroup` and `ConsumerGroupMember` to `ModernGroup` and `ModernGroupMember` respectively. The new abstract classes are created to share common functionality with `ShareGroup` and `ShareGroupMember` which are being introduced with KIP-932.
The patch is majorly code refactoring from existing classes to abstract classes. Also created a new package called `modern` where `MemberState` class is moved, in upcoming patches, I will move common classes for `Share` and `Consumer` Group in `modern` package itself.
Reviewers: Lianet Magrans <lianetmr@gmail.com>, Andrew Schofield <aschofield@confluent.io>, David Jacot <djacot@confluent.io>
About
Implemented release acquired records functionality in SharePartition. This functionality is used when a share session gets closed, hence all the acquired records should either move to AVAILABLE or ARCHIVED state. Implemented the following functions -
1. releaseAcquiredRecords - This function is executed when the acquisition lock timeout is reached. The function releases the acquired records.
2. releaseAcquiredRecordsForCompleteBatch - Function which releases acquired records maintained at a batch level.
3. releaseAcquiredRecordsForPerOffsetBatch - Function which releases acquired records maintained at an offset level.
Testing
Added unit tests to cover the new functionality added.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Apoorv Mittal <apoorvmittal10@gmail.com>
About
Implemented releaseAcquiredRecords functionality in SharePartitionManager which will act as a bridge between the call from KafkaApis to SharePartition for releasing the acquired records when a share session gets closed.
Testing
The added function has been tested with unit tests.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Apoorv Mittal <apoorvmittal10@gmail.com>
Remove > characters from the trogdor readme so that people who follow the guide don't accidentally write over their bin files.
Reviewers: Justine Olshan <jolshan@confluent.io>
All public interface changes should be briefly mentioned in the
upgrade guide.
Reviewers: Matthias J. Sax <matthias@confluent.io>, Anna Sophie Blee-Goldman <sophie@responsive.dev>, Nick Telford <nick.telford@gmail.com>
About
Implemented close() functionality of SharePartitionManager to avoid any chances of memory leak. The functionality will be utilized when the sharePartitionObject is closed, at the time when BrokerServer is killed.
Testing
Added unit tests to cover the new functionality added.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Apoorv Mittal <apoorvmittal10@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
Implement request handling for the updated versions of the KRaft RPCs (Fetch, FetchSnapshot, Vote,
BeginQuorumEpoch and EndQuorumEpoch). This doesn't add support for KRaft replicas to send the new
version of the KRaft RPCs. That will be implemented in KAFKA-16529.
All of the RPCs responses were extended to include the leader's endpoint for the listener of the
channel used in the request. EpochState was extended to include the leader's endpoint information
but only the FollowerState and LeaderState know the leader id and its endpoint(s).
For the Fetch request, the replica directory id was added. The leader now tracks the follower's log
end offset using both the replica id and replica directory id.
For the FetchSnapshot request, the replica directory id was added. This is not used by the KRaft
leader and it is there for consistency with Fetch and for help debugging.
For the Vote request, the replica key for both the voter (destination) and the candidate (source)
were added. The voter key is checked for consistency. The candidate key is persisted when the vote
is granted.
For the BeginQuorumEpoch request, all of the leader's endpoints are included. This is needed so
that the voters can return the leader's endpoint for all of the supported listeners.
For the EndQuorumEpoch request, all of the leader's endpoints are included. This is needed so that
the voters can return the leader's endpoint for all of the supported listeners. The successor list
has been extended to include the directory id. Receiving voters can use the entire replica key when
searching their position in the successor list.
Updated the existing test in KafkaRaftClientTest and KafkaRaftClientSnapshotTest to execute using
both the old version and new version of the RPCs.
Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>
This reverts commit e95e91a.
With the change to include the group.version flag, these tests fail due to trying to set the feature for the old version.
It is unclear if these tests originally worked as intended and given the upgrade is not expected for 3.8, we will just revert from 3.8.
Reviewers: David Jacot <djacot@confluent.io>
KIP-932 introduces FindCoordinator v6 for finding share coordinators. The initial implementation:
Checks that share coordinators are only requested with v6 or above.
Share coordinator requests are authorized as cluster actions (this is for inter-broker use only)
Responds with COORDINATOR_NOT_AVAILABLE because share coordinators are not yet available.
When the share coordinator code is delivered, the request handling will be gated by configurations which enable share groups and the share coordinator specifically. If these are not enabled, COORDINATOR_NOT_AVAILABLE is the response.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
Add field tieredEpoch to RemoteLogSegmentMetadata
Update relevant tests
Add two fields tieredEpoch and tieredState to TopicRecord.json
Reviewers: Luke Chen <showuon@gmail.com>, Christo Lolov <lolovc@amazon.com>
Implemented acquisition lock timeout functionality in SharePartition. Implemented the following functions -
1. releaseAcquisitionLockOnTimeout - This function is executed when the acquisition lock timeout is reached. The function releases the acquired records.
2. releaseAcquisitionLockOnTimeoutForCompleteBatch - Function which releases acquired records due to acquisition lock timeout maintained at a batch level.
3. releaseAcquisitionLockOnTimeoutForPerOffsetBatch - Function which releases acquired records due to acquisition lock timeout maintained at an offset level.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Apoorv Mittal <apoorvmittal10@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>,
Add the cause of TimeoutException for Producer send() errors.
Reviewers: Matthias J. Sax <matthias@confluent.io>, Lianet Magrans <lianetmr@gmail.com>, Artem Livshits <alivshits@confluent.io>, Justine Olshan <jolshan@confluent.io>
Removed an extra synchronized block encapsulating cache.remove(key). remove(ShareSessionKey key) function is already a synchronized method, hence we don't need it.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Apoorv Mittal <apoorvmittal10@gmail.com>
As part of KIP-584, brokers expose a range of supported versions for each feature. For example,
metadata.version might be supported from 1 to 21. (Note that feature level ranges are always
inclusive, so this would include both level 1 and 21.)
These supported ranges are supposed to be able to include 0. For example, it should be possible for
a broker to support a kraft.version between 0 and 1. However, in older software versions, there is
an assertion in org.apache.kafka.common.feature.SupportedVersionRange that prevents this. This
causes problems when the older software attempts to deserialize an ApiVersionsResponse containing
such a range.
In order to resolve this dilemma, this PR bumps the version of ApiVersionsRequest from 3 to 4.
Clients which send v4 promise to be able to handle ranges including 0. Clients which send v3 will
not be exposed to these ranges -- the feature will simply be omitted from the response. This work
is part of KIP-1022.
Share group consumers use the ShareFetch API to retrieve messages they've claimed (acquired records) from the leader brokers of share partitions.
The replica manager provides an API to retrieve messages directly from the underlying topic partition. The implementation of the fetch messages uses replica manager to fetch messages from specific offset known by share partition leader.
The requests are sent to a queue and processed asynchronously.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>
Integrated Persister Read API while initilazing Share Partition.
The share partition checks for any pre-existing data from persister and initializes the state accordingly.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Abhinav Dixit <adixit@confluent.io>
In the check-quorum calculation, the leader should not assume that it is part of the voter set. This may happen when the leader is removing itself from the voter set. This PR improves it by checking if leader is in the voter set or not, and decide how many follower fetches required. Also add tests.
Co-authored-by: Colin P. McCabe <cmccabe@apache.org>
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, José Armando García Sancio <jsancio@apache.org>
There is no Kafka Administrator action needed for an InvalidProducerEpochException, ERROR level is worrisome while such exception can happen for a variety of valid reason, by design
Proposing to lower the log level from ERROR to INFO
Reviewers: Justine Olshan <jolshan@confluent.io>
Colin P. McCabe
Matthias J. Sax
Shawn Hsu
gongxuanzhang
Apoorv Mittal
Omnia Ibrahim
Abhinav Dixit
Ken Huang
Ian McDonald
Lucas Brutschy
José Armando García Sancio
Chia Chuan Yu
Justine Olshan
vamossagar12
PoAn Yang
Andrew Schofield
Kuan-Po (Cooper) Tseng
dujian0068
Murali Basani
Alieh Saeedi
TingIāu "Ting" Kì
Pavel Pozdeev
Dmitry Werner
gongxuanzhang
Mickael Maison
Luke Chen
NICOLAS GUYOMAR
Wang Xiaoqing