This PR replaces the terms endorsing with acknowledging for voters which have recognised the current leader.
Reviewers: Jason Gustafson <jason@confluent.io>
This patch contains the following improvements:
- Separate inbound/outbound request flows so that we can open the door for concurrent inbound request handling
- Rewrite `KafkaNetworkChannel` to use `InterBrokerSendThread` which fixes a number of bugs/shortcomings
- Get rid of a lot of boilerplate conversions in `KafkaNetworkChannel`
- Improve validation of inbound responses in `KafkaRaftClient` by checking correlationId. This fixes a bug which could cause an out of order Fetch to be applied incorrectly.
Reviewers: David Arthur <mumrah@gmail.com>
ISR-related cleanup in ReplicaManager and Partition. Removes ISR change logic from ReplicaManager and adds a new ZkIsrManager class which adheres to a new AlterIsrManager trait. Unifies all of the ISR logic in Partition so we don't have separate code paths for ZK vs AlterIsr. Also removes PartitionStateStore
This change propagates topic ids to brokers in LeaderAndIsr Request. It also removes the topic name from the LeaderAndIsr Response, reorganizes the response to be sorted by topic, and includes the topic ID.
In addition, the topic ID is persisted to each replica in Log as well as in a file on disk. This file is read on startup and if the topic ID exists, it will be reloaded.
Reviewers: David Jacot <djacot@confluent.io>, dengziming <dengziming1993@gmail.com>, Nikhil Bhatia <rite2nikhil@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Includes:
- Bump the version of MetadataRequest and MetadataResponse, add topicId in MetadataResponse
- Alter describeTopic in AdminClientTopicService and ZookeeperTopicService
- TopicMetadata is cached in MetadataCache, so we need to add topicId to MetadataCache
- MetadataCache is updated by UpdateMetadataRequest, bump the version of UpdateMetadataReq and UpdateMetadataResp, add topicId in UpdateMetadataReq.
Reviewers: Justine Olshan <jolshan@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com>
Includes:
- New API to authorize by resource type
- Default implementation for the method that supports super users and ACLs
- Optimized implementation in AclAuthorizer that supports ACLs, super users and allow.everyone.if.no.acl.found
- Benchmarks and tests
- InitProducerIdRequest authorized for Cluster:IdempotentWrite or WRITE to any topic, ProduceRequest authorized only for topic even if idempotent
Reviewers: Lucas Bradstreet <lucas@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com>
This patch follows up https://github.com/apache/kafka/pull/9547. It refactors AbstractFetcherThread and its descendants to use `OffsetForLeaderEpochRequestData.OffsetForLeaderPartition` instead of `OffsetsForLeaderEpochRequest.PartitionData`. The patch relies on existing tests.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Jason Gustafson <jason@confluent.io>
I wasn't able to reproduce the failure locally, but it looks like there is a race condition with the sending of the records in the first producer. The test case assumes that these records have been completed before the call to `sendOffsetsToTransaction`, but they very well might not be. It is even possible for the writes from the second producer to arrive first which would then result in the test failure that we are seeing. The solution is to force the send with `flush()`.
Reviewers: Guozhang Wang <guozhang@apache.org>, Boyang Chen <boyang@confluent.io>
Add withLoggingEnabled and withLoggingDisabled for StreamJoined
to give StreamJoined the same flexibility as Materialized
Reviewers: Bruno Cadonna <bruno@confluent.io>, John Roesler <vvcephei@apache.org>
* The naming for `ListOffsets` was inconsistent, in some places it was `ListOffset` and in others
it was `ListOffsets`. Picked the latter since it was used in metrics and the protocol documentation
and made it consistent.
* Removed unused methods in ApiKeys.
* Deleted `CommonFields`.
* Added `lowestSupportedVersion` and `highestSupportedVersion` to `ApiMessageType`
* Removed tests in `MessageTest` that are no longer relevant.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
print out the feature flags received at DEBUG level, as well as the other version information.
Example log line:
[2020-11-03 17:47:17,076] DEBUG Node 0 has finalizedFeaturesEpoch: 42, finalizedFeatures: [FinalizedFeatureKey(name='feature_1', maxVersionLevel=2, minVersionLevel=1), FinalizedFeatureKey(name='feature_2', maxVersionLevel=4, minVersionLevel=3)], supportedFeatures: [SupportedFeatureKey(name='feature_1', minVersion=1, maxVersion=2), SupportedFeatureKey(name='feature_2', minVersion=3, maxVersion=4)] (org.apache.kafka.clients.NetworkClient:926)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
This PR introduces a new interface 'TransferableChannel' to replace GatheringByteChannel to avoid casting in write path. `TransferableChannel ` extends GatheringByteChannel with the minimal set of methods required by the Send interface. Supporting TLS and efficient zero copy transfers are the main reasons for the additional methods.
Co-authored-by: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Missed this in #9729. The substitution in `markCoordinatorUnknown` does not work because the argument is not provided as a parameter.
Reviewers: Ismael Juma <ismael@juma.me.uk>
When a consumer encounters an issue that triggers marking it to mark coordinator as unknown, the error message it prints does not give much context about the error that triggered it. This change includes the response error that triggered the transition or any other cause if not triggered by an error code in a response.
Reviewers: Jason Gustafson <jason@confluent.io>
The non-local return when `maybeHandleInvalidEnvelope` returns true causes the default error handler to execute after a response has already been sent. This patch rewrites the check as a local return.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Correct the implementation of the Cogroup
processor to implement KTableProcessorSupplier.
Change the cogrouped processor from PassThrough to
KTablePassThrough to allow for sending old values.
KTablePassThrough extends KTableProcessorSupplier instead of
ProcessorSupplier to implement sending old values and the view() method.
Reviewers: Walker Carlson <wcarlson@confluent.io>, John Roesler <vvcephei@apache.org>
This patch updates the request logger to output request and response payloads in JSON. Payloads are converted to JSON based on their auto-generated schema.
Reviewers: Lucas Bradstreet <lucas@confluent.io>, David Mao <dmao@confluent.io>, David Jacot <djacot@confluent.io>
Set it as a cluster action and update the handler in KafkaApis. We keep the `throttleTimeMs` field
since we intend to enable throttling in the future (especially relevant when we switch to the
built-in quorum mode).
Reviewers: David Arthur <mumrah@gmail.com>
StreamThreads can now be replaced in the streams uncaught exception handler
Reviewers: Bruno Cadonna <bruno@confluent.io>, John Roesler <vvcephei@apache.org>, Leah Thomas <lthomas@confluent.io>
Also updated the jmh readme to make it easier for new people to know
what's possible and best practices.
There were some changes in the generated benchmarking code that
required adjusting `spotbugs-exclude.xml` and for a `javac` warning
to be suppressed for the benchmarking module. I took the chance
to make the spotbugs exclusion mode maintainable via a regex
pattern.
Tested the commands on Linux and macOS with zsh.
JMH highlights:
* async-profiler integration. Can be used with -prof async,
pass -prof async:help to look for the accepted options.
* perf c2c [2] integration. Can be used with -prof perfc2c,
if available.
* JFR profiler integration. Can be used with -prof jfr, pass
-prof jfr:help to look for the accepted options.
Full details:
* 1.24: https://mail.openjdk.java.net/pipermail/jmh-dev/2020-August/002982.html
* 1.25: https://mail.openjdk.java.net/pipermail/jmh-dev/2020-August/002987.html
* 1.26: https://mail.openjdk.java.net/pipermail/jmh-dev/2020-October/003024.html
* 1.27: https://mail.openjdk.java.net/pipermail/jmh-dev/2020-December/003096.html
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Lucas Bradstreet <lucasbradstreet@gmail.com>
This patch adds logic to complete fetches immediately after resigning by returning the BROKER_NOT_AVAILABLE error. This ensures that the new election cannot be delayed by fetches which are stuck in purgatory.
Reviewers: Jason Gustafson <jason@confluent.io>
This PR adds support for IP entities to the `DescribeClientQuotas` and `AlterClientQuotas` APIs. This PR also adds support for describing/altering IP quotas via `kafka-configs` tooling.
Reviewers: Brian Byrne <bbyrne@confluent.io>, Anna Povzner <anna@confluent.io>, David Jacot <djacot@confluent.io>
As suggested, ensure InvalidProducerEpoch gets caught properly on stream side.
Reviewers: Guozhang Wang <wangguoz@gmail.com>, A. Sophie Blee-Goldman <sophie@confluent.io>, Matthias J. Sax <matthias@confluent.io>
FileChannel.close() does not guarantee modified buffer would be written on the file system. We are changing it with force() semantics to enforce file buffer and metadata written to filesystem (FileChannel.force(true) updates buffer and metadata).
Reviewers: Jun Rao <junrao@gmail.com>, Jason Gustafson <jason@confluent.io>
In Python 3, `filter` functions return iterators rather than `list` so it can traverse only once. Hence, the following loop will only see "empty" and then validation fails.
```python
src_messages = self.source.committed_messages() # return iterator
sink_messages = self.sink.flushed_messages()) # return iterator
for task in range(num_tasks):
# only first task can "see" the result. following tasks see empty result
src_seqnos = [msg['seqno'] for msg in src_messages if msg['task'] == task]
```
Reference: https://portingguide.readthedocs.io/en/latest/iterators.html#new-behavior-of-map-and-filter.
Reviewers: Jason Gustafson <jason@confluent.io>
Matthias J. Sax
Bill Bejeck
vamossagar12
Jason Gustafson
David Arthur
APaMio
Chia-Ping Tsai
Justine Olshan
dengziming
Scott Hendricks
Cheng Tan
Tom Bentley
Govinda Sakhare
David Jacot
leah
Ismael Juma
Julien Jean Paul Sirocchi
Lucas Bradstreet
Viktor Somogyi-Vass
Luke Chen
Anastasia Vela
panguncle
Kowshik Prakasam
Walker Carlson
David Mao
Boyang Chen
bertber
Brajesh Kumar