In the initial EOS example, a retry logic was implemented within the resetToLastCommittedPositions method. During refactoring, this logic was removed becasue a poison pill prevented the example from reaching the final phase of consuming from the output topic.
In this change, I suggest to add it back, but with a retry limit defined as MAX_RETRIES. Once this limit is reached, the problematic batch will be logged and skipped, allowing the processor to move on and process remaining records. If some records are skipped, the example will still hit the hard timeout (2 minutes), but after consuming all processed records.
Reviewers: Luke Chen <showuon@gmail.com>
We are not properly closing Closeable resources in the code base at multiple places especially when we have an exception. This code change fixes multiple of these leaks.
Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, Mickael Maison <mickael.maison@gmail.com>
This adds comments to the ConsumerRebalanceListener overrides, in order to briefly explain why we are overriding these methods, when they are called, and what you can or can't do. Especially onPartitionsLost can create some confusion given the default implementation.
Reviewers: Luke Chen <showuon@gmail.com>, David Jacot <djacot@confluent.io>
KAFKA-14752: Kafka examples improvements - consumer changes
This is extracted from the original PR for better review.
https://github.com/apache/kafka/pull/13492
Signed-off-by: Federico Valeri <fedevaleri@gmail.com>
Reviewers: Christo Lolov <christololov@gmail.com>, Luke Chen <showuon@gmail.com>
Remove "kafka.examples.Consumer" dependency on ShutdownableThread. "examples" module should be dependent only on public APIs but not to be dependent upon server common/internal components.
Reviewers: Luke Chen <showuon@gmail.com>
Modify Intellij to IntelliJ IDEA, and mention KRaft in example docs.
Reviewers: Divij Vaidya <divijvaidya13@gmail.com>, Kvicii <kvicii.yu@gmail.com>, dengziming <dengziming1993@gmail.com>
Class kafka.examples.SimpleConsumerDemo was removed. But the java-simple-consumer-demo.sh was not removed and README was not updated.
This commit removes java-simple-consumer-demo.sh and updates the demo instructions in the examples README.
Author: Jiamei Xie <jiamei.xie@arm.com>
Reviewers: Konstantine Karantasis <konstantine@confluent.io>
The current EOS example mixes fatal and non-fatal error handling. This patch fixes this problem and simplifies the example.
Reviewers: Jason Gustafson <jason@confluent.io>
This change mainly have 2 components:
1. extend the existing transactions_test.py to also try out new sendTxnOffsets(groupMetadata) API to make sure we are not introducing any regression or compatibility issue
a. We shrink the time window to 10 seconds for the txn timeout scheduler on broker so that we could trigger expiration earlier than later
2. create a completely new system test class called group_mode_transactions_test which is more complicated than the existing system test, as we are taking rebalance into consideration and using multiple partitions instead of one. For further breakdown:
a. The message count was done on partition level, instead of global as we need to visualize
the per partition order throughout the test. For this sake, we extend ConsoleConsumer to print out the data partition as well to help message copier interpret the per partition data.
b. The progress count includes the time for completing the pending txn offset expiration
c. More visibility and feature improvements on TransactionMessageCopier to better work under either standalone or group mode.
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
With the improvement of 447, we are now offering developers a better experience on writing their customized EOS apps with group subscription, instead of manual assignments. With the demo, user should be able to get started more quickly on writing their own EOS app, and understand the processing logic much better.
Reviewers: Guozhang Wang <wangguoz@gmail.com>
- Removed Scala consumers (`SimpleConsumer` and `ZooKeeperConsumerConnector`)
and their tests.
- Removed Scala request/response/message classes.
- Removed any mention of new consumer or new producer in the code
with the exception of MirrorMaker where the new.consumer option was
never deprecated so we have to keep it for now. The non-code
documentation has not been updated either, that will be done
separately.
- Removed a number of tools that only made sense in the context
of the Scala consumers (see upgrade notes).
- Updated some tools that worked with both Scala and Java consumers
so that they only support the latter (see upgrade notes).
- Removed `BaseConsumer` and related classes apart from `BaseRecord`
which is used in `MirrorMakerMessageHandler`. The latter is a pluggable
interface so effectively public API.
- Removed `ZkUtils` methods that were only used by the old consumers.
- Removed `ZkUtils.registerBroker` and `ZKCheckedEphemeral` since
the broker now uses the methods in `KafkaZkClient` and no-one else
should be using that method.
- Updated system tests so that they don't use the Scala consumers except
for multi-version tests.
- Updated LogDirFailureTest so that the consumer offsets topic would
continue to be available after all the failures. This was necessary for it
to work with the Java consumer.
- Some multi-version system tests had not been updated to include
recently released Kafka versions, fixed it.
- Updated findBugs and checkstyle configs not to refer to deleted
classes and packages.
Reviewers: Dong Lin <lindong28@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
Reading the configuration field names from ProducerConfig class and taking the key and value serializer names from class name directly instead of hardcoding.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
This patch had conflicts when merged, resolved by
Committer: Ismael Juma <ismael@juma.me.uk>
Closes#2328 from vahidhashemian/KAFKA-3264
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2303 from mjsax/licenseHeader
I think we should add a space here, otherwise the two words will join together.
And replace the host string with a constant, otherwise when I need to modify the host, I need to modify several files.
Author: 郑谦00117553 <00117553@zte.intra>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2132 from ZhengQian1/trunk
For enums and other constant strings, use locale independent case conversions to enable comparisons to work regardless of the default locale.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Manikumar Reddy, Ismael Juma, Guozhang Wang, Gwen Shapira
Closes#1220 from rajinisivaram/KAFKA-3548
These are minor, but no reason to make our example code look worse than it has to.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke
Closes#940 from ijuma/example-style-improvements
- Adds CheckStyle to core and examples modules
- Fixes any existing CheckStyle issues
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#703 from granthenke/checkstyle-core
Federico Valeri
Divij Vaidya
Said Boudjelda
Philip Nee
liuzhuang2017
jiameixie
Lee Dongjin
Boyang Chen
huxi
Viktor Somogyi
Ismael Juma
UVN
Vahid Hashemian
Matthias J. Sax
Qian Zheng
Rajini Sivaram
Grant Henke
Ashish Singh
Ismael Juma
Jay Kreps
Geoffrey Anderson
Joe Stein
Ailing Zhang
Jun Rao
Swapnil Ghike
Joe Stein
Derek Chen-Becker
Edward Jay Kreps
Neha Narkhede
Jun Rao