The name of the command should be kafka-metadata-quorum not
kafka-metatada-quorum.
Reviewers: Ron Dagostino <rdagostino@confluent.io>, Divij Vaidya <diviv@amazon.com>
`core` should only be used for legacy cli tools and tools that require
access to `core` classes instead of communicating via the kafka protocol
(typically by using the client classes).
Summary of changes:
1. Convert the command implementation and tests to Java and move it to
the `tools` module.
2. Introduce mechanism to capture stdout and stderr from tests.
3. Change `kafka-metadata-quorum.sh` to point to the new command class.
4. Adjusted the test classpath of the `tools` module so that it supports tests
that rely on the `@ClusterTests` annotation.
5. Improved error handling when an exception different from `TerseFailure` is
thrown.
6. Changed `ToolsUtils` to avoid usage of arrays in favor of `List`.
Reviewers: dengziming <dengziming1993@gmail.com>
Add `MetadataQuorumCommand` to describe quorum status, I'm trying to use arg4j style command format, currently, we only support one sub-command which is "describe" and we can specify 2 arguments which are --status and --replication.
```
# describe quorum status
kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --replication
ReplicaId LogEndOffset Lag LastFetchTimeMs LastCaughtUpTimeMs Status
0 10 0 -1 -1 Leader
1 10 0 -1 -1 Follower
2 10 0 -1 -1 Follower
kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
ClusterId: fMCL8kv1SWm87L_Md-I2hg
LeaderId: 3002
LeaderEpoch: 2
HighWatermark: 10
MaxFollowerLag: 0
MaxFollowerLagTimeMs: -1
CurrentVoters: [3000,3001,3002]
CurrentObservers: [0,1,2]
# specify AdminClient properties
kafka-metadata-quorum.sh --bootstrap-server localhost:9092 --command-config config.properties describe --status
```
Reviewers: Jason Gustafson <jason@confluent.io>
This includes TASTy Reader support for Scala 3.0.0. This makes it easier
for Kafka libraries to be used in Scala 3.0 projects
Release notes: https://github.com/scala/scala/releases/tag/v2.13.6
Reviewers: Ismael Juma <ismael@juma.me.uk>
The `kafka-preferred-replica-election` command was deprecated in 2.4. This path removes it for 3.0. `kafka-leader-election` can be used instead.
Reviewers: Ismael Juma <ismael@juma.me.uk>, Chia-Ping Tsai <chia7712@gmail.com>, Jason Gustafson <jason@confluent.io>
This patch implements KIP-635 which mainly adds support for querying offsets of multiple topics/partitions.
Reviewers: David Jacot <djacot@confluent.io>
Scala 2.13.4 restores default global `ExecutionContext` to 2.12 behavior
(to fix a perf regression in some use cases) and improves pattern matching
(especially exhaustiveness checking). Most of the changes are related
to the latter as I have enabled the newly introduced `-Xlint:strict-unsealed-patmat`.
More details on the code changes:
* Don't swallow exception in `ReassignPartitionsCommand.topicDescriptionFutureToState`.
* `RequestChannel.Response` should be `sealed`.
* Introduce sealed ClientQuotaManager.BaseUserEntity to avoid false positive
exhaustiveness warning.
* Handle a number of cases where pattern matches were not exhaustive:
either by marking them with @unchecked or by adding a catch-all clause.
* Workaround scalac bug related to exhaustiveness warnings in ZooKeeperClient
* Remove warning suppression annotations related to the optimizer that are no
longer needed in ConsumerGroupCommand and AclAuthorizer.
* Use `forKeyValue` in `AclAuthorizer.acls` as the scala bug preventing us from
using it seems to be fixed.
* Also update scalaCollectionCompat to 2.3.0, which includes minor improvements.
Full release notes:
* https://github.com/scala/scala/releases/tag/v2.13.4
* https://github.com/scala/scala-collection-compat/releases/tag/v2.3.0
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
I had to fix several compiler errors due to deprecation of auto application of `()`. A related
Xlint config (`-Xlint:nullary-override`) is no longer valid in 2.13, so we now only enable it
for 2.12. The compiler flagged two new inliner warnings that required suppression and
the semantics of `&` in `@nowarn` annotations changed, requiring a small change in
one of the warning suppressions.
I also removed the deprecation of a number of methods in `KafkaZkClient` as
they should not have been deprecated in the first place since `KafkaZkClient` is an
internal class and we still use these methods in the Controller and so on. This
became visible because the Scala compiler now respects Java's `@Deprecated`
annotation.
Finally, I included a few minor clean-ups (eg using `toBuffer` instead `toList`) when fixing
the compilation warnings.
Noteworthy bug fixes in Scala 2.13.3:
* Fix 2.13-only bug in Java collection converters that caused some operations to perform an extra pass
* Fix 2.13.2 performance regression in Vector: restore special cases for small operands in appendedAll and prependedAll
* Increase laziness of #:: for LazyList
* Fixes related to annotation parsing of @Deprecated from Java sources in mixed compilation
Full release notes:
https://github.com/scala/scala/releases/tag/v2.13.3
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
Scala 2.13.2 introduces support for suppressing warnings,
which makes it possible to enable fatal warnings. This is
useful enough from a development perspective to justify
this change.
In addition, Scala 2.13.2 also has a Vector implementation
with significant performance improvements and encoding
of String matches to switches.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
Highlights:
* Performance improvements in the ollections
library: algorithmic improvements and
changes to avoid unnecessary allocations.
* Performance improvements in the compiler.
* ASM was upgraded to 7.3.1, allowing the
optimizer to run on JDK 13+.
Full release notes: https://github.com/scala/scala/releases/tag/v2.12.11
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Some shell scripts don't have corresponding batch files in bin\windows.
For improving Windows platform support, This PR adds the following batch files:
- bin\windows\kafka-delete-records.bat
- bin\windows\kafka-log-dirs.bat
Reviewers: Jason Gustafson <jason@confluent.io>
Removed quotes from LogDir variable generation as there are additional quotes in Line 127.
This caused problems when those batch files are invoked from a path that contains space characters.
Scala 2.12 has better support for newer Java versions and includes additional
compiler warnings that are helpful during development. In addition, Scala 2.11
hasn't been supported by the Scala community for a long time, the soon to be
released Spark 2.4.0 will finally support Scala 2.12 (this was the main reason
preventing many from upgrading to Scala 2.12) and Scala 2.13 is at the RC stage.
It's time to start recommending the Scala 2.12 build as we prepare support for
Scala 2.13 and start thinking about removing support for Scala 2.11.
In the meantime, Jenkins will continue to build all supported Scala versions (including
Scala 2.11) so the PR and trunk jobs will fail if people accidentally use methods
introduced in Scala 2.12.
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>
It seems a tipo was made on a avariable for the log dir with commit 81e789ae3d. Then Windows tries to access by default the /log directory which of cause not exists.
klesta490 Is it ok so? Or was the tilde ~ intentional?
To test:
On Windows with the a fresh downloaded Kafka, adapt the properties:
* dataDir in config/zookeeper.properties with what you want, windows-compatible
* log.dirs in config/server.properties with what you want, windows-compatible
and executes:
bin\windows\zookeeper-server-start.bat config\zookeeper.properties
Author: u214578 <florian.hof@sbb.ch>
Reviewers: Vladimír Kleštinec <klestinec@gmail.com>, Vahid Hashemian <vahid.hashemian@gmail.com>
Closes#5837 from florianhof/feature/fix_windows_log_param
- Removed Scala consumers (`SimpleConsumer` and `ZooKeeperConsumerConnector`)
and their tests.
- Removed Scala request/response/message classes.
- Removed any mention of new consumer or new producer in the code
with the exception of MirrorMaker where the new.consumer option was
never deprecated so we have to keep it for now. The non-code
documentation has not been updated either, that will be done
separately.
- Removed a number of tools that only made sense in the context
of the Scala consumers (see upgrade notes).
- Updated some tools that worked with both Scala and Java consumers
so that they only support the latter (see upgrade notes).
- Removed `BaseConsumer` and related classes apart from `BaseRecord`
which is used in `MirrorMakerMessageHandler`. The latter is a pluggable
interface so effectively public API.
- Removed `ZkUtils` methods that were only used by the old consumers.
- Removed `ZkUtils.registerBroker` and `ZKCheckedEphemeral` since
the broker now uses the methods in `KafkaZkClient` and no-one else
should be using that method.
- Updated system tests so that they don't use the Scala consumers except
for multi-version tests.
- Updated LogDirFailureTest so that the consumer offsets topic would
continue to be available after all the failures. This was necessary for it
to work with the Java consumer.
- Some multi-version system tests had not been updated to include
recently released Kafka versions, fixed it.
- Updated findBugs and checkstyle configs not to refer to deleted
classes and packages.
Reviewers: Dong Lin <lindong28@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
- Add capability to create delegation token
- Add authentication based on delegation token.
- Add capability to renew/expire delegation tokens.
- Add units tests and integration tests
Author: Manikumar Reddy <manikumar.reddy@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#3616 from omkreddy/KAFKA-4541
Missed a few places in the previous commit.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#4200 from ijuma/scala-2.11.12-part2
When we install kafka on path with spaces, batch files were failing, this PR is trying to fix this issue.
Author: Vladimír Kleštinec <klestinec@gmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>
Closes#2649 from klesta490/trunk
This is important because Bits.reserveMemory calls System.gc() hoping to free native
memory in order to avoid throwing an OutOfMemoryException. This call is currently
a no-op due to -XX:+DisableExplicitGC.
It's worth mentioning that -XX:MaxDirectMemorySize can be used to increase the
amount of native memory available for allocation of direct byte buffers.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Apurva Mehta <apurva@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3371 from ijuma/kafka-5470-explicit-gc-invokes-concurrent
Added kafka-broker-api-versions.bat for Windows platform
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Vahid Hashemian, Ismael Juma, Guozhang Wang
Closes#2943 from amethystic/kafka-5143_offer_broker_api_versions_bat
handle existing classpath with spaces
list each lib dependency individually
Author: Jon Freedman <jon.freedman@zoho.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2433 from jonfreedman/trunk
Some of the recent changes to `kafka-run-class.sh` have not been applied to `kafka-run-class.bat`.
These recent changes include setting proper streams or connect classpaths. So any streams or connect use case that leverages `kafka-run-class.bat` would fail with an error like
```
Error: Could not find or load main class org.apache.kafka.streams.???
```
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2238 from vahidhashemian/minor/sync_up_kafka-run-class.bat
Without this fix the new consumer fails to run on a 32-bit Windows OS.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson, Guozhang Wang
Closes#2189 from vahidhashemian/KAFKA-4271
Will allow users to set one fewer environment variable if they need to
change scala version. Still, SCALA_BINARY_VERSION can be explicitly set.
Author: Konstantine Karantasis <konstantine@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2130 from kkonstantine/MINOR-Extract-SCALA_BINARY_VERSION-from-SCALA_VERSION
add file 'kafka-consumer-groups.bat' for Windows platform
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>
Closes#2037 from amethystic/kafka-4310_add_consumer_groups_script
On Windows, the following output is seen when starting Zookeeper and Kafka servers:
```
'#' is not recognized as an internal or external command,
operable program or batch file.
```
This pull request makes a minor correction to the Windows `kafka-run-class.bat` script to replace the use of `#` with `rem`.
Author: P. Thorpe <paul.thorpe@uk.ibm.com>
Reviewers: Vahid Hashemian, Gwen Shapira
Closes#1740 from p-thorpe/trunk
…able remote debugging to Kafka tools scripts
Author: Christian Posta <christian.posta@gmail.com>
Reviewers: Grant Henke, Gwen Shapira
Closes#955 from christian-posta/ceposta-enable-jvm-debugging-opts
A few minor fixes to reconcile differences between the windows and unix versions of the kafka/zookeeper start scripts that were causing cross-platform inconsistencies during deployment.
- Resolve differences in CLASSPATH setup between .bat and .sh start scripts
- .bat start scripts honor externally provided KAFKA_HEAP_OPTS and KAFKA_LOG4J_OPTS consistent with .sh
- .bat start scripts configure log4j similar to .sh
Author: Matt Fluet <matt.fluet@appian.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#908 from fluetm/scripts-patch
Chia-Ping Tsai
José Armando García Sancio
Federico Valeri
Ismael Juma
Matthew de Detrich
dengziming
Orsák Maroš
GauthamM-official
Jason Gustafson
Josep Prat
David Jacot
Dániel Urbán
ssugar
Ron Dagostino
Kengo Seki
Chris Steingen
Bert Roos
u214578
Manikumar Reddy O
hmcl
Vahid Hashemian
Vladimír Kleštinec
amethystic
Jon Freedman
Xi Hu
Konstantine Karantasis
P. Thorpe
Christian Posta
Matt Fluet