Commit Graph

243 Commits

Author SHA1 Message Date
Konstantine Karantasis dd62ef2eda
KAFKA-13748: Do not include file stream connectors in Connect's CLASSPATH and plugin.path by default (#11908)
With this change we stop including the non-production grade connectors that are meant to be used for demos and quick starts by default in the CLASSPATH and plugin.path of Connect deployments. The package of these connector will still be shipped with the Apache Kafka distribution and will be available for explicit inclusion. 

The changes have been tested through the system tests and the existing unit and integration tests. 

Reviewers: Mickael Maison <mickael.maison@gmail.com>, Randall Hauch <rhauch@gmail.com>
2022-03-30 13:15:42 -07:00
GauthamM-official eb6c5baf3e
MINOR: Adding kafka-storage.bat file (similar to kafka-storage.sh) for windows. (#11816)
Reviewers: Jun Rao <junrao@gmail.com>
2022-03-14 09:20:28 -07:00
Jason Gustafson fce771579c
KAFKA-12888; Add transaction tool from KIP-664 (#10814)
This patch adds the transaction tool specified in KIP-664: https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions. This includes all of the logic for describing transactional state and for aborting transactions. The only thing that is left out is the `--find-hanging` implementation, which will be left for a subsequent patch.

Reviewers: Boyang Chen <boyang@apache.org>, David Jacot <djacot@confluent.io>
2021-06-22 09:47:30 -07:00
Josep Prat a8bd649dbf
MINOR: Update Scala to 2.13.6 (#10711)
This includes TASTy Reader support for Scala 3.0.0. This makes it easier
for Kafka libraries to be used in Scala 3.0 projects

Release notes: https://github.com/scala/scala/releases/tag/v2.13.6

Reviewers: Ismael Juma <ismael@juma.me.uk>
2021-05-19 05:26:08 -07:00
Shay Elkin fc405d792d
Minor: Move trogdor out of tools and into its own gradle module (#10539)
Move Trogdor out of tools and into its own gradle module.  This allows us to minimize
the dependencies of the tools module.  We still keep Trogdor in the CLASSPATH
created by kafka-run-class.sh.

Reviewers: Colin P. McCabe <cmccabe@apache.org>
2021-04-15 11:37:15 -07:00
David Jacot 6d7a9012dc
KAFKA-8405; Remove deprecated `kafka-preferred-replica-election` command (#10443)
The `kafka-preferred-replica-election` command was deprecated in 2.4. This path removes it for 3.0. `kafka-leader-election` can be used instead.

Reviewers: Ismael Juma <ismael@juma.me.uk>, Chia-Ping Tsai <chia7712@gmail.com>, Jason Gustafson <jason@confluent.io>
2021-03-31 20:45:18 +02:00
Colin Patrick McCabe 5813446730
MINOR: fix kafka-metadata-shell.sh (#10226)
* Fix CLASSPATH issues in the startup script

* Fix overly verbose log messages during loading

* Update to use the new MetadataRecordSerde (this is needed now that we
  have a frame version)

* Fix initialization

Reviewers: Jason Gustafson <jason@confluent.io>
2021-02-26 17:13:20 -08:00
Ismael Juma dd34e40f3e
MINOR: Update Scala to 2.13.5 (#10169)
This includes a fix from Chia-Ping that removes tuple
allocations when `Map.forKeyValue` is used
(https://github.com/scala/scala/pull/9425) and support
for JDK 16.

Release notes:
https://github.com/scala/scala/releases/tag/v2.13.5

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2021-02-22 22:02:38 -08:00
Colin P. Mccabe 690f72dd69 KAFKA-12334: Add the KIP-500 metadata shell
The Kafka Metadata shell is a new command which allows users to
interactively examine the metadata stored in a KIP-500 cluster.
It can examine snapshot files that are specified via --snapshot.

The metadata tool works by replaying the log and storing the state into
in-memory nodes.  These nodes are presented in a fashion similar to
filesystem directories.

Reviewers: Jason Gustafson <jason@confluent.io>, David Arthur <mumrah@gmail.com>, Igor Soarez <soarez@apple.com>
2021-02-19 15:46:34 -08:00
Dániel Urbán 202ff6336f
KAFKA-5235: GetOffsetShell: Support for multiple topics and consumer configuration override (KIP-635) (#9430)
This patch implements KIP-635 which mainly adds support for querying offsets of multiple topics/partitions.

Reviewers: David Jacot <djacot@confluent.io>
2021-02-11 12:06:21 +01:00
Colin Patrick McCabe 1f508ea8c3
MINOR: Add StorageTool as specified in KIP-631 (#10043)
Add StorageTool as specified in KIP-631. It can format and describe storage directories.  Fix a bug in `ZkMetaProperties#toString`.

Reviewers: David Arthur <mumrah@gmail.com>
2021-02-08 12:42:40 -08:00
Colin Patrick McCabe 1d3e293c08
MINOR: Add ClusterTool as specified in KIP-631 (#10047)
Add ClusterTool as specified in KIP-631. It can report the current cluster ID, and also send the new RPC for removing broker registrations.

Reviewers: David Arthur <mumrah@gmail.com>
2021-02-08 12:07:39 -08:00
ssugar 70404baffa
MINOR: Fix windows startup scripts to use connect-log4j.properties instead of tools-log4j.properties (#10034)
Co-authored-by: Scott Sugar <ssugar@proserveit.com>

Reviewers: Konstantine Karantasis <k.karantasis@gmail.com>
2021-02-03 11:47:49 -08:00
Jesse Gorzinski 976e6ea0f7
MINOR: Tweak IBM i support in "stop" scripts (#9810)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2021-01-21 18:18:48 +00:00
Ismael Juma cbf8ad277a
MINOR: Upgrade to Scala 2.13.4 (#9643)
Scala 2.13.4 restores default global `ExecutionContext` to 2.12 behavior
(to fix a perf regression in some use cases) and improves pattern matching
(especially exhaustiveness checking). Most of the changes are related
to the latter as I have enabled the newly introduced `-Xlint:strict-unsealed-patmat`.

More details on the code changes:
* Don't swallow exception in `ReassignPartitionsCommand.topicDescriptionFutureToState`.
* `RequestChannel.Response` should be `sealed`.
* Introduce sealed ClientQuotaManager.BaseUserEntity to avoid false positive
exhaustiveness warning.
* Handle a number of cases where pattern matches were not exhaustive:
either by marking them with @unchecked or by adding a catch-all clause.
* Workaround scalac bug related to exhaustiveness warnings in ZooKeeperClient
* Remove warning suppression annotations related to the optimizer that are no
longer needed in ConsumerGroupCommand and AclAuthorizer.
* Use `forKeyValue` in `AclAuthorizer.acls` as the scala bug preventing us from
using it seems to be fixed.
* Also update scalaCollectionCompat to 2.3.0, which includes minor improvements.

Full release notes:
* https://github.com/scala/scala/releases/tag/v2.13.4
* https://github.com/scala/scala-collection-compat/releases/tag/v2.3.0

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2020-11-24 10:28:53 -08:00
Kowshik Prakasam d99fe49234
KAFKA-10599: Implement basic CLI tool for feature versioning system (#9409)
This PR implements a basic CLI tool for feature versioning system. The KIP-584 write up has been updated to suit this PR. The following is implemented in this PR:

--describe:
Describe supported and finalized features.
Usage: $> ./bin/kafka-features.sh --bootstrap-server host1:port1, host2:port2 --describe [--from-controller] [--command-config <path_to_java_properties_file>]
Optionally, use the --from-controller option to get features from the controller.
--upgrade-all:
Upgrades all features known to the tool to their highest max version levels.
Usage: $> ./bin/kafka-features.sh --bootstrap-server host1:port1, host2:port2 --upgrade-all [--dry-run] [--command-config <path_to_java_properties_file>]
Optionally, use the --dry-run CLI option to preview the feature updates without actually applying them.
--downgrade-all:
Downgrades existing finalized features to the highest max version levels known to this tool.
Usage: $> ./bin/kafka-features.sh --bootstrap-server host1:port1, host2:port2 --downgrade-all [--dry-run] [--command-config <path_to_java_properties_file>].
Optionally, use the --dry-run CLI option to preview the feature updates without actually applying them.

Reviewers: Boyang Chen <boyang@confluent.io>, Jun Rao <junrao@gmail.com>
2020-10-19 09:24:26 -07:00
Luke Chen 3669cde00b KAFKA-10390; Remove ignore case option when grep process info to be more specific
Remove ignore case option when grep process info to be more specific since our entry point is definitely `kafka.Kafka`.

Author: Luke Chen <showuon@gmail.com>

Reviewers: Gwen Shapira, Lucas Bradstreet

Closes #9179 from showuon/KAFKA-10390
2020-09-30 11:54:39 -07:00
Jesse Gorzinski 55c1ab62c9
KAFKA-10272: Add IBM i support to "stop" scripts (#9023)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2020-09-02 17:46:24 +02:00
Ismael Juma 7f90a58b69
MINOR: Update Scala to 2.13.3 (#8931)
I had to fix several compiler errors due to deprecation of auto application of `()`. A related
Xlint config (`-Xlint:nullary-override`) is no longer valid in 2.13, so we now only enable it
for 2.12. The compiler flagged two new inliner warnings that required suppression and
the semantics of `&` in `@nowarn` annotations changed, requiring a small change in
one of the warning suppressions.

I also removed the deprecation of a number of methods in `KafkaZkClient` as
they should not have been deprecated in the first place since `KafkaZkClient` is an
internal class and we still use these methods in the Controller and so on. This
became visible because the Scala compiler now respects Java's `@Deprecated`
annotation.

Finally, I included a few minor clean-ups (eg using `toBuffer` instead `toList`) when fixing
the compilation warnings.

Noteworthy bug fixes in Scala 2.13.3:

* Fix 2.13-only bug in Java collection converters that caused some operations to perform an extra pass
* Fix 2.13.2 performance regression in Vector: restore special cases for small operands in appendedAll and prependedAll
* Increase laziness of #:: for LazyList
* Fixes related to annotation parsing of @Deprecated from Java sources in mixed compilation

Full release notes:
https://github.com/scala/scala/releases/tag/v2.13.3

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
2020-06-26 10:19:04 -07:00
Xavier Léauté 58794d2f2e
MINOR: fix incorrect GC log size with JDK9+ (#8800)
Author: Xavier Léauté <xavier@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Chia-Ping Tsai <chia7712@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
2020-06-09 14:53:40 +05:30
jiameixie 2c1096cb0e
KAFKA-10018: Change command line tools from /bin/sh to /bin/bash (#8692)
Reviewers: Tom Bentley @tombentley, Boyang Chen <boyang@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>, Ismael Juma <ismael@confluent.io>, Colin P. McCabe <cmccabe@apache.org>, Matthias J. Sax <matthias@confluent.io>
2020-05-27 14:31:31 -07:00
sebwills c34f3d066e KAFKA-9034: Use double quotes around $JAVA_HOME evn variable
This allows kafka tools to work on Cygwin where $JAVA_HOME typically contains a space (e.g. "C:\Program Files\Java\jdkXXX")

Author: sebwills <sw@sebwills.com>

Reviewers: Ron Dagostino <rdagostino@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>

Closes #8481 from sebwills/patch-1
2020-04-30 13:51:23 +05:30
Ismael Juma dbe618c63c
KAFKA-9907: Switch default build to Scala 2.13 (#8537)
Scala 2.13.2 introduces support for suppressing warnings,
which makes it possible to enable fatal warnings. This is
useful enough from a development perspective to justify
this change.

In addition, Scala 2.13.2 also has a Vector implementation
with significant performance improvements and encoding
of String matches to switches.

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
2020-04-23 07:02:54 -07:00
Ismael Juma 93f082e093
MINOR: Update Scala to 2.12.11 (#8308)
Highlights:
* Performance improvements in the ollections
library: algorithmic improvements and
changes to avoid unnecessary allocations.
* Performance improvements in the compiler.
* ASM was upgraded to 7.3.1, allowing the
optimizer to run on JDK 13+.

Full release notes: https://github.com/scala/scala/releases/tag/v2.12.11

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2020-03-18 09:33:14 -07:00
Ego 0d83894fec
MINOR: Double quote `CLASSPATH` to prevent shell glob expansion. (#8191)
In the event that `CLASSPATH` does not have an ending ":", the shell
can expand the CLASSPATH globs to be space-separated list of paths/jars,
which is not how the JVM CLI accepts arguments to -cp switch. So
double quote the variable to prevent pattern expansion, and pass the
glob pattern directly to the JVM.

Reviewers: Ismael Juma <ismael@juma.me.uk>
2020-02-29 15:13:00 -08:00
Andras Katona bf16e35861 MINOR: set scala version automatically based on gradle.properties
Author: Andras Katona <akatona@cloudera.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Manikumar Reddy <manikumar.reddy@gmail.com>

Closes #7953 from akatona84/kafkarunclass-auto-scala-version
2020-02-22 22:14:51 +05:30
zshuo 6d4cfad013
MINOR: Update shell scripts to support z/OS system (#7913)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2020-02-20 14:19:50 +00:00
Ron Dagostino 342f13a838 KAFKA-8843: KIP-515: Zookeeper TLS support
Signed-off-by: Ron Dagostino <rdagostinoconfluent.io>

Author: Ron Dagostino <rdagostino@confluent.io>

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>

Closes #8003 from rondagostino/KAFKA-8843
2020-02-08 21:16:48 +05:30
Ismael Juma 03da52ecac
MINOR: Set MaxInlineLevel to 15 (#7811)
OpenJDK has changed the default from 9 to
15 in JDK 14:

https://bugs.openjdk.java.net/browse/JDK-8234863
https://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2019-December/036332.html

Scala applications tend to see the biggest
benefit (one of the Scala benchmarks improved
by 3x).

Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
2020-01-08 03:57:57 -08:00
John Roesler 717ce42a6d
KAFKA-9138: Add system test for relational joins (#7664)
Add a system test to verify the new foreign-key join introduced in KIP-213

Reviewers: Guozhang Wang <wangguoz@gmail.com>
2019-12-11 09:48:23 -08:00
Ryanne Dolan 4ac892ca78 KAFKA-7500: MirrorMaker 2.0 (KIP-382)
Implementation of [KIP-382 "MirrorMaker 2.0"](https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0)

Author: Ryanne Dolan <ryannedolan@gmail.com>
Author: Arun Mathew <arunmathew88@gmail.com>
Author: In Park <inpark@cloudera.com>
Author: Andre Price <obsoleted@users.noreply.github.com>
Author: christian.hagel@rio.cloud <christian.hagel@rio.cloud>

Reviewers: Eno Thereska <eno.thereska@gmail.com>, William Hammond <william.t.hammond@gmail.com>, Viktor Somogyi <viktorsomogyi@gmail.com>, Jakub Korzeniowski, Tim Carey-Smith, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>, Arun Mathew, Jeremy-l-ford, vpernin, Oleg Kasian <oleg.kasian@gmail.com>, Mickael Maison <mickael.maison@gmail.com>, Qihong Chen, Sriharsha Chintalapani <sriharsha@apache.org>, Jun Rao <junrao@gmail.com>, Randall Hauch <rhauch@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>

Closes #6295 from ryannedolan/KIP-382
2019-10-07 13:57:54 +05:30
Ismael Juma f718c5e1e1
MINOR: Update dependencies for Kafka 2.4 (part 2) (#7333)
Upgrade to Gradle 5.6.2 as a step towards Gradle 6.0 (necessary
for Java 13 support).

https://docs.gradle.org/5.5.1/release-notes.html
https://docs.gradle.org/5.6.2/release-notes.html

The other updates are mostly bug fixes:

* Scala 2.13.1: https://github.com/scala/scala/releases/tag/v2.13.1
* Scala 2.12.10: https://github.com/scala/scala/releases/tag/v2.12.10
* Jetty 9.4.20: https://www.eclipse.org/lists/jetty-announce/msg00133.html
* SLF4J 1.7.28: adds Automatic-Module-Name in MANIFEST.MF
* Bouncy castle 1.63: https://www.bouncycastle.org/releasenotes.html
* zstd 1.4.3: https://github.com/facebook/zstd/releases/tag/v1.4.3

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
2019-09-18 12:51:47 -07:00
Ismael Juma 66d81a0e50
MINOR: Update dependencies for Kafka 2.4 (#7126)
Scala 2.12.9 brings another 5% ~ 10% improvement in compiler performance,
improved compatibility with JDK 11/12/13, and experimental infrastructure for
build pipelining.

zstd update includes performance improvements, among which the
primary improvement is that decompression is ~7% faster.

Level | v1.4.0 | v1.4.1 | Delta
-- | -- | -- | --
1 | 1390 MB/s | 1453 MB/s | +4.5%
3 | 1208 MB/s | 1301 MB/s | +7.6%
5 | 1129 MB/s | 1233 MB/s | +9.2%
7 | 1224 MB/s | 1347 MB/s | +10.0%
16 | 1278 MB/s | 1430 MB/s | +11.8%

Jetty 9.4.19 includes a number of bug fixes:
https://github.com/eclipse/jetty.project/releases/tag/jetty-9.4.19.v20190610

Mockito 3.0.0 switched the Java requirement from 7 to 8.

Several updates to owaspDepCheckPlugin (4.0.2 -> 5.2.1).

The rest are patch updates.

Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
2019-08-08 06:01:22 -07:00
Matthias J. Sax 39bcc8447c
MINOR: Fix Streams metadata upgrade system test (#7118)
Reviewers: A. Sophie Blee-Goldmann <sophie@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2019-07-28 18:18:59 -07:00
Kengo Seki 58938a045f KAFKA-8513: Add kafka-streams-application-reset.bat for Windows platform (#6909)
Reviewers: Boyang Chen <boyang@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
2019-07-19 10:53:37 -07:00
José Armando García Sancio 121308cc7a KAFKA-8286; Generalized Leader Election Admin RPC (KIP-460) (#6686)
Implements KIP-460: https://cwiki.apache.org/confluence/display/KAFKA/KIP-460%3A+Admin+Leader+Election+RPC.

Reviewers: Jun Rao <junrao@gmail.com>, Jason Gustafson <jason@confluent.io>
2019-05-29 10:44:52 -07:00
Kengo Seki a29a005316 KAFKA-8349; Add Windows batch files corresponding to kafka-delete-records.sh and kafka-log-dirs.sh (#6709)
Some shell scripts don't have corresponding batch files in bin\windows.
For improving Windows platform support, This PR adds the following batch files:

- bin\windows\kafka-delete-records.bat
- bin\windows\kafka-log-dirs.bat

Reviewers: Jason Gustafson <jason@confluent.io>
2019-05-18 15:18:50 -07:00
Sönke Liebau 2bf153f6a7 KAFKA-8131; Move --version implementation into CommandLineUtils (#6481)
This patch refactors the implementation of the --version option and moves it into the default command options. This has the benefit of automatically including it in the usage output of the command line tools. Several tools had to be manually updated because they did not use the common options.

Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
2019-05-07 08:26:24 -07:00
Michael Gruben Trejo 8592baf784 MINOR: Comment spelling nit
`CYGINW` probably should be `CYGWIN`

*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*

*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*

Author: Michael Gruben Trejo <mgrubentrejo@linkedin.com>

Reviewers: Gwen Shapira

Closes #6523 from mgrubent/patch-1
2019-03-29 17:38:26 -07:00
Chris Steingen a8ed8e7495 Update kafka-run-class.bat (#6291)
Removed quotes from LogDir variable generation as there are additional quotes in Line 127.
This caused problems when those batch files are invoked from a path that contains space characters.
2019-02-21 12:55:14 -08:00
Bert Roos 203f58bb84 Added quotes around the class path (#4469)
The script otherwise fails when the classpath contains paths with spaces.
2019-01-25 20:37:31 -08:00
Ismael Juma 4154eb9f11
MINOR: Update dependencies for Kafka 2.2 (#6116)
- Scala 2.12.7 -> 2.12.8
- Gradle 5.0 -> 5.1
- Jetty 9.4.12 -> 9.4.14
- Rat 0.12 -> 0.13
- Patch bumps for easymock, jackson, powermock
- Patch bumps for gradle plugins: shadow, spotbugs, dependency-check, spotless

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
2019-01-10 01:14:30 -08:00
Ismael Juma b4722cc16f
KAFKA-7524: Recommend Scala 2.12 and use it for development (#5530)
Scala 2.12 has better support for newer Java versions and includes additional
compiler warnings that are helpful during development. In addition, Scala 2.11
hasn't been supported by the Scala community for a long time, the soon to be
released Spark 2.4.0 will finally support Scala 2.12 (this was the main reason
preventing many from upgrading to Scala 2.12) and Scala 2.13 is at the RC stage.
It's time to start recommending the Scala 2.12 build as we prepare support for
Scala 2.13 and start thinking about removing support for Scala 2.11.

In the meantime, Jenkins will continue to build all supported Scala versions (including
Scala 2.11) so the PR and trunk jobs will fail if people accidentally use methods
introduced in Scala 2.12.

Reviewers: Ewen Cheslack-Postava <me@ewencp.org>
2018-10-28 11:31:39 -07:00
u214578 4fd5fe2af3 MINOR: Remove unintentional tilde character from kafka-run-class.bat script
It seems a tipo was made on a avariable for the log dir with commit 81e789ae3d. Then Windows tries to access by default the /log directory which of cause not exists.
klesta490 Is it ok so? Or was the tilde ~ intentional?

To test:
On Windows with the a fresh downloaded Kafka, adapt the properties:
* dataDir in config/zookeeper.properties with what you want, windows-compatible
* log.dirs in config/server.properties with what you want, windows-compatible
and executes:
bin\windows\zookeeper-server-start.bat config\zookeeper.properties

Author: u214578 <florian.hof@sbb.ch>

Reviewers: Vladimír Kleštinec <klestinec@gmail.com>, Vahid Hashemian <vahid.hashemian@gmail.com>

Closes #5837 from florianhof/feature/fix_windows_log_param
2018-10-28 11:59:08 +05:30
Bill Bejeck b74e7e407c MINOR: Enable ignored upgrade system tests - trunk (#5605)
Removed ignore annotations from the upgrade tests. This PR includes the following changes for updating the upgrade tests:

* Uploaded new versions 0.10.2.2, 0.11.0.3, 1.0.2, 1.1.1, and 2.0.0 (in the associated scala versions) to kafka-packages
* Update versions in version.py, Dockerfile, base.sh
* Added new versions to StreamsUpgradeTest.test_upgrade_downgrade_brokers including version 2.0.0
* Added new versions StreamsUpgradeTest.test_simple_upgrade_downgrade test excluding version 2.0.0
* Version 2.0.0 is excluded from the streams upgrade/downgrade test as StreamsConfig needs an update for the new version, requiring a KIP. Once the community votes the KIP in, a minor follow-up PR can be pushed to add the 2.0.0 version to the upgrade test.
* Fixed minor bug in kafka-run-class.sh for classpath in upgrade/downgrade tests across versions.
* Follow on PRs for 0.10.2x, 0.11.0x, 1.0.x, 1.1.x, and 2.0.x will be pushed soon with the same updates required for the specific version.

Reviewers: Eno Thereska <eno.thereska@gmail.com>, John Roesler <vvcephei@users.noreply.github.com>, Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <matthias@confluent.io>
2018-09-13 13:46:47 -07:00
Ismael Juma cc4dce94af
KAFKA-2983: Remove Scala consumers and related code (#5230)
- Removed Scala consumers (`SimpleConsumer` and `ZooKeeperConsumerConnector`)
and their tests.
- Removed Scala request/response/message classes.
- Removed any mention of new consumer or new producer in the code
with the exception of MirrorMaker where the new.consumer option was
never deprecated so we have to keep it for now. The non-code
documentation has not been updated either, that will be done
separately.
- Removed a number of tools that only made sense in the context
of the Scala consumers (see upgrade notes).
- Updated some tools that worked with both Scala and Java consumers
so that they only support the latter (see upgrade notes).
- Removed `BaseConsumer` and related classes apart from `BaseRecord`
which is used in `MirrorMakerMessageHandler`. The latter is a pluggable
interface so effectively public API.
- Removed `ZkUtils` methods that were only used by the old consumers.
- Removed `ZkUtils.registerBroker` and `ZKCheckedEphemeral` since
the broker now uses the methods in `KafkaZkClient` and no-one else
should be using that method.
- Updated system tests so that they don't use the Scala consumers except
for multi-version tests.
- Updated LogDirFailureTest so that the consumer offsets topic would
continue to be available after all the failures. This was necessary for it
to work with the Java consumer.
- Some multi-version system tests had not been updated to include
recently released Kafka versions, fixed it.
- Updated findBugs and checkstyle configs not to refer to deleted
classes and packages.

Reviewers: Dong Lin <lindong28@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
2018-06-19 07:32:54 -07:00
Magesh Nandakumar 642a09168c KAFKA-6991: Fix ServiceLoader issue with PluginClassLoader (KIP-285)
Fix ServiceLoader issue with PluginClassLoader and add basic-auth-extension packaging & classpath

*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*

*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*

Author: Magesh Nandakumar <magesh.n.kumar@gmail.com>

Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #5135 from mageshn/KAFKA-6991
2018-06-06 21:09:16 -07:00
Magesh Nandakumar e5492d7726 KAFKA-6997: Exclude test-sources.jar when $INCLUDE_TEST_JARS is FALSE
Exclude test-sources.jar when $INCLUDE_TEST_JARS is FALSE

Author: Magesh Nandakumar <magesh.n.kumar@gmail.com>

Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Robert Yokota <rayokota@gmail.com>, Arjun Satish <arjun@confluent.io>, Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #5139 from mageshn/KAFKA-6997
2018-06-05 16:27:22 -07:00
Manikumar Reddy O 202ab22b98 KAFKA-5523: Remove ReplayLogProducer tool (#5092)
Reviewers: Ismael Juma <ismael@juma.me.uk>
2018-06-03 01:52:11 -07:00
Sasaki Toru 440445e7c5 KAFKA-2061; Offer a --version flag to print the kafka version [KIP-278] (#639)
Reviewers: Andy Lindeman, Jeremy Donahue, Jason Gustafson <jason@confluent.io>
2018-05-25 00:47:54 -07:00
Ismael Juma f2514e67e0 MINOR: Fix kafka-run-class for Java 10 (#4895)
We need to match to the end of the line to make it work with Java 10 as explained in the expanded comment.

Tested manually for all supported versions:

```shell
echo $(.../jdk1.8.0_152.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
1

echo $(.../jdk-9.0.4.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
9

echo $(.../jdk-10.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
10

echo $(.../jdk-10.0.1.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
10
```
2018-04-19 15:34:13 -07:00
Matthias J. Sax 0c0d8363e5
KAFKA-6054: Fix upgrade path from Kafka Streams v0.10.0 (#4779)
Reviewers: Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>, John Roesler <john@confluent.io>, Damian Guy <damian@confluent.io>
2018-04-06 17:00:52 -07:00
hmcl aa14ec08b7 MINOR: Remove kafka-consumer-offset-checker.bat for KAFKA-3356 (#4703) 2018-03-13 16:41:54 -07:00
Jimin Hsieh 596b604862 MINOR: Fix wrong message in `bin/kafka-run-class.sh` (#4682)
To build jar you need to specify `scalaVersion` instead of `scala_version`.

Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
2018-03-12 14:34:59 -07:00
Jason Gustafson 23c1c52c85
KAFKA-6615; Add scripts for DumpLogSegments (#4653)
Reviewers: Ismael Juma <ismael@juma.me.uk>
2018-03-06 15:37:50 -08:00
Manikumar Reddy 27a8d0f9e7 KAFKA-4541; Support for delegation token mechanism
- Add capability to create delegation token
- Add authentication based on delegation token.
- Add capability to renew/expire delegation tokens.
- Add units tests and integration tests

Author: Manikumar Reddy <manikumar.reddy@gmail.com>

Reviewers: Jun Rao <junrao@gmail.com>

Closes #3616 from omkreddy/KAFKA-4541
2018-01-16 09:50:31 -08:00
Chris Egerton abf4c54701 KAFKA-4247; Prevent CLASSPATH from beginning with a single colon
Different fix for problem addressed by https://github.com/apache/kafka/pull/1953. Should prevent the CLASSPATH environment variable from being prefixed by a single colon before the JVM is invoked in the run-class script, which will then prevent the current working directory from being unintentionally included in the classpath when using the Reflections library.

If the current working directory should still be included in the classpath, it just needs to be explicitly specified either with its fully-qualified pathname or as a single dot (".").

Author: Chris Egerton <chrise@confluent.io>

Reviewers: Randall Hauch <rhauch@gmail.com>, Jun Rao <junrao@gmail.com>

Closes #4406 from C0urante/fix-colon-prefixed-classpath
2018-01-09 14:49:18 -08:00
Prasanna Gautam e5daa40e31 KAFKA-5473; handle ZK session expiration properly when a new session can't be established
(WIP: this commit isn't ready to be reviewed yet. I was checking the travis-ci build with the configuration changes in my account and opened the PR prematurely against trunk. I will make it consistent with Contribution guidelines once it's well tested.)

https://issues.apache.org/jira/browse/KAFKA-5473

Design:
`zookeeper.connection.retry.timeout.ms` => this determines how long to wait before triggering the shutdown. The default is 60000ms.

Currently the implementation only handles the `handleSessionEstablishmentError` by waiting for the sessionTimeout.

Author: Prasanna Gautam <prasannagautam@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>

Closes #3990 from prasincs/KAFKA-5473
2017-12-15 14:48:30 -08:00
Arjun Satish cfa6a78c7c KAFKA-4115: Increasing the heap settings for Connect scripts
Signed-off-by: Arjun Satish <arjunconfluent.io>

Author: Arjun Satish <arjun@confluent.io>

Reviewers: Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4213 from wicknicks/KAFKA-4115
2017-11-21 16:07:06 -08:00
Ismael Juma 3e86161b84 MINOR: Update to Scala 2.11.12 (part 2)
Missed a few places in the previous commit.

Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>

Closes #4200 from ijuma/scala-2.11.12-part2
2017-11-09 18:24:48 +00:00
Vahid Hashemian bfa82fc0b6 KAFKA-6055; Fix typo in KAFKA_JVM_PERFORMANCE_OPTS for Windows
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #4062 from vahidhashemian/KAFKA-6055
2017-10-12 10:09:16 +01:00
Mickael Maison a0f533266a KAFKA-3356: Remove ConsumerOffsetChecker
Author: Mickael Maison <mickael.maison@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>

Closes #3036 from mimaison/KAFKA-3356
2017-09-24 10:08:52 +08:00
Vladimír Kleštinec 81e789ae3d KAFKA-4860; Allow spaces in paths on windows
When we install kafka on path with spaces, batch files were failing, this PR is trying to fix this issue.

Author: Vladimír Kleštinec <klestinec@gmail.com>

Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>

Closes #2649 from klesta490/trunk
2017-09-05 16:27:08 -07:00
Dong Lin adefc8ea07 KAFKA-5694; Add AlterReplicaDirRequest and DescribeReplicaDirRequest (KIP-113 part-1)
Author: Dong Lin <lindong28@gmail.com>

Reviewers: Jun Rao <junrao@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Colin P. Mccabe <cmccabe@confluent.io>

Closes #3621 from lindong28/KAFKA-5694
2017-09-02 23:20:13 -07:00
Colin P. Mccabe 0772fde562 KAFKA-5776; Add the Trogdor fault injection daemon
Author: Colin P. Mccabe <cmccabe@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>

Closes #3699 from cmccabe/trogdor-review
2017-08-25 12:29:40 -07:00
Xavier Léauté 75b0f30c44 KAFKA-5077; fix GC logging arguments for Java 9
Author: Xavier Léauté <xavier@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #2863 from xvrl/fix-jdk9-gc-logs
2017-08-10 02:06:53 +01:00
Evgeny Veretennikov d2a2b80cc6 KAFKA-5507; Check if classpath is empty in kafka-run-class.sh
Author: Evgeny Veretennikov <evg.veretennikov@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #3421 from evis/kafka-run-class-check-classpath
2017-08-09 23:58:25 +01:00
Ismael Juma 69d3d4aad6 KAFKA-5470; Replace -XX:+DisableExplicitGC with -XX:+ExplicitGCInvokesConcurrent in kafka-run-class
This is important because Bits.reserveMemory calls System.gc() hoping to free native
memory in order to avoid throwing an OutOfMemoryException. This call is currently
a no-op due to -XX:+DisableExplicitGC.

It's worth mentioning that -XX:MaxDirectMemorySize can be used to increase the
amount of native memory available for allocation of direct byte buffers.

Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Apurva Mehta <apurva@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>

Closes #3371 from ijuma/kafka-5470-explicit-gc-invokes-concurrent
2017-08-09 23:38:57 +01:00
Ismael Juma bcf447e93e KAFKA-4422; Drop support for Scala 2.10 (KIP-119)
Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Ewen Cheslack-Postava <me@ewencp.org>

Closes #2956 from ijuma/kafka-4422-drop-support-scala-2.10
2017-05-11 08:08:11 +01:00
amethystic 29994dd10d KAFKA-5143: add kafka-broker-api-version.bat for Windows platform
Added kafka-broker-api-versions.bat for Windows platform

Author: amethystic <huxi_2b@hotmail.com>

Reviewers: Vahid Hashemian, Ismael Juma, Guozhang Wang

Closes #2943 from amethystic/kafka-5143_offer_broker_api_versions_bat
2017-05-05 16:53:32 -07:00
Ryan P 8d74920163 KAFKA-3754; Add GC log retention policy to limit size of log
Add a default log retention policy to keep GC logs from growing too large

Author: Ryan P <ryan.n.pridgeon@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1431 from rnpridgeon/KAFKA-3754
2017-05-03 10:51:32 +01:00
Dong Lin 8b05ad406d KAFKA-4586; Add purgeDataBefore() API (KIP-107)
Author: Dong Lin <lindong28@gmail.com>

Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com>

Closes #2476 from lindong28/KAFKA-4586
2017-03-28 09:59:44 -07:00
Jon Freedman dc4d1bb81b MINOR: Update kafka-run-class.bat to handle spaces in classpath
handle existing classpath with spaces
list each lib dependency individually

Author: Jon Freedman <jon.freedman@zoho.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #2433 from jonfreedman/trunk
2017-01-26 14:17:09 -08:00
Xi Hu e9426291aa KAFKA-4684: add kafka-configs.bat for Windows boxes
Add kafka-configs.bat script for Windows.

Author: huxi <huxi@zhenrongbao.com>

Reviewers: Guozhang Wang, Vahid Hashemian

Closes #2419 from amethystic/kafka4684_offer_kafkaconfigs_script
2017-01-25 17:53:51 -08:00
Colin P. Mccabe 4a6f2c6cc0 KAFKA-4457; Add BrokerApiVersionsCommand
Author: Colin P. Mccabe <cmccabe@confluent.io>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Apurva Mehta <apurva.1618@gmail.com>, Ismael Juma <ismael@juma.me.uk>

Closes #2184 from cmccabe/KAFKA-4457
2017-01-18 23:44:11 +00:00
Shikhar Bhushan 2f90488323 KAFKA-3209: KIP-66: single message transforms
Besides API and runtime changes, this PR also includes 2 data transformations (`InsertField`, `HoistToStruct`) and 1 routing transformation (`TimestampRouter`).

There is some gnarliness in `ConnectorConfig` / `ConfigDef` around creating, parsing and validating a dynamic `ConfigDef`.

Author: Shikhar Bhushan <shikhar@confluent.io>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #2299 from shikhar/smt-2017
2017-01-12 16:14:53 -08:00
Vahid Hashemian e799546377 MINOR: Sync up 'kafka-run-class.bat' with 'kafka-run-class.sh'
Some of the recent changes to `kafka-run-class.sh` have not been applied to `kafka-run-class.bat`.
These recent changes include setting proper streams or connect classpaths. So any streams or connect use case that leverages `kafka-run-class.bat` would fail with an error like
```
Error: Could not find or load main class org.apache.kafka.streams.???
```

Author: Vahid Hashemian <vahidhashemian@us.ibm.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #2238 from vahidhashemian/minor/sync_up_kafka-run-class.bat
2016-12-29 12:03:24 -05:00
Vahid Hashemian 055ca9b7a2 KAFKA-4272; Add missing 'connect' Windows batch scripts
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>

Reviewers: Jason Gustafson <jason@confluent.io>

Closes #2146 from vahidhashemian/KAFKA-4272
2016-12-09 11:40:50 -08:00
Vahid Hashemian 8d188c9110 KAFKA-4271: Fix the server start script for Windows 32-bit OS
Without this fix the new consumer fails to run on a 32-bit Windows OS.

Author: Vahid Hashemian <vahidhashemian@us.ibm.com>

Reviewers: Jason Gustafson, Guozhang Wang

Closes #2189 from vahidhashemian/KAFKA-4271
2016-11-30 12:19:23 -08:00
Konstantine Karantasis aad9384066 MINOR: Extract SCALA_BINARY_VERSION from SCALA_VERSION
Will allow users to set one fewer environment variable if they need to
change scala version. Still, SCALA_BINARY_VERSION can be explicitly set.

Author: Konstantine Karantasis <konstantine@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #2130 from kkonstantine/MINOR-Extract-SCALA_BINARY_VERSION-from-SCALA_VERSION
2016-11-17 02:15:14 +00:00
amethystic 6f842c1e9e KAFKA-4310; Add missing Windows script `kafka-consumer-groups.bat`
add file 'kafka-consumer-groups.bat' for Windows platform

Author: amethystic <huxi_2b@hotmail.com>

Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>

Closes #2037 from amethystic/kafka-4310_add_consumer_groups_script
2016-10-18 11:08:10 -07:00
1ambda dca63fc0a5 KAFKA-3742: (FIX) Can't run bin/connect-*.sh with -daemon flag
## Problem

Current connect scripts (`connect-distributed.sh`, `connect-standalone.sh`) do not support `-daemon` flag even if users specify the flag
since `kafka-run-class.sh` requires that the`-daemon` flag should precede other arguments (e.g. class name)

## Solution

Do the same thing like in `kafka-server-start.sh`

- Parse a command
- Add `-daemon` to `$EXTRA_ARGS` if exists

Author: 1ambda <1amb4a@gmail.com>

Reviewers: Gwen Shapira

Closes #1717 from 1ambda/KAFKA-3742-connect-running-as-daemon
2016-08-25 20:23:05 -07:00
P. Thorpe e3ede84344 MINOR: Remove # from .bat start script
On Windows, the following output is seen when starting Zookeeper and Kafka servers:
```
'#' is not recognized as an internal or external command,
operable program or batch file.
```

This pull request makes a minor correction to the Windows `kafka-run-class.bat` script to replace the use of `#` with `rem`.

Author: P. Thorpe <paul.thorpe@uk.ibm.com>

Reviewers: Vahid Hashemian, Gwen Shapira

Closes #1740 from p-thorpe/trunk
2016-08-18 20:50:20 -07:00
Grant Henke 1572274583 KAFKA-3934: Start scripts enable GC by default with no way to disable
…o disable

Author: Grant Henke <granthenke@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1631 from granthenke/garbage-flag
2016-08-09 09:16:17 -07:00
deragon cb7af0b7fb MINOR: Make kafka-run-class.sh work under Cygwin
kafka-run-class.sh now runs under Cygwin; paths and classpath are set up properly.

**WARNING:**  The script was not tested on a Linux machine, only under Cygwin.  Prior to merge it into trunk, if accepted, please run a quick test to ensure nothing broke.  From my own code review, there should not be any problem, but we can never be too sure.

I do not have the environment to test it under Linux at this moment.

Author: deragon <32nx9812masakjds>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1487 from deragon/trunk
2016-08-09 08:33:53 -07:00
Matthias J. Sax f7976d2fc1 KAFKA-4008: Module "tools" should not be dependent on "core"
moved streams application reset tool from tools to core

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1685 from mjsax/moveResetTool

(cherry picked from commit f2405a73ea)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
2016-08-01 20:12:38 -07:00
Matthias J. Sax 8deedcacb6 KAFKA-3185: Allow users to cleanup internal Kafka Streams data
- added Kafka Stream Application Reset Tool

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Guozhang Wang, Michael G. Noll

Closes #1636 from mjsax/kafka-3185
2016-07-27 14:11:40 -07:00
Liquan Pei fb421dbcfe KAFKA-3692; Add quotes to variables in kafka-run-class.sh
Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Geoff Anderson <geoff@confluent.io>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>

Closes #1364 from Ishiihara/add-quote-classpath
2016-05-13 12:10:09 +01:00
Dana Powers 1e4dd66b19 HOTFIX: Fix main classpath libs glob for release (fixup KAFKA-3615 regression)
bin/kafka-run-class.sh does not correctly setup the CLASSPATH in release rc2.

Author: Dana Powers <dana.powers@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1302 from dpkp/KAFKA-3615-fix
2016-04-30 18:53:44 -07:00
Liquan Pei eb50d2f6ca KAFKA-3615: Exclude test jars in kafka-run-class.sh
granders hachikuji Can you take a look when you have time? Appreciate your time to review.

Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Grant Henke <granthenke@gmail.com>, Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1263 from Ishiihara/classpath-no-test-jar
2016-04-29 10:29:00 -07:00
Liquan Pei 316389d6ad KAFKA-3611: Remove warnings when using reflections
ewencp granders Can you take a look? Thanks!

Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1259 from Ishiihara/fix-warning
2016-04-28 11:59:02 -07:00
Matt McClure 732b111f45 KAFKA-3384: Conform to POSIX kill usage
I believe this addresses KAFKA-3384.

The POSIX kill manpage is at http://pubs.opengroup.org/onlinepubs/9699919799/utilities/kill.html

Author: Matt McClure <mlm@aya.yale.edu>

Reviewers: Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1148 from matthewlmcclure/KAFKA-3384
2016-04-04 22:07:20 -07:00
Yasuhiro Matsuda de06244338 MINOR: remove streams-smoke-test.sh
guozhangwang

Author: Yasuhiro Matsuda <yasuhiro@confluent.io>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #1125 from ymatsuda/remove_smoketest_shell_script
2016-03-23 14:57:03 -07:00
Sasaki Toru f7887bd6f9 KAFKA-3281: Improve message of *-server-stop.sh when process is not running
Stop scritps such as kafka-server-stop.sh log messages of kill command's error when processes aren't running.
This PR changes this message to "No kafka server to stop".

Author: Sasaki Toru <sasakitoa@nttdata.co.jp>

Reviewers: Gwen Shapira

Closes #971 from sasakitoa/stop_scripts_says_not_good_message
2016-03-03 17:41:47 -08:00
Guozhang Wang edeb11bc56 MINOR: Move streams-examples source files under src folder
Also remove some unused imports.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #992 from guozhangwang/KSExamples
2016-03-01 18:53:58 -08:00
Christian Posta aeb9c2adcf KAFKA-3272: Add debugging options to kafka-run-class.sh so we can easily run remote debugging
…able remote debugging to Kafka tools scripts

Author: Christian Posta <christian.posta@gmail.com>

Reviewers: Grant Henke, Gwen Shapira

Closes #955 from christian-posta/ceposta-enable-jvm-debugging-opts
2016-02-23 15:55:09 -08:00
Yasuhiro Matsuda 3358e1682f KAFKA-2802: kafka streams system tests
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>

Reviewers: Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #930 from ymatsuda/streams_systest
2016-02-23 12:14:26 -08:00
Matt Fluet 3ee1878d80 MINOR: Reconcile differences in .bat & .sh start scripts
A few minor fixes to reconcile differences between the windows and unix versions of the kafka/zookeeper start scripts that were causing cross-platform inconsistencies during deployment.

- Resolve differences in CLASSPATH setup between .bat and .sh start scripts
- .bat start scripts honor externally provided KAFKA_HEAP_OPTS and KAFKA_LOG4J_OPTS consistent with .sh
- .bat start scripts configure log4j similar to .sh

Author: Matt Fluet <matt.fluet@appian.com>

Reviewers: Eno Thereska <eno.thereska@gmail.com>, Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #908 from fluetm/scripts-patch
2016-02-16 15:23:24 -05:00
Ismael Juma 523562c109 KAFKA-3152; kafka-acl doesn't allow space in principal name
* Add quotes to `$` in shell scripts
This is necessary for correct processing of quotes in the
user command.

* Minor improvements to AclCommand messages

* Use a principal with a space in `SslEndToEndAuthorizationTest`
This passed without any other changes, but good avoid regressions.

* Clean-up `TestSslUtils`:
Remove unused methods, fix unnecessary verbosity and don't set security.protocol (it should be done at a higher-level).

Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com

Closes #818 from ijuma/kafka-3152-kafka-acl-space-in-principal
2016-01-27 08:23:25 -08:00
Grant Henke 91ba074e4a MINOR: Remove remnants of hadoop clients from kafka-run-class.sh
Author: Grant Henke <granthenke@gmail.com>

Reviewers: Gwen Shapira

Closes #799 from granthenke/hadoop-contrib
2016-01-21 18:33:39 -08:00