Commit Graph

246 Commits

Author SHA1 Message Date
Magesh Nandakumar e5492d7726 KAFKA-6997: Exclude test-sources.jar when $INCLUDE_TEST_JARS is FALSE
Exclude test-sources.jar when $INCLUDE_TEST_JARS is FALSE

Author: Magesh Nandakumar <magesh.n.kumar@gmail.com>

Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Robert Yokota <rayokota@gmail.com>, Arjun Satish <arjun@confluent.io>, Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #5139 from mageshn/KAFKA-6997
2018-06-05 16:27:22 -07:00
Manikumar Reddy O 202ab22b98 KAFKA-5523: Remove ReplayLogProducer tool (#5092)
Reviewers: Ismael Juma <ismael@juma.me.uk>
2018-06-03 01:52:11 -07:00
Sasaki Toru 440445e7c5 KAFKA-2061; Offer a --version flag to print the kafka version [KIP-278] (#639)
Reviewers: Andy Lindeman, Jeremy Donahue, Jason Gustafson <jason@confluent.io>
2018-05-25 00:47:54 -07:00
Ismael Juma f2514e67e0 MINOR: Fix kafka-run-class for Java 10 (#4895)
We need to match to the end of the line to make it work with Java 10 as explained in the expanded comment.

Tested manually for all supported versions:

```shell
echo $(.../jdk1.8.0_152.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
1

echo $(.../jdk-9.0.4.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
9

echo $(.../jdk-10.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
10

echo $(.../jdk-10.0.1.jdk/Contents/Home/bin/java -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
10
```
2018-04-19 15:34:13 -07:00
Matthias J. Sax 0c0d8363e5
KAFKA-6054: Fix upgrade path from Kafka Streams v0.10.0 (#4779)
Reviewers: Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>, John Roesler <john@confluent.io>, Damian Guy <damian@confluent.io>
2018-04-06 17:00:52 -07:00
hmcl aa14ec08b7 MINOR: Remove kafka-consumer-offset-checker.bat for KAFKA-3356 (#4703) 2018-03-13 16:41:54 -07:00
Jimin Hsieh 596b604862 MINOR: Fix wrong message in `bin/kafka-run-class.sh` (#4682)
To build jar you need to specify `scalaVersion` instead of `scala_version`.

Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
2018-03-12 14:34:59 -07:00
Jason Gustafson 23c1c52c85
KAFKA-6615; Add scripts for DumpLogSegments (#4653)
Reviewers: Ismael Juma <ismael@juma.me.uk>
2018-03-06 15:37:50 -08:00
Manikumar Reddy 27a8d0f9e7 KAFKA-4541; Support for delegation token mechanism
- Add capability to create delegation token
- Add authentication based on delegation token.
- Add capability to renew/expire delegation tokens.
- Add units tests and integration tests

Author: Manikumar Reddy <manikumar.reddy@gmail.com>

Reviewers: Jun Rao <junrao@gmail.com>

Closes #3616 from omkreddy/KAFKA-4541
2018-01-16 09:50:31 -08:00
Chris Egerton abf4c54701 KAFKA-4247; Prevent CLASSPATH from beginning with a single colon
Different fix for problem addressed by https://github.com/apache/kafka/pull/1953. Should prevent the CLASSPATH environment variable from being prefixed by a single colon before the JVM is invoked in the run-class script, which will then prevent the current working directory from being unintentionally included in the classpath when using the Reflections library.

If the current working directory should still be included in the classpath, it just needs to be explicitly specified either with its fully-qualified pathname or as a single dot (".").

Author: Chris Egerton <chrise@confluent.io>

Reviewers: Randall Hauch <rhauch@gmail.com>, Jun Rao <junrao@gmail.com>

Closes #4406 from C0urante/fix-colon-prefixed-classpath
2018-01-09 14:49:18 -08:00
Prasanna Gautam e5daa40e31 KAFKA-5473; handle ZK session expiration properly when a new session can't be established
(WIP: this commit isn't ready to be reviewed yet. I was checking the travis-ci build with the configuration changes in my account and opened the PR prematurely against trunk. I will make it consistent with Contribution guidelines once it's well tested.)

https://issues.apache.org/jira/browse/KAFKA-5473

Design:
`zookeeper.connection.retry.timeout.ms` => this determines how long to wait before triggering the shutdown. The default is 60000ms.

Currently the implementation only handles the `handleSessionEstablishmentError` by waiting for the sessionTimeout.

Author: Prasanna Gautam <prasannagautam@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>

Closes #3990 from prasincs/KAFKA-5473
2017-12-15 14:48:30 -08:00
Arjun Satish cfa6a78c7c KAFKA-4115: Increasing the heap settings for Connect scripts
Signed-off-by: Arjun Satish <arjunconfluent.io>

Author: Arjun Satish <arjun@confluent.io>

Reviewers: Randall Hauch <rhauch@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4213 from wicknicks/KAFKA-4115
2017-11-21 16:07:06 -08:00
Ismael Juma 3e86161b84 MINOR: Update to Scala 2.11.12 (part 2)
Missed a few places in the previous commit.

Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>

Closes #4200 from ijuma/scala-2.11.12-part2
2017-11-09 18:24:48 +00:00
Vahid Hashemian bfa82fc0b6 KAFKA-6055; Fix typo in KAFKA_JVM_PERFORMANCE_OPTS for Windows
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #4062 from vahidhashemian/KAFKA-6055
2017-10-12 10:09:16 +01:00
Mickael Maison a0f533266a KAFKA-3356: Remove ConsumerOffsetChecker
Author: Mickael Maison <mickael.maison@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>

Closes #3036 from mimaison/KAFKA-3356
2017-09-24 10:08:52 +08:00
Vladimír Kleštinec 81e789ae3d KAFKA-4860; Allow spaces in paths on windows
When we install kafka on path with spaces, batch files were failing, this PR is trying to fix this issue.

Author: Vladimír Kleštinec <klestinec@gmail.com>

Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>

Closes #2649 from klesta490/trunk
2017-09-05 16:27:08 -07:00
Dong Lin adefc8ea07 KAFKA-5694; Add AlterReplicaDirRequest and DescribeReplicaDirRequest (KIP-113 part-1)
Author: Dong Lin <lindong28@gmail.com>

Reviewers: Jun Rao <junrao@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Colin P. Mccabe <cmccabe@confluent.io>

Closes #3621 from lindong28/KAFKA-5694
2017-09-02 23:20:13 -07:00
Colin P. Mccabe 0772fde562 KAFKA-5776; Add the Trogdor fault injection daemon
Author: Colin P. Mccabe <cmccabe@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>

Closes #3699 from cmccabe/trogdor-review
2017-08-25 12:29:40 -07:00
Xavier Léauté 75b0f30c44 KAFKA-5077; fix GC logging arguments for Java 9
Author: Xavier Léauté <xavier@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #2863 from xvrl/fix-jdk9-gc-logs
2017-08-10 02:06:53 +01:00
Evgeny Veretennikov d2a2b80cc6 KAFKA-5507; Check if classpath is empty in kafka-run-class.sh
Author: Evgeny Veretennikov <evg.veretennikov@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #3421 from evis/kafka-run-class-check-classpath
2017-08-09 23:58:25 +01:00
Ismael Juma 69d3d4aad6 KAFKA-5470; Replace -XX:+DisableExplicitGC with -XX:+ExplicitGCInvokesConcurrent in kafka-run-class
This is important because Bits.reserveMemory calls System.gc() hoping to free native
memory in order to avoid throwing an OutOfMemoryException. This call is currently
a no-op due to -XX:+DisableExplicitGC.

It's worth mentioning that -XX:MaxDirectMemorySize can be used to increase the
amount of native memory available for allocation of direct byte buffers.

Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Apurva Mehta <apurva@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>

Closes #3371 from ijuma/kafka-5470-explicit-gc-invokes-concurrent
2017-08-09 23:38:57 +01:00
Ismael Juma bcf447e93e KAFKA-4422; Drop support for Scala 2.10 (KIP-119)
Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Ewen Cheslack-Postava <me@ewencp.org>

Closes #2956 from ijuma/kafka-4422-drop-support-scala-2.10
2017-05-11 08:08:11 +01:00
amethystic 29994dd10d KAFKA-5143: add kafka-broker-api-version.bat for Windows platform
Added kafka-broker-api-versions.bat for Windows platform

Author: amethystic <huxi_2b@hotmail.com>

Reviewers: Vahid Hashemian, Ismael Juma, Guozhang Wang

Closes #2943 from amethystic/kafka-5143_offer_broker_api_versions_bat
2017-05-05 16:53:32 -07:00
Ryan P 8d74920163 KAFKA-3754; Add GC log retention policy to limit size of log
Add a default log retention policy to keep GC logs from growing too large

Author: Ryan P <ryan.n.pridgeon@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1431 from rnpridgeon/KAFKA-3754
2017-05-03 10:51:32 +01:00
Dong Lin 8b05ad406d KAFKA-4586; Add purgeDataBefore() API (KIP-107)
Author: Dong Lin <lindong28@gmail.com>

Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com>

Closes #2476 from lindong28/KAFKA-4586
2017-03-28 09:59:44 -07:00
Jon Freedman dc4d1bb81b MINOR: Update kafka-run-class.bat to handle spaces in classpath
handle existing classpath with spaces
list each lib dependency individually

Author: Jon Freedman <jon.freedman@zoho.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #2433 from jonfreedman/trunk
2017-01-26 14:17:09 -08:00
Xi Hu e9426291aa KAFKA-4684: add kafka-configs.bat for Windows boxes
Add kafka-configs.bat script for Windows.

Author: huxi <huxi@zhenrongbao.com>

Reviewers: Guozhang Wang, Vahid Hashemian

Closes #2419 from amethystic/kafka4684_offer_kafkaconfigs_script
2017-01-25 17:53:51 -08:00
Colin P. Mccabe 4a6f2c6cc0 KAFKA-4457; Add BrokerApiVersionsCommand
Author: Colin P. Mccabe <cmccabe@confluent.io>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Apurva Mehta <apurva.1618@gmail.com>, Ismael Juma <ismael@juma.me.uk>

Closes #2184 from cmccabe/KAFKA-4457
2017-01-18 23:44:11 +00:00
Shikhar Bhushan 2f90488323 KAFKA-3209: KIP-66: single message transforms
Besides API and runtime changes, this PR also includes 2 data transformations (`InsertField`, `HoistToStruct`) and 1 routing transformation (`TimestampRouter`).

There is some gnarliness in `ConnectorConfig` / `ConfigDef` around creating, parsing and validating a dynamic `ConfigDef`.

Author: Shikhar Bhushan <shikhar@confluent.io>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #2299 from shikhar/smt-2017
2017-01-12 16:14:53 -08:00
Vahid Hashemian e799546377 MINOR: Sync up 'kafka-run-class.bat' with 'kafka-run-class.sh'
Some of the recent changes to `kafka-run-class.sh` have not been applied to `kafka-run-class.bat`.
These recent changes include setting proper streams or connect classpaths. So any streams or connect use case that leverages `kafka-run-class.bat` would fail with an error like
```
Error: Could not find or load main class org.apache.kafka.streams.???
```

Author: Vahid Hashemian <vahidhashemian@us.ibm.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #2238 from vahidhashemian/minor/sync_up_kafka-run-class.bat
2016-12-29 12:03:24 -05:00
Vahid Hashemian 055ca9b7a2 KAFKA-4272; Add missing 'connect' Windows batch scripts
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>

Reviewers: Jason Gustafson <jason@confluent.io>

Closes #2146 from vahidhashemian/KAFKA-4272
2016-12-09 11:40:50 -08:00
Vahid Hashemian 8d188c9110 KAFKA-4271: Fix the server start script for Windows 32-bit OS
Without this fix the new consumer fails to run on a 32-bit Windows OS.

Author: Vahid Hashemian <vahidhashemian@us.ibm.com>

Reviewers: Jason Gustafson, Guozhang Wang

Closes #2189 from vahidhashemian/KAFKA-4271
2016-11-30 12:19:23 -08:00
Konstantine Karantasis aad9384066 MINOR: Extract SCALA_BINARY_VERSION from SCALA_VERSION
Will allow users to set one fewer environment variable if they need to
change scala version. Still, SCALA_BINARY_VERSION can be explicitly set.

Author: Konstantine Karantasis <konstantine@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #2130 from kkonstantine/MINOR-Extract-SCALA_BINARY_VERSION-from-SCALA_VERSION
2016-11-17 02:15:14 +00:00
amethystic 6f842c1e9e KAFKA-4310; Add missing Windows script `kafka-consumer-groups.bat`
add file 'kafka-consumer-groups.bat' for Windows platform

Author: amethystic <huxi_2b@hotmail.com>

Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Jason Gustafson <jason@confluent.io>

Closes #2037 from amethystic/kafka-4310_add_consumer_groups_script
2016-10-18 11:08:10 -07:00
1ambda dca63fc0a5 KAFKA-3742: (FIX) Can't run bin/connect-*.sh with -daemon flag
## Problem

Current connect scripts (`connect-distributed.sh`, `connect-standalone.sh`) do not support `-daemon` flag even if users specify the flag
since `kafka-run-class.sh` requires that the`-daemon` flag should precede other arguments (e.g. class name)

## Solution

Do the same thing like in `kafka-server-start.sh`

- Parse a command
- Add `-daemon` to `$EXTRA_ARGS` if exists

Author: 1ambda <1amb4a@gmail.com>

Reviewers: Gwen Shapira

Closes #1717 from 1ambda/KAFKA-3742-connect-running-as-daemon
2016-08-25 20:23:05 -07:00
P. Thorpe e3ede84344 MINOR: Remove # from .bat start script
On Windows, the following output is seen when starting Zookeeper and Kafka servers:
```
'#' is not recognized as an internal or external command,
operable program or batch file.
```

This pull request makes a minor correction to the Windows `kafka-run-class.bat` script to replace the use of `#` with `rem`.

Author: P. Thorpe <paul.thorpe@uk.ibm.com>

Reviewers: Vahid Hashemian, Gwen Shapira

Closes #1740 from p-thorpe/trunk
2016-08-18 20:50:20 -07:00
Grant Henke 1572274583 KAFKA-3934: Start scripts enable GC by default with no way to disable
…o disable

Author: Grant Henke <granthenke@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1631 from granthenke/garbage-flag
2016-08-09 09:16:17 -07:00
deragon cb7af0b7fb MINOR: Make kafka-run-class.sh work under Cygwin
kafka-run-class.sh now runs under Cygwin; paths and classpath are set up properly.

**WARNING:**  The script was not tested on a Linux machine, only under Cygwin.  Prior to merge it into trunk, if accepted, please run a quick test to ensure nothing broke.  From my own code review, there should not be any problem, but we can never be too sure.

I do not have the environment to test it under Linux at this moment.

Author: deragon <32nx9812masakjds>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1487 from deragon/trunk
2016-08-09 08:33:53 -07:00
Matthias J. Sax f7976d2fc1 KAFKA-4008: Module "tools" should not be dependent on "core"
moved streams application reset tool from tools to core

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1685 from mjsax/moveResetTool

(cherry picked from commit f2405a73ea)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
2016-08-01 20:12:38 -07:00
Matthias J. Sax 8deedcacb6 KAFKA-3185: Allow users to cleanup internal Kafka Streams data
- added Kafka Stream Application Reset Tool

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Guozhang Wang, Michael G. Noll

Closes #1636 from mjsax/kafka-3185
2016-07-27 14:11:40 -07:00
Liquan Pei fb421dbcfe KAFKA-3692; Add quotes to variables in kafka-run-class.sh
Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Geoff Anderson <geoff@confluent.io>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>

Closes #1364 from Ishiihara/add-quote-classpath
2016-05-13 12:10:09 +01:00
Dana Powers 1e4dd66b19 HOTFIX: Fix main classpath libs glob for release (fixup KAFKA-3615 regression)
bin/kafka-run-class.sh does not correctly setup the CLASSPATH in release rc2.

Author: Dana Powers <dana.powers@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1302 from dpkp/KAFKA-3615-fix
2016-04-30 18:53:44 -07:00
Liquan Pei eb50d2f6ca KAFKA-3615: Exclude test jars in kafka-run-class.sh
granders hachikuji Can you take a look when you have time? Appreciate your time to review.

Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Grant Henke <granthenke@gmail.com>, Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1263 from Ishiihara/classpath-no-test-jar
2016-04-29 10:29:00 -07:00
Liquan Pei 316389d6ad KAFKA-3611: Remove warnings when using reflections
ewencp granders Can you take a look? Thanks!

Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1259 from Ishiihara/fix-warning
2016-04-28 11:59:02 -07:00
Matt McClure 732b111f45 KAFKA-3384: Conform to POSIX kill usage
I believe this addresses KAFKA-3384.

The POSIX kill manpage is at http://pubs.opengroup.org/onlinepubs/9699919799/utilities/kill.html

Author: Matt McClure <mlm@aya.yale.edu>

Reviewers: Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1148 from matthewlmcclure/KAFKA-3384
2016-04-04 22:07:20 -07:00
Yasuhiro Matsuda de06244338 MINOR: remove streams-smoke-test.sh
guozhangwang

Author: Yasuhiro Matsuda <yasuhiro@confluent.io>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #1125 from ymatsuda/remove_smoketest_shell_script
2016-03-23 14:57:03 -07:00
Sasaki Toru f7887bd6f9 KAFKA-3281: Improve message of *-server-stop.sh when process is not running
Stop scritps such as kafka-server-stop.sh log messages of kill command's error when processes aren't running.
This PR changes this message to "No kafka server to stop".

Author: Sasaki Toru <sasakitoa@nttdata.co.jp>

Reviewers: Gwen Shapira

Closes #971 from sasakitoa/stop_scripts_says_not_good_message
2016-03-03 17:41:47 -08:00
Guozhang Wang edeb11bc56 MINOR: Move streams-examples source files under src folder
Also remove some unused imports.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #992 from guozhangwang/KSExamples
2016-03-01 18:53:58 -08:00
Christian Posta aeb9c2adcf KAFKA-3272: Add debugging options to kafka-run-class.sh so we can easily run remote debugging
…able remote debugging to Kafka tools scripts

Author: Christian Posta <christian.posta@gmail.com>

Reviewers: Grant Henke, Gwen Shapira

Closes #955 from christian-posta/ceposta-enable-jvm-debugging-opts
2016-02-23 15:55:09 -08:00
Yasuhiro Matsuda 3358e1682f KAFKA-2802: kafka streams system tests
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>

Reviewers: Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #930 from ymatsuda/streams_systest
2016-02-23 12:14:26 -08:00
Matt Fluet 3ee1878d80 MINOR: Reconcile differences in .bat & .sh start scripts
A few minor fixes to reconcile differences between the windows and unix versions of the kafka/zookeeper start scripts that were causing cross-platform inconsistencies during deployment.

- Resolve differences in CLASSPATH setup between .bat and .sh start scripts
- .bat start scripts honor externally provided KAFKA_HEAP_OPTS and KAFKA_LOG4J_OPTS consistent with .sh
- .bat start scripts configure log4j similar to .sh

Author: Matt Fluet <matt.fluet@appian.com>

Reviewers: Eno Thereska <eno.thereska@gmail.com>, Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #908 from fluetm/scripts-patch
2016-02-16 15:23:24 -05:00
Ismael Juma 523562c109 KAFKA-3152; kafka-acl doesn't allow space in principal name
* Add quotes to `$` in shell scripts
This is necessary for correct processing of quotes in the
user command.

* Minor improvements to AclCommand messages

* Use a principal with a space in `SslEndToEndAuthorizationTest`
This passed without any other changes, but good avoid regressions.

* Clean-up `TestSslUtils`:
Remove unused methods, fix unnecessary verbosity and don't set security.protocol (it should be done at a higher-level).

Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com

Closes #818 from ijuma/kafka-3152-kafka-acl-space-in-principal
2016-01-27 08:23:25 -08:00
Grant Henke 91ba074e4a MINOR: Remove remnants of hadoop clients from kafka-run-class.sh
Author: Grant Henke <granthenke@gmail.com>

Reviewers: Gwen Shapira

Closes #799 from granthenke/hadoop-contrib
2016-01-21 18:33:39 -08:00
Atul Soman c06542edea KAFKA-3119: Adding -daemon option to zookeeper-server-start.sh USAGE, similar to kafka-server-start.sh
Output after fix:

#satul# ./kafka-server-start.sh
USAGE: ./kafka-server-start.sh [-daemon] server.properties
#satul# ./zookeeper-server-start.sh
USAGE: ./zookeeper-server-start.sh [-daemon] zookeeper.properties

Author: Atul Soman <atul.soman@microfocus.com>

Reviewers: Gwen Shapira

Closes #785 from atulsm/trunk
2016-01-18 18:16:12 -08:00
Michael G. Noll d9cabfde6c KAFKA-2874; shutdown ZK process reliably
Author: Michael G. Noll <michael@confluent.io>

Reviewers: Flavio Junqueira <fpj@apache.org>, Jun Rao <junrao@gmail.com>

Closes #573 from miguno/KAFKA-2874
2016-01-06 15:21:49 -08:00
Guozhang Wang d05fa0a03b KAFKA-2804: manage changelog topics through ZK in PartitionAssignor
Author: Guozhang Wang <wangguoz@gmail.com>
Author: wangguoz@gmail.com <guozhang@Guozhang-Macbook.local>
Author: Guozhang Wang <guozhang@Guozhang-Macbook.local>

Reviewers: Yasuhiro Matsuda

Closes #579 from guozhangwang/K2804
2015-12-07 15:12:09 -08:00
Grant Henke 8f6ffe1c28 KAFKA-2800; Update outdated dependencies
Changes:
* org.scala-lang:scala-library [2.10.5 -> 2.10.6]
   * Scala 2.10.6 resolves a license incompatibility in scala.util.Sorting
   * Otherwise identical to Scala 2.10.5
* org.xerial.snappy:snappy-java [1.1.1.7 -> 1.1.2]
   * Fixes SnappyOutputStream.close() is not idempotent
* net.jpountz.lz4:lz4 [1.2.0 -> 1.3]
* junit:junit [4.11 -> 4.12]
* org.easymock:easymock [3.3.1 -> 3.4]
* org.powermock:powermock-api-easymock [1.6.2 -> 1.6.3]
* org.powermock:powermock-module-junit4 [1.6.2 -> 1.6.3]
* org.slf4j:slf4j-api [1.7.6 -> 1.7.12]
* org.slf4j:slf4j-log4j12 [1.7.6 -> 1.7.12]
* com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider [2.5.4 -> 2.6.3]
* com.fasterxml.jackson.core:jackson-databind [2.5.4 -> 2.6.3]
* org.eclipse.jetty:jetty-server [9.2.12.v20150709 -> 9.2.14.v20151106]
* org.eclipse.jetty:jetty-servlet [9.2.12.v20150709 -> 9.2.14.v20151106]
* org.bouncycastle:bcpkix-jdk15on [1.52 -> 1.53]
* net.sf.jopt-simple:jopt-simple [3.2 -> 4.9]
* removed explicit entry for org.objenesis:objenesis:2.2 (resolved transitively)

Author: Grant Henke <granthenke@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>

Closes #513 from granthenke/update-deps
2015-11-18 10:42:15 -08:00
Ewen Cheslack-Postava 8db55618d5 KAFKA-2752: Add VerifiableSource/Sink connectors and rolling bounce Copycat system tests.
Author: Ewen Cheslack-Postava <me@ewencp.org>

Reviewers: Ben Stopford, Geoff Anderson, Guozhang Wang

Closes #432 from ewencp/kafka-2752-copycat-clean-bounce-test
2015-11-10 14:54:15 -08:00
Jason Gustafson bce664b42a KAFKA-2274: verifiable consumer and integration testing
Author: Jason Gustafson <jason@confluent.io>

Reviewers: Guozhang Wang, Geoff Anderson

Closes #465 from hachikuji/KAFKA-2274
2015-11-09 18:38:22 -08:00
Ewen Cheslack-Postava f2031d4063 KAFKA-2774: Rename Copycat to Kafka Connect
Author: Ewen Cheslack-Postava <me@ewencp.org>

Reviewers: Gwen Shapira

Closes #456 from ewencp/kafka-2774-rename-copycat
2015-11-08 22:11:03 -08:00
Michael G. Noll 7ded64bc2e KAFKA-2740: Convert Windows bin scripts from CRLF to LF line encodings
There are no functional changes to the modified scripts.

Author: Michael G. Noll <michael@confluent.io>

Reviewers: Guozhang Wang

Closes #419 from miguno/KAFKA-2740
2015-11-04 09:19:49 -08:00
Rajini Sivaram 694e03c355 KAFKA-2719; Use wildcard classpath for dependant-libs
PR switches to wildcard classpath for dependant libs to restrict the length of classpath, thereby reducing command line length.

Author: Rajini Sivaram <rajinisivaram@googlemail.com>

Reviewers: Jun Rao <junrao@gmail.com>

Closes #400 from rajinisivaram/KAFKA-2719
2015-11-03 08:10:37 -08:00
Manikumar reddy O 1cc44830b9 KAFKA-2562: update kafka scripts to use new tools/code
Updated  kafka-producer-perf-test.sh to use org.apache.kafka.clients.tools.ProducerPerformance.
Updated build.gradle to add kafka-tools-0.9.0.0-SNAPSHOT.jar to kafka/libs  folder.

Author: Manikumar reddy O <manikumar.reddy@gmail.com>

Reviewers: Gwen Shapira, Ismael Juma

Closes #242 from omkreddy/KAFKA-2562
2015-10-30 15:30:34 -07:00
Grant Henke 2e4aed7070 KAFKA-2516: Rename o.a.k.client.tools to o.a.k.tools
Author: Grant Henke <granthenke@gmail.com>

Reviewers: Gwen Shapira, Ewen Cheslack-Postava

Closes #310 from granthenke/tools-packaging
2015-10-27 07:44:32 -07:00
flavio junqueira 21443f214f KAFKA-2641; Upgrade path for ZK authentication
This pull request adds a configuration parameter and a migration tool. It is also based on pull request #303, which should go in first.

Author: flavio junqueira <fpj@apache.org>
Author: Flavio Junqueira <fpj@apache.org>
Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>

Closes #313 from fpj/KAFKA-2641
2015-10-23 15:11:07 -07:00
Parth Brahmbhatt 5764e54de1 KAFKA-2212: Authorizer CLI implementation.
Author: Parth Brahmbhatt <brahmbhatt.parth@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>

Closes #230 from Parth-Brahmbhatt/KAFKA-2212
2015-10-01 18:04:09 -07:00
Manikumar reddy O c7bc5366dd KAFKA-2514: change default JVM options in kafka-run-class.sh.
GC is set to G1 collector.

Author: Manikumar reddy O <manikumar.reddy@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>

Closes #218 from omkreddy/KAFKA-2514
2015-09-28 08:40:32 -07:00
Guozhang Wang 263c10ab7c KIP-28: Add a processor client for Kafka Streaming
This work has been contributed by Jesse Anderson, Randall Hauch, Yasuhiro Matsuda and Guozhang Wang. The detailed design can be found in https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client.

Author: Guozhang Wang <wangguoz@gmail.com>
Author: Yasuhiro Matsuda <yasuhiro.matsuda@gmail.com>
Author: Yasuhiro Matsuda <yasuhiro@confluent.io>
Author: ymatsuda <yasuhiro.matsuda@gmail.com>
Author: Randall Hauch <rhauch@gmail.com>
Author: Jesse Anderson <jesse@smokinghand.com>
Author: Ismael Juma <ismael@juma.me.uk>
Author: Jesse Anderson <eljefe6a@gmail.com>

Reviewers: Ismael Juma, Randall Hauch, Edward Ribeiro, Gwen Shapira, Jun Rao, Jay Kreps, Yasuhiro Matsuda, Guozhang Wang

Closes #130 from guozhangwang/streaming
2015-09-25 17:27:58 -07:00
Ewen Cheslack-Postava 48b4d6938d KAFKA-2373: Add Kafka-backed offset storage for Copycat.
Author: Ewen Cheslack-Postava <me@ewencp.org>

Reviewers: Gwen Shapira, James Cheng

Closes #202 from ewencp/kafka-2373-copycat-distributed-offset
2015-09-24 19:01:11 -07:00
Ewen Cheslack-Postava 492bfdfa87 KAFKA-2367; Add Copycat runtime data API.
Author: Ewen Cheslack-Postava <me@ewencp.org>

Reviewers: Gwen Shapira, Jay Kreps

Closes #163 from ewencp/kafka-2367-copycat-runtime-data-api
2015-08-27 11:58:42 -07:00
Ewen Cheslack-Postava f6acfb0891 KAFKA-2366; Initial patch for Copycat
This is an initial patch implementing the basics of Copycat for KIP-26.

The intent here is to start a review of the key pieces of the core API and get a reasonably functional, baseline, non-distributed implementation of Copycat in place to get things rolling. The current patch has a number of known issues that need to be addressed before a final version:

* Some build-related issues. Specifically, requires some locally-installed dependencies (see below), ignores checkstyle for the runtime data library because it's lifted from Avro currently and likely won't last in its current form, and some Gradle task dependencies aren't quite right because I haven't gotten rid of the dependency on `core` (which should now be an easy patch since new consumer groups are in a much better state).
* This patch currently depends on some Confluent trunk code because I prototyped with our Avro serializers w/ schema-registry support. We need to figure out what we want to provide as an example built-in set of serializers. Unlike core Kafka where we could ignore the issue, providing only ByteArray or String serializers, this is pretty central to how Copycat works.
* This patch uses a hacked up version of Avro as its runtime data format. Not sure if we want to go through the entire API discussion just to get some basic code committed, so I filed KAFKA-2367 to handle that separately. The core connector APIs and the runtime data APIs are entirely orthogonal.
* This patch needs some updates to get aligned with recent new consumer changes (specifically, I'm aware of the ConcurrentModificationException issue on exit). More generally, the new consumer is in flux but Copycat depends on it, so there are likely to be some negative interactions.
* The layout feels a bit awkward to me right now because I ported it from a Maven layout. We don't have nearly the same level of granularity in Kafka currently (core and clients, plus the mostly ignored examples, log4j-appender, and a couple of contribs). We might want to reorganize, although keeping data+api separate from runtime and connector plugins is useful for minimizing dependencies.
* There are a variety of other things (e.g., I'm not happy with the exception hierarchy/how they are currently handled, TopicPartition doesn't really need to be duplicated unless we want Copycat entirely isolated from the Kafka APIs, etc), but I expect those we'll cover in the review.

Before commenting on the patch, it's probably worth reviewing https://issues.apache.org/jira/browse/KAFKA-2365 and https://issues.apache.org/jira/browse/KAFKA-2366 to get an idea of what I had in mind for a) what we ultimately want with all the Copycat patches and b) what we aim to cover in this initial patch. My hope is that we can use a WIP patch (after the current obvious deficiencies are addressed) while recognizing that we want to make iterative progress with a bunch of subsequent PRs.

Author: Ewen Cheslack-Postava <me@ewencp.org>

Reviewers: Ismael Juma, Gwen Shapira

Closes #99 from ewencp/copycat and squashes the following commits:

a3a47a6 [Ewen Cheslack-Postava] Simplify Copycat exceptions, make them a subclass of KafkaException.
8c108b0 [Ewen Cheslack-Postava] Rename Coordinator to Herder to avoid confusion with the consumer coordinator.
7bf8075 [Ewen Cheslack-Postava] Make Copycat CLI speific to standalone mode, clean up some config and get rid of config storage in standalone mode.
656a003 [Ewen Cheslack-Postava] Clarify and expand the explanation of the Copycat Coordinator interface.
c0e5fdc [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat
0fa7a36 [Ewen Cheslack-Postava] Mark Copycat classes as unstable and reduce visibility of some classes where possible.
d55d31e [Ewen Cheslack-Postava] Reorganize Copycat code to put it all under one top-level directory.
b29cb2c [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat
d713a21 [Ewen Cheslack-Postava] Address Gwen's review comments.
6787a85 [Ewen Cheslack-Postava] Make Converter generic to match serializers since some serialization formats do not require a base class of Object; update many other classes to have generic key and value class type parameters to match this change.
b194c73 [Ewen Cheslack-Postava] Split Copycat converter option into two options for key and value.
0b5a1a0 [Ewen Cheslack-Postava] Normalize naming to use partition for both source and Kafka, adjusting naming in CopycatRecord classes to clearly differentiate.
e345142 [Ewen Cheslack-Postava] Remove Copycat reflection utils, use existing Utils and ConfigDef functionality from clients package.
be5c387 [Ewen Cheslack-Postava] Minor cleanup
122423e [Ewen Cheslack-Postava] Style cleanup
6ba87de [Ewen Cheslack-Postava] Remove most of the Avro-based mock runtime data API, only preserving enough schema functionality to support basic primitive types for an initial patch.
4674d13 [Ewen Cheslack-Postava] Address review comments, clean up some code styling.
25b5739 [Ewen Cheslack-Postava] Fix sink task offset commit concurrency issue by moving it to the worker thread and waking up the consumer to ensure it exits promptly.
0aefe21 [Ewen Cheslack-Postava] Add log4j settings for Copycat.
220e42d [Ewen Cheslack-Postava] Replace Avro serializer with JSON serializer.
1243a7c [Ewen Cheslack-Postava] Merge remote-tracking branch 'origin/trunk' into copycat
5a618c6 [Ewen Cheslack-Postava] Remove offset serializers, instead reusing the existing serializers and removing schema projection support.
e849e10 [Ewen Cheslack-Postava] Remove duplicated TopicPartition implementation.
dec1379 [Ewen Cheslack-Postava] Switch to using new consumer coordinator instead of manually assigning partitions. Remove dependency of copycat-runtime on core.
4a9b4f3 [Ewen Cheslack-Postava] Add some helpful Copycat-specific build and test targets that cover all Copycat packages.
31cd1ca [Ewen Cheslack-Postava] Add CLI tools for Copycat.
e14942c [Ewen Cheslack-Postava] Add Copycat file connector.
0233456 [Ewen Cheslack-Postava] Add copycat-avro and copycat-runtime
11981d2 [Ewen Cheslack-Postava] Add copycat-data and copycat-api
2015-08-14 16:00:51 -07:00
Grant Henke fc40016cf3 KAFKA-2407: Only create log directory when it will be used
Author: Grant Henke <granthenke@gmail.com>

Reviewers: Gwen Shapira

Closes #115 from granthenke/log-fix and squashes the following commits:

de36138 [Grant Henke] Small comment fix
49a8dd4 [Grant Henke] KAFKA-2407: Only create log directory when it will be used
2015-08-05 11:46:11 -07:00
Aditya Auradkar a56a79055d kafka-2205; Generalize TopicConfigManager to handle multiple entity configs; patched by Aditya Auradkar; reviewed Jun Rao 2015-08-04 15:11:27 -07:00
Geoff Anderson e43c9aff92 KAFKA-2276; KIP-25 initial patch
Initial patch for KIP-25

Note that to install ducktape, do *not* use pip to install ducktape. Instead:

```
$ git clone gitgithub.com:confluentinc/ducktape.git
$ cd ducktape
$ python setup.py install
```

Author: Geoff Anderson <geoff@confluent.io>
Author: Geoff <granders@gmail.com>
Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Ewen, Gwen, Jun, Guozhang

Closes #70 from granders/KAFKA-2276 and squashes the following commits:

a62fb6c [Geoff Anderson] fixed checkstyle errors
a70f0f8 [Geoff Anderson] Merged in upstream trunk.
8b62019 [Geoff Anderson] Merged in upstream trunk.
47b7b64 [Geoff Anderson] Created separate tools jar so that the clients package does not pull in dependencies on the Jackson JSON tools or argparse4j.
a9e6a14 [Geoff Anderson] Merged in upstream changes
d18db7b [Geoff Anderson] fixed :rat errors (needed to add licenses)
321fdf8 [Geoff Anderson] Ignore tests/ and vagrant/ directories when running rat build task
795fc75 [Geoff Anderson] Merged in changes from upstream trunk.
1d93f06 [Geoff Anderson] Updated provisioning to use java 7 in light of KAFKA-2316
2ea4e29 [Geoff Anderson] Tweaked README, changed default log collection behavior on VerifiableProducer
0eb6fdc [Geoff Anderson] Merged in system-tests
69dd7be [Geoff Anderson] Merged in trunk
4034dd6 [Geoff Anderson] Merged in upstream trunk
ede6450 [Geoff] Merge pull request #4 from confluentinc/move_muckrake
7751545 [Geoff Anderson] Corrected license headers
e6d532f [Geoff Anderson] java 7 -> java 6
8c61e2d [Geoff Anderson] Reverted jdk back to 6
f14c507 [Geoff Anderson] Removed mode = "test" from Vagrantfile and Vagrantfile.local examples. Updated testing README to clarify aws setup.
98b7253 [Geoff Anderson] Updated consumer tests to pre-populate kafka logs
e6a41f1 [Geoff Anderson] removed stray println
b15b24f [Geoff Anderson] leftover KafkaBenchmark in super call
0f75187 [Geoff Anderson] Rmoved stray allow_fail. kafka_benchmark_test -> benchmark_test
f469f84 [Geoff Anderson] Tweaked readme, added example Vagrantfile.local
3d73857 [Geoff Anderson] Merged downstream changes
42dcdb1 [Geoff Anderson] Tweaked behavior of stop_node, clean_node to generally fail fast
7f7c3e0 [Geoff Anderson] Updated setup.py for kafkatest
c60125c [Geoff Anderson] TestEndToEndLatency -> EndToEndLatency
4f476fe [Geoff Anderson] Moved aws scripts to vagrant directory
5af88fc [Geoff Anderson] Updated README to include aws quickstart
e5edf03 [Geoff Anderson] Updated example aws Vagrantfile.local
96533c3 [Geoff] Update aws-access-keys-commands
25a413d [Geoff] Update aws-example-Vagrantfile.local
884b20e [Geoff Anderson] Moved a bunch of files to kafkatest directory
fc7c81c [Geoff Anderson] added setup.py
632be12 [Geoff] Merge pull request #3 from confluentinc/verbose-client
51a94fd [Geoff Anderson] Use argparse4j instead of joptsimple. ThroughputThrottler now has more intuitive behavior when targetThroughput is 0.
a80a428 [Geoff Anderson] Added shell program for VerifiableProducer.
d586fb0 [Geoff Anderson] Updated comments to reflect that throttler is not message-specific
6842ed1 [Geoff Anderson] left out a file from last commit
1228eef [Geoff Anderson] Renamed throttler
9100417 [Geoff Anderson] Updated command-line options for VerifiableProducer. Extracted throughput logic to make it reusable.
0a5de8e [Geoff Anderson] Fixed checkstyle errors. Changed name to VerifiableProducer. Added synchronization for thread safety on println statements.
475423b [Geoff Anderson] Convert class to string before adding to json object.
bc009f2 [Geoff Anderson] Got rid of VerboseProducer in core (moved to clients)
c0526fe [Geoff Anderson] Updates per review comments.
8b4b1f2 [Geoff Anderson] Minor updates to VerboseProducer
2777712 [Geoff Anderson] Added some metadata to producer output.
da94b8c [Geoff Anderson] Added number of messages option.
07cd1c6 [Geoff Anderson] Added simple producer which prints status of produced messages to stdout.
a278988 [Geoff Anderson] fixed typos
f1914c3 [Liquan Pei] Merge pull request #2 from confluentinc/system_tests
81e4156 [Liquan Pei] Bootstrap Kafka system tests
2015-07-28 17:22:14 -07:00
Ismael Juma 66c6f9b1c4 kafka-2039; Update Scala to 2.10.5 and 2.11.6; patched by Ismael Juma; reviewed by Jun Rao 2015-04-01 13:26:45 -07:00
Onur Karaman 7c82afddc8 Added script for bin/kafka-consumer-groups.sh that was mistakenly dropped from the original patch for KAFKA-1476 2015-02-24 08:25:33 -08:00
Manikumar Reddy 995d0d3693 KAFKA-1109 Need to fix GC log configuration code, not able to override KAFKA_GC_LOG_OPTS; reviewed by Neha Narkhede 2015-01-25 18:43:28 -08:00
Jun Rao 09234f0e52 trivial change to remove -UseCompressedOops option from script 2015-01-16 19:04:11 -08:00
Guozhang Wang 72601f783e KAFKA-1624; bump up default scala version to 2.11.4 to compile with java 8; reviewed by Joe Stein, Gwen Shapira and Joel Koshy 2014-11-24 11:45:14 -08:00
Jarek Jarcec Cecho 409c367ceb KAFKA-1654 Provide a way to override server configuration from command line; reviewed by Neha Narkhede 2014-11-23 16:41:24 -08:00
Stevo Slavic f9d9b38143 kafka-1419 (followup); cross build for scala 2.11; patched by Stevo Slavic; reviewed by Jun Rao 2014-09-14 10:45:37 -07:00
Ivan Lyutov 4d0759713f kafka-1419; cross build for scala 2.11; patched by Ivan Lyutov; reviewed by Joe Stein and Jun Rao 2014-08-10 21:20:30 -07:00
Jay Kreps 6b0ae4bba0 KAFKA-1291 Add wrapper scripts and usage information to each command. 2014-06-19 14:04:06 -07:00
Sriharsha Chintalapani 1363ed7c52 KAFKA-1438 Migrate client tools out of perf; reviewed by Neha Narkhede 2014-06-04 22:20:55 -07:00
Stevo Slavic 4bd33e5ba7 kafka-1210; Windows Bat files are not working properly; patched by Stevo Slavic; reviewed by Jun Rao 2014-04-13 08:43:53 -07:00
Martin Kleppmann c3520fe7e0 kafka-server-stop.sh doesn't stop broker; reviewed by Neha Narkhede 2014-03-06 09:35:20 -08:00
Nathan Brown edbed2823f kafka-1278; More flexible helper scripts; patched by Nathan Brown; reviewed by Jun Rao 2014-02-25 09:37:15 -08:00
Jun Rao d392f56b80 trivial change to use the correct path of kafka-clients.jar in kafka-run-class.sh 2014-02-19 21:32:46 -08:00
Joe Stein a8297cc6ca KAFKA-1254 remove vestigial sbt patch by Joe Stein; reviewed by Jun Rao 2014-02-20 00:11:31 -05:00
Jay Kreps 6b80dbb97b KAFKA-1236 Fix various breakages in the perf tests. Make the producer test use either the old or the new producer. 2014-02-10 20:49:08 -08:00
Joe Stein 8014b6ff69 KAFKA-1158 run rat is not needed this is documented now in the release not part of the server running 2014-02-09 09:16:06 -05:00
Jun Rao 52614ad65e kafka-1171; Gradle build for Kafka; patched by David Arthur, Joe Stein, Chris Freeman and Jun Rao; reviewed by Guozhang Wang, Joel Koshy and Neha Narkhede 2014-02-07 13:48:04 -08:00
Jason Rosenberg 74b42a8f81 KAFKA-1164 kafka should depend on snappy 1.0.5 (instead of 1.0.4.1); reviewed by Neha Narkhede 2014-02-03 21:16:55 -08:00
Philp K. Warren c9028ad8c5 kafka-1214; Support arguments to zookeeper-shell.sh script; patched by Philp K. Warren; reviewed by Jun Rao 2014-01-24 09:58:28 -08:00
Jun Rao 58789d7cbe kafka-1127; kafka and zookeeper server should start in daemon mode and log to correct position; patched by Raymond Liu; reviewed by Jun Rao 2013-11-18 21:02:00 -08:00
Jay Kreps 3e7c9c6abf KAFKA-1081 Clean up shell scripts. Removed re-run of failed command. Use exec instead of subprocess execution. 2013-10-14 09:39:10 -07:00
Jun Rao ace95c48a9 trivial change to remove unused command bin/kafka-add-partitions.sh 2013-10-10 07:58:10 -07:00
Jun Rao bfc4ba4995 Merge remote branch 'origin/0.8' into trunk 2013-10-07 14:29:16 -07:00
Jun Rao 71ed6ca336 kafka-1073; CheckReassignmentStatus is broken; patched by Jun Rao; reviewed by Guozhang Wang, Swapnil Ghike and Neha Narkhede 2013-10-07 09:22:12 -07:00
Jun Rao 75d95d9b07 merge from 0.8 da4512174b and resolve conflicts 2013-09-11 09:49:48 -07:00
Vladislav Pernin 51de7c55d2 kafka-1031; Little modification to the stop script to be able to kill the proper process; patched by Vladislav Pernin; reviewed by Jun Rao 2013-08-28 09:51:19 -07:00
Jay Kreps 14af713252 Misc. minor house-keeping fixes: add reasonable GC settings, fix up README, fix up example configs, fix the logging for tools, use a log directory for logs instead of the root directory. 2013-08-05 10:10:18 -07:00
Sriram Subramanian 3817857b15 kafka-347; change number of partitions of a topic online; patched by Sriram Subramanian; reviewed by Neha Narkehede, Guozhang Wang, Joel Koshy and Jun Rao 2013-07-22 21:33:28 -07:00
Jun Rao db37ed0054 kafka-965; merge c39d37e9dd from 0.8 to trunk; patched by Jun Rao; reviewed by Jay Kreps 2013-07-10 23:36:51 -07:00
Jun Rao c98bdd3e4f kafka-896; merge 0.8 (988d4d8e65) to trunk; patched by Jun Rao; reviewed by Jay Kreps 2013-07-08 15:42:24 -07:00
Jun Rao 436dd25a48 kafka-259; Give better error message when trying to run shell scripts without having built/downloaded the jars yet; patched by Ashwanth Fernando; reviewed by Jun Rao 2013-05-29 20:26:08 -07:00
Jun Rao 731ba9007d kafka-823; merge 0.8 (51421fcc01) to trunk; patched by Jun Rao; reviewed by Jay Kreps 2013-04-17 21:52:04 -07:00
Cosmin Lehene 158baf661c kafka-843; Re-add the release-zip sbt target; patched by Cosmin Lehene; reviewed by Jun Rao 2013-04-11 11:01:11 -07:00
Neha Narkhede 6dbf9212ae KAFKA-826 Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x; reviewed by Swapnil Ghike, Neha Narkhede, Matt Christiansen, Scott Carey 2013-04-09 16:07:58 -07:00
Jay Kreps c1ed12e44d KAFKA-554 Dynamic per-topic configuration. This patch adds a mechanism for storing per-topic configurations in zookeeper and dynamically making config changes across the cluster. Reviewed by Neha and Jun. 2013-03-08 15:07:39 -08:00
Swapnil Ghike 2457bc49ef KAFKA-513 Add state change log to Kafka brokers; reviewed by Neha Narkhede 2013-03-06 09:35:35 -08:00
Jun Rao b84168a3e3 Fix windows build script - kafka-run-class.bat; patched by Mark deVilliers; reviewed by Jun Rao; kafka-751 2013-02-14 08:23:48 -08:00
Joe Stein cb3359f9f1 KAFKA-731 use a variable for the ivy2 path in kafka-run-class.sh patch by Joe Stein reviewed by Neha Narkhede 2013-01-24 01:23:44 -05:00
Joe Stein aeac593a90 KAFKA-728 snappy jar missing from path in kafka-run-class.sh patch by John Fung reviewed by Joe Stein 2013-01-23 21:02:12 -05:00
John Fung b5d685bb7c KAFKA-722 Fix the classpath in kafka-run-class.sh; reviewed by Neha Narkhede 2013-01-22 12:15:20 -08:00
Joe Stein 5e3e181ace KAFKA-139 cross-compile multiple Scala versions the dependency jars moved and the bin scripts needed to point to the new locations 2013-01-21 21:33:53 -05:00
Derek Chen-Becker 45cb03d456 Re-applied old patch from KAFKA-139
Had to make some additional changes based on further mainline work. Leaving
KafkaProject.scala in place as a reference for now.
2012-12-14 17:12:38 -07:00
Jun Rao e15432efd3 provides windows batch script for starting Kafka/Zookeeper; patched by Antoine Vianey; reviewed by Jun Rao; kafka-581
git-svn-id: https://svn.apache.org/repos/asf/kafka/branches/0.8@1420001 13f79535-47bb-0310-9956-ffa450edef68
2012-12-11 05:38:37 +00:00
Jun Rao 7b2d9c3312 remove ProducerShell; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-638
git-svn-id: https://svn.apache.org/repos/asf/kafka/branches/0.8@1415002 13f79535-47bb-0310-9956-ffa450edef68
2012-11-29 00:03:13 +00:00
Jun Rao 0d9f7ffe4d provides windows batch script for starting Kafka/Zookeeper; patched by Antoine Vianey; reviewed by Jun Rao; kafka-581
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1401782 13f79535-47bb-0310-9956-ffa450edef68
2012-10-24 16:57:56 +00:00
Neha Narkhede 3015d0a447 KAFKA-43 Move leader to preferred replica; patched by Neha Narkhede; reviewed by Joel Koshy and Jun Rao
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1397747 13f79535-47bb-0310-9956-ffa450edef68
2012-10-13 00:39:41 +00:00
Jun Rao 2b13581711 trival fix to remove 2 empty files
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1397623 13f79535-47bb-0310-9956-ffa450edef68
2012-10-12 15:50:10 +00:00
Neha Narkhede 3629f6f882 Changing kafka-server-stop.sh to use SIGTERM for Kafka server process termination. No review
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1397434 13f79535-47bb-0310-9956-ffa450edef68
2012-10-12 04:48:33 +00:00
Neha Narkhede dd44ec672a KAFKA-432 allow consumer to read from followers; patched by Yang Ye; reviewed by Neha and Jun
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1397422 13f79535-47bb-0310-9956-ffa450edef68
2012-10-12 02:56:28 +00:00
Neha Narkhede 93fdcfa0e2 KAFKA-42 part 2: new files
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1396814 13f79535-47bb-0310-9956-ffa450edef68
2012-10-10 21:43:44 +00:00
Neha Narkhede a1698e2b70 Reverting KAFKA-42 since it accidentally contained changes to metrics package. Didn't catch it due to a stale sbt cache. It is better to redo the patch and then commit
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1396726 13f79535-47bb-0310-9956-ffa450edef68
2012-10-10 18:42:57 +00:00
Neha Narkhede 6d0042b4f4 KAFKA-42 Cluster expansion feature; patched by Neha Narkhede; reviewed by Jun Rao
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1396713 13f79535-47bb-0310-9956-ffa450edef68
2012-10-10 18:03:07 +00:00
Edward Jay Kreps 623e2b7e53 KAFKA-424. Fix command line usage description. Patch from Tommie.
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1373056 13f79535-47bb-0310-9956-ffa450edef68
2012-08-14 19:59:11 +00:00
Edward Jay Kreps d482531810 KAFKA-425 Wrong class name in scripts. Patch from Akira Kitada reviewed by Jay Kreps.
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1373054 13f79535-47bb-0310-9956-ffa450edef68
2012-08-14 19:56:36 +00:00
Neha Narkhede d73355017a KAFKA-301 Implement broker startup procedure; patched by Neha Narkhede; reviewed by Jun Rao and Jay Kreps
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1329509 13f79535-47bb-0310-9956-ffa450edef68
2012-04-23 23:32:59 +00:00
Jun Rao e913a06533 trivial fix to make create topic script executable
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1231403 13f79535-47bb-0310-9956-ffa450edef68
2012-01-13 23:52:21 +00:00
Jun Rao ff668f4fc9 create/delete ZK path for a topic in an admin tool; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-237
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1231400 13f79535-47bb-0310-9956-ffa450edef68
2012-01-13 23:48:27 +00:00
Neha Narkhede 60d88a3ac9 KAFKA-202 Make request processing in kafka asynchronous; patched by jaykreps; reviewed by junrao and nehanarkhede
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/branches/0.8@1230845 13f79535-47bb-0310-9956-ffa450edef68
2012-01-13 00:03:02 +00:00
Neha Narkhede 336410a777 KAFKA-227 Broker failure system test; patched by johnfung; reviewed by nehanarkhede
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1227916 13f79535-47bb-0310-9956-ffa450edef68
2012-01-06 00:43:46 +00:00
Neha Narkhede f53eb16f87 KAFKA 222 Mavenize contrib;patched by nehanarkhede; reviewed jakobhoman and junrao
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1211616 13f79535-47bb-0310-9956-ffa450edef68
2011-12-07 20:14:15 +00:00
Edward Jay Kreps ce070548e4 KAFKA-165 Add a cli script to use zkCli without having to download ZK.
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1187983 13f79535-47bb-0310-9956-ffa450edef68
2011-10-23 22:03:49 +00:00
Neha Narkhede 4ad98872ad KAFKA-151 Standard .rat-excludes file and rat script;patched by nehanarkhede; reviewed by junrao
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1183190 13f79535-47bb-0310-9956-ffa450edef68
2011-10-14 01:30:12 +00:00
Neha Narkhede fa09aacdbd KAFKA-141 Check and make sure that the files that have been donated have been updated to reflect the new ASF copyright;patched by nehanarkhede; reviewd by jjkoshy
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1180185 13f79535-47bb-0310-9956-ffa450edef68
2011-10-07 19:51:28 +00:00
Neha Narkhede f34dfc002d Reverting accidental commit 1180110, which is pending review for KAFKA-141
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1180117 13f79535-47bb-0310-9956-ffa450edef68
2011-10-07 17:33:43 +00:00
Neha Narkhede a99becbe49 Minor patch, removing the .gitigore file
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1180110 13f79535-47bb-0310-9956-ffa450edef68
2011-10-07 17:18:56 +00:00
Neha Narkhede f7b2819bff reverting r1178669 to fix KAFKA-147 kafka integration tests fail on a fresh checkout; patched by nehanarkhede; reviewed by junrao
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1179487 13f79535-47bb-0310-9956-ffa450edef68
2011-10-06 01:57:38 +00:00
Chris Burroughs 4c2d121b26 Update to sbt 0.7.7
patch by cburroughs; reviewed by junrao for KAFKA-92

git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1178669 13f79535-47bb-0310-9956-ffa450edef68
2011-10-04 01:41:02 +00:00
Edward Jay Kreps 59a5c3215a KAFKA-130 Add a "console producer" that sends messages from standard input
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1178659 13f79535-47bb-0310-9956-ffa450edef68
2011-10-04 00:54:14 +00:00
Jun Rao 42c2241cfb kafka-server-start.sh ignores JMX_PORT; patched by Mathias Herberts; reviewed by Jun Rao; KAFKA-144
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1178434 13f79535-47bb-0310-9956-ffa450edef68
2011-10-03 15:49:36 +00:00
Jun Rao d7e0b11d87 fix kafka-server-start.sh to take an optional producer property
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1157560 13f79535-47bb-0310-9956-ffa450edef68
2011-08-14 16:42:35 +00:00
Edward Jay Kreps 642da2f28c Initial checkin of Kafka to Apache SVN. This corresponds to 709afe4ec7 except that git specific files have been removed and code has been put into trunk/branches/site/etc. This is just a copy of master, branches and history are not being converted since we can't find a good tool for it.
git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1152970 13f79535-47bb-0310-9956-ffa450edef68
2011-08-01 23:41:24 +00:00