Commit Graph

2582 Commits

Author SHA1 Message Date
Jason Gustafson 1fb0d796bb KAFKA-3659: Handle coordinator disconnects more gracefully in client
Author: Jason Gustafson <jason@confluent.io>

Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1322 from hachikuji/KAFKA-3659

(cherry picked from commit 32bf83e5a7)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
2016-05-05 12:03:51 -07:00
Liquan Pei e9d10108b4 KAFKA-3527: Consumer commitAsync should not expose internal exceptions
Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Grant Henke <granthenke@gmail.com>, Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1300 from Ishiihara/kafka-3527

(cherry picked from commit ad31650978)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
2016-05-04 19:21:15 -07:00
Rajini Sivaram 91130e4242 KAFKA-3652; Return error response for unsupported version of ApiVersionsRequest
Handle unsupported version of ApiVersionsRequest during SASL auth as well as normal operation by returning an error response.

Author: Rajini Sivaram <rajinisivaram@googlemail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>

Closes #1310 from rajinisivaram/KAFKA-3652

(cherry picked from commit 64451af9e0)
Signed-off-by: Jun Rao <junrao@gmail.com>
2016-05-04 18:16:19 -07:00
Guozhang Wang 21351b5755 KAFKA-3639; Configure default serdes upon construction
Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Michael G. Noll <michael@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk>

Closes #1311 from guozhangwang/K3639

(cherry picked from commit c8c6ac3f6d)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>
2016-05-05 00:23:53 +01:00
Liquan Pei 1587aeed9a KAFKA 3656: Remove logging outstanding messages when producer flush fails
Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1319 from Ishiihara/kafka-3656
2016-05-04 16:10:46 -07:00
William Thurston 946d074f0e KAFKA-2236; Offset request reply racing with segment rolling
Author: William Thurston <wthurston@linkedin.com>
Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Ismael Juma, Guozhang Wang

Closes #1318 from ijuma/KAFKA-2236-offset-request-reply-segment-rolling-race

(cherry picked from commit c46cc48021)
Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
2016-05-04 14:26:46 -07:00
Yuto Kawamura a0f40770ab KAFKA-3642: Fix NPE from ProcessorStateManager when the changelog topic not exists
Issue: https://issues.apache.org/jira/browse/KAFKA-3642

Author: Yuto Kawamura <kawamuray.dadada@gmail.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #1289 from kawamuray/KAFKA-3642-streams-NPE
2016-05-04 14:05:11 -07:00
Eno Thereska f4bb4a6515 MINOR: Added more integration tests
Author: Eno Thereska <eno.thereska@gmail.com>

Reviewers: Ismael Juma, Michael G. Noll, Guozhang Wang

Closes #1285 from enothereska/more-integration-tests
2016-05-04 14:04:20 -07:00
Guozhang Wang 9b6761ccd6 MINOR: Modify checkstyle to allow import classes only used in javadoc
Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Gwen Shapira, Ismael Juma

Closes #1317 from guozhangwang/KJavaDocImport
2016-05-04 13:59:38 -07:00
Jason Gustafson 21e930b4b4 KAFKA-3632; remove fetcher metrics on shutdown and leader migration
Author: Jason Gustafson <jason@confluent.io>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1312 from hachikuji/KAFKA-3632

(cherry picked from commit 7a0821d653)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>
2016-05-04 20:11:07 +01:00
Ewen Cheslack-Postava 41a8cb2520 MINOR: Handle null values in validators
Author: Ewen Cheslack-Postava <me@ewencp.org>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1316 from ewencp/minor-handle-null-values-validators

(cherry picked from commit 03a1f7d39c)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>
2016-05-04 02:27:16 +01:00
Liquan Pei 2d93696634 KAFKA-3654: Config validation should validate both common and connector specific configurations
Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1313 from Ishiihara/config-short-circuit

(cherry picked from commit 50aacd660d)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
2016-05-03 18:05:36 -07:00
Chen Zhu 138b37d62b KAFKA-3648; maxTimeToBlock in BufferPool.allocate should be enforced
`maxTimeToBlock` needs to be updated in each loop iteration. Also record waitTime before throwing `TimeoutException`

Author: Chen Zhu <amandazhu19620701@gmail.com>

Reviewers: Dong Lin <lindong28@gmail.com>, Ismael Juma <ismael@juma.me.uk>

Closes #1304 from zhuchen1018/KAFKA-3648

(cherry picked from commit 94e12a2e1f)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>
2016-05-04 00:18:29 +01:00
Liquan Pei 4de0ed7d52 MINOR: Clean up of ConsumerCoordinator and PartitionAssignor
Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>

Closes #1306 from Ishiihara/minor-consumer-cleanup

(cherry picked from commit af01378843)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>
2016-05-03 23:54:59 +01:00
Som Sahu 96f6aa8c4d KAFKA-3448; Support zone index in IPv6 regex
When an address is written textually, the zone index is appended to the address, separated by a percent sign (%). The actual syntax of zone indices depends on the operating system.

Author: Som Sahu <sosahu@microsoft.com>
Author: Soumyajit Sahu <soumyajit-sahu@users.noreply.github.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1305 from soumyajit-sahu/fixIPV6RegexPattern_trunk

(cherry picked from commit 88e5d5a9a1)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>
2016-05-03 19:24:34 +01:00
Ismael Juma e5363e1104 KAFKA-3128; Add metrics for ZooKeeper events zookeeper metrics
Also:
* Remove redundant `time.milliseconds` call in `Sensor.record`
* Clean-up a number of tests and remove a manual test that is no longer required

Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Liquan Pei <liquanpei@gmail.com>, Jun Rao <junrao@gmail.com>

Closes #1265 from ijuma/kafka-3128-zookeeper-metrics

(cherry picked from commit c7425be5be)
Signed-off-by: Jun Rao <junrao@gmail.com>
2016-05-03 10:56:04 -07:00
dan norwood e6d814a3eb MINOR: add logfilename to error message when file missing
ewencp

Author: dan norwood <norwood@confluent.io>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1307 from norwood/log-filename-when-missing-file

(cherry picked from commit 62253539d8)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
2016-05-03 10:22:24 -07:00
Rajini Sivaram 2f39fb0d15 KAFKA-3517; Add documentation for SASL/PLAIN
Documentation corresponding to KIP-43 - SASL/PLAIN and multiple mechanism support.

Author: Rajini Sivaram <rajinisivaram@googlemail.com>

Reviewers: Magnus Edenhill <apache_m@edenhill.se>, Jun Rao <junrao@gmail.com>

Closes #1232 from rajinisivaram/KAFKA-3517

(cherry picked from commit abac51dc0b)
Signed-off-by: Jun Rao <junrao@gmail.com>
2016-05-03 09:49:34 -07:00
Geoff Anderson 3cfe73a2ca MINOR: Add version check on enable-systest-events flag
Recent patch adding enable-systest-events flag without any version check breaks all uses of versioned console consumer. E.g. upgrade tests, compatibility tests etc.

Added a check to only apply the flag if running 0.10.0 or greater.

Author: Geoff Anderson <geoff@confluent.io>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1298 from granders/minor-systest-fix-versioned-console-consumer

(cherry picked from commit 5c47b9f80e)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
2016-05-02 21:42:19 -07:00
Dana Powers 548638af11 HOTFIX: Fix main classpath libs glob for release (fixup KAFKA-3615 regression)
bin/kafka-run-class.sh does not correctly setup the CLASSPATH in release rc2.

Author: Dana Powers <dana.powers@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1302 from dpkp/KAFKA-3615-fix

(cherry picked from commit 1e4dd66b19)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
2016-04-30 18:54:06 -07:00
Eno Thereska f38bba89e4 HOTFIX: Fix equality semantics of KeyValue
Fixes wrong KeyValue equals logic when keys not equal but values equal.

Original hotfix PR at https://github.com/apache/kafka/pull/1293 (/cc enothereska)

Please review: ewencp ijuma guozhangwang

Author: Eno Thereska <eno.thereska@gmail.com>
Author: Michael G. Noll <michael@confluent.io>

Reviewers: Michael G. Noll <michael@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1294 from miguno/KeyValue-equality-hotfix

(cherry picked from commit 60380e31d4)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
2016-04-29 15:15:07 -07:00
Gwen Shapira f0d9611866 Merge branch 'trunk' into 0.10.0 2016-04-29 14:50:50 -07:00
Liquan Pei d0dedc6314 KAFKA-3459: Returning zero task configurations from a connector does not properly clean up existing tasks
hachikuji ewencp Can you take a look when you have time?

Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1248 from Ishiihara/kafka-3459
2016-04-29 14:49:22 -07:00
Grant Henke 096b8b84e0 KAFKA-3641; Fix RecordMetadata constructor backward compatibility
Author: Grant Henke <granthenke@gmail.com>

Reviewers: Gwen Shapira, Ismael Juma

Closes #1292 from granthenke/recordmeta-compat
2016-04-29 14:48:25 -07:00
Matthias J. Sax 3414d56121 KAFKA-3440: Update streams javadocs
- add class doc for KTable, KStream, JoinWindows
- add missing return tags

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Guozhang Wang <wangguoz@gmail.com>, Michael G. Noll <michael@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1287 from mjsax/kafka-3440-JavaDoc
2016-04-29 12:50:02 -07:00
Rajini Sivaram 69d9a669d7 KAFKA-3618; Handle ApiVersionsRequest before SASL authentication
Server-side implementation and tests for handling ApiVersionsRequest before SaslHandshakeRequest.

Author: Rajini Sivaram <rajinisivaram@googlemail.com>

Reviewers: Gwen Shapira, Ismael Juma

Closes #1286 from rajinisivaram/KAFKA-3618
2016-04-29 11:15:20 -07:00
Liquan Pei e50327331d KAFKA-3578: Allow cross origin HTTP requests on all HTTP methods
Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1288 from Ishiihara/kip-56
2016-04-29 10:52:42 -07:00
Anna Povzner e29eac4bbb KAFKA-3597; Query ConsoleConsumer and VerifiableProducer if they shutdown cleanly
Even if a test calls stop() on console_consumer or verifiable_producer, it is still possible that producer/consumer will not shutdown cleanly, and will be killed forcefully after a timeout. It will be useful for some tests to know whether a clean shutdown happened or not. This PR adds methods to console_consumer and verifiable_producer to query whether clean shutdown happened or not.

hachikuji and/or granders Please review.

Author: Anna Povzner <anna@confluent.io>

Reviewers: Jason Gustafson, Geoff Anderson, Gwen Shapira

Closes #1278 from apovzner/kafka-3597
2016-04-29 10:51:29 -07:00
Liquan Pei eb50d2f6ca KAFKA-3615: Exclude test jars in kafka-run-class.sh
granders hachikuji Can you take a look when you have time? Appreciate your time to review.

Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Grant Henke <granthenke@gmail.com>, Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1263 from Ishiihara/classpath-no-test-jar
2016-04-29 10:29:00 -07:00
Jason Gustafson fe6c481b38 KAFKA-3418: add javadoc section describing consumer failure detection
Author: Jason Gustafson <jason@confluent.io>

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1129 from hachikuji/KAFKA-3418
2016-04-29 10:26:01 -07:00
Rajini Sivaram cea01af125 KAFKA-2693: Ducktape tests for SASL/PLAIN and multiple mechanisms
Run a sanity test with SASL/PLAIN and a couple of replication tests with SASL/PLAIN and multiple mechanisms.

Author: Rajini Sivaram <rajinisivaram@googlemail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1282 from rajinisivaram/KAFKA-2693
2016-04-29 09:41:12 -07:00
Matthias J. Sax 4ab4e4af81 KAFKA-3598: Improve JavaDoc of public API
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Michael G. Noll, Guozhang Wang

Closes #1250 from mjsax/JavaDoc-publicAPI
2016-04-29 08:49:16 -07:00
Michael G. Noll 68433dcfdc KAFKA-3613: Consolidate TumblingWindows and HoppingWindows into TimeWindows
This PR includes the same code as https://github.com/apache/kafka/pull/1261 but is rebased on latest trunk.

Author: Michael G. Noll <michael@confluent.io>

Reviewers: Matthias J. Sax, Guozhang Wang

Closes #1277 from miguno/KAFKA-3613-v2
2016-04-29 07:44:03 -07:00
Ashish Singh 0ada3b1fc2 KAFKA-3382: Add system test for ReplicationVerificationTool
Author: Ashish Singh <asingh@cloudera.com>

Reviewers: Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1160 from SinghAsDev/KAFKA-3382
2016-04-28 15:49:01 -07:00
Liquan Pei 346df72736 MINOR: Avoid compiler warnings when registering metrics in KafkaServer
Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1280 from Ishiihara/minor-warning-fix
2016-04-28 14:46:47 -07:00
Jason Gustafson fdded2da6e KAFKA-3636; Change default max session timeout to 5 minutes
Author: Jason Gustafson <jason@confluent.io>

Reviewers: Guozhang Wang <wangguoz@gmail.com>, Onur Karaman <okaraman@linkedin.com>, Ismael Juma <ismael@juma.me.uk>

Closes #1284 from hachikuji/KAFKA-3636
2016-04-28 14:08:10 -07:00
Rajini Sivaram 3a496f480d KAFKA-3617; Unit tests for SASL authenticator
Unit tests for SASL authenticator, tests for SASL/PLAIN and multiple mechanisms, authorization test for SASL/PLAIN

Author: Rajini Sivaram <rajinisivaram@googlemail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1273 from rajinisivaram/KAFKA-3617
2016-04-28 13:39:22 -07:00
Liquan Pei 316389d6ad KAFKA-3611: Remove warnings when using reflections
ewencp granders Can you take a look? Thanks!

Author: Liquan Pei <liquanpei@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #1259 from Ishiihara/fix-warning
2016-04-28 11:59:02 -07:00
Ashish Singh 57831a56a5 KAFKA-3621; Add tests for ApiVersionRequest/Response
Author: Ashish Singh <asingh@cloudera.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1275 from SinghAsDev/KAFKA-3621
2016-04-27 17:31:31 -07:00
Grant Henke 669be7fadc KAFKA-3631; Fix Struct.toString for nullable arrayOf
Author: Grant Henke <granthenke@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1279 from granthenke/struct-fix
2016-04-27 17:15:37 -07:00
Eno Thereska 94aee2143e KAFKA-3612: Added structure for integration tests
Author: Eno Thereska <eno.thereska@gmail.com>

Reviewers: Ismael Juma, Damian Guy, Michael G. Noll, Guozhang Wang

Closes #1260 from enothereska/KAFKA-3612-integration-tests
2016-04-27 16:55:51 -07:00
Ashish Singh 8407dac6ee KAFKA-3307; Add ApiVersions Request/Response and server side handling.
The patch does the following.
1. Adds ApiVersionsRequest/Response.
2. Adds UNSUPPORTED_VERSION error and UnsupportedVersionException.
3. Adds broker side handling of ApiVersionsRequest.

Author: Ashish Singh <asingh@cloudera.com>

Reviewers: Gwen Shapira, Ismael Juma, Magnus Edenhill

Closes #986 from SinghAsDev/KAFKA-3307
2016-04-27 11:28:32 -07:00
Damian Guy 4c76b5fa6a KAFKA-3629; KStreamImpl.to(...) throws NPE when the value SerDe is null
guozhangwang

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Matthias J. Sax, Guozhang Wang

Closes #1272 from dguy/kstreamimpl-to-npe and squashes the following commits:

49d48fb [Damian Guy] actually commit the fix
07ce589 [Damian Guy] fix npe in KStreamImpl.to(..)
74d396d [Damian Guy] fix npe in KStreamImpl.to(..)
2016-04-27 10:50:20 -07:00
Manikumar reddy O 2885bc33da KAFKA-3580; Improve error logging in ReplicaFetchThread
Author: Manikumar reddy O <manikumar.reddy@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1237 from omkreddy/KAFKA-3580
2016-04-27 06:53:16 -07:00
Manikumar reddy O 77fa0b116f KAFKA-3406; Update CommonClientConfigs.RETRY_BACKOFF_MS_DOC doc string
Author: Manikumar reddy O <manikumar.reddy@gmail.com>

Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>, Ismael Juma <ismael@juma.me.uk>

Closes #1230 from omkreddy/KAFKA-3406
2016-04-27 00:03:00 -07:00
Grant Henke 33d745e2dc KAFKA-3306: Change metadata response to include required additional fi…
…elds

- Adds boolean type to the protocol
- Allows protocol arrays to be null (optionally)
- Adds support to ask for no topics in the metadata request
- Adds new fields to the Metadata response protocol
- Adds server code to handle new fields
   - Support no-topic metadata requests
   - Track controller id in the metadata cache
   - Check if a topic is considered internal
   - Included rack information if present
   - Include all replicas and ISRs, even if node is down
- Adds test code to test new functionality independent of the client

Author: Grant Henke <granthenke@gmail.com>

Reviewers: Gwen Shapira, Ismael Juma, Ashish Singh

Closes #1095 from granthenke/metadata-changes
2016-04-26 17:03:18 -07:00
Rajini Sivaram 5b375d7bf9 KAFKA-3149; Extend SASL implementation to support more mechanisms
Code changes corresponding to KIP-43 to enable review of the KIP.

Author: Rajini Sivaram <rajinisivaram@googlemail.com>

Reviewers: Jun Rao <junrao@apache.org>, Ismael Juma <ismael@juma.me.uk>

Closes #812 from rajinisivaram/KAFKA-3149
2016-04-26 16:56:42 -07:00
Ismael Juma cd427c9b90 MINOR: Upgrade to Gradle 2.13
There are a few improvements in 2.12 and 2.13. I am particularly interested in the performance improvements:
* 2.12: "This release brings support for compile only dependencies, improved build script compilation speed and even better IDE support."
* 2.13: "We've achieved performance improvements during Gradle's configuration and execution phase, where we have measured up to 25% improvements to build time in our performance tests. No changes to your build script are necessary to start taking advantage of these improvements."

Author: Ismael Juma <ismael@juma.me.uk>

Reviewers: Gwen Shapira

Closes #1271 from ijuma/gradle-2.13
2016-04-26 15:14:33 -07:00
Rafael Winterhalter 2790d0cb1f MINOR: Changed port of bootstrap server to default
Kafka is typically running on port 9092. The example named a different port what makes it difficult to run a bootstrap example without any further configuration.

Author: Rafael Winterhalter <rafael.wth@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1074 from raphw/trunk
2016-04-26 11:52:33 -07:00
Stig fbac525c08 MINOR: Fix zk path in KafkaHealthCheck comment
Author: Stig <stig@linux-pr4q.suse>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #1182 from srdo/comment-fix
2016-04-26 11:46:03 -07:00