Commit Graph

534 Commits

Author SHA1 Message Date
Guozhang Wang 6a3382c9d3
MINOR: Streams Update for KIP-330 / KIP-356 (#5794)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-10-12 14:48:28 -07:00
Matthias J. Sax 349b5ad56c
KAFKA-4932: Update docs for KIP-206 (#5769)
Reviewers: Bill Bejeck <bill@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-10-10 12:09:09 -07:00
Lee Dongjin 741cb761c5 KAFKA-4514; Add Codec for ZStandard Compression (#2267)
This patch adds support for zstandard compression to Kafka as documented in KIP-110: https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression. 

Reviewers: Ivan Babrou <ibobrik@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
2018-10-09 17:13:33 -07:00
Manikumar Reddy O 15dbab0e35 KAFKA-3097: Update docs to mention PrincipalType "User" is case sensitive (#5734)
Reviewers: Jun Rao <junrao@gmail.com>
2018-10-09 11:52:56 -07:00
Bill Bejeck 862e09f994 HOTFIX: Fix broken links (#5676)
Reviewers: Joel Hamill <11722533+joel-hamill@users.noreply.github.com>, Guozhang Wang <wangguoz@gmail.com>
2018-10-03 18:27:15 -07:00
Guozhang Wang 7bd8ada8e2
MINOR: Docs on state store instantiation (#5698)
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-10-03 11:20:47 -07:00
Guozhang Wang 08dfab06e7
KAFKA-3514: Upgrade Documentation (#5714)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2018-09-30 12:01:11 -07:00
Lee Dongjin e7b6d65713 KAFKA-6620: Fix documentation about 'exactly_once'
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2018-09-28 14:41:50 -07:00
Attila Sasvari 5916db11f6 MINOR: Fix broken link in security.html (#5670)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
2018-09-26 10:20:11 +01:00
Manikumar Reddy b21c66f948 KAFKA-5690; Add support to list ACLs for a given principal (KIP-357)
Author: Manikumar Reddy <manikumar.reddy@gmail.com>

Reviewers: Dong Lin <lindong28@gmail.com>

Closes #5633 from omkreddy/KAFKA-5690-LIST-PER-PRICIPAL
2018-09-17 09:35:00 -07:00
Guozhang Wang 68b2f49ea7
KAFKA-3514, Documentations: Add out of ordering in concepts. (#5458)
Reviewers: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-09-11 16:28:52 -07:00
Manikumar Reddy O f348f10ef8 KAFKA-7117: Support AdminClient API in AclCommand (KIP-332) (#5463)
Reviewers: Colin Patrick McCabe <colin@cmccabe.xyz>, Jun Rao <junrao@gmail.com>
2018-09-07 17:40:59 -07:00
Ismael Juma 6a1bb547c0 KAFKA-7372: Upgrade Jetty for preliminary Java 11 and TLS 1.3 support (#5600)
"Jetty 9.4.12 includes compatibility for JDK 11. Additionally, TLS 1.3 support has been implemented. While full functionality for new JDK features is not yet supported, this release has been built and tested for compatibility with the latest releases from Oracle."

http://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00124.html

Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
2018-09-03 15:21:20 +01:00
huxi aa7358e8cc KAFKA-7326: KStream.print() should flush on each line for PrintStream (#5579)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>
2018-09-01 09:48:50 -07:00
lucapette 2514a42e84 KAFKA-7269: Add docs for KStream.merge (#5512)
Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-08-29 09:34:54 -07:00
Gantigmaa Selenge 42610e5aa1 MINOR: Fix broken link to record format in protocol docs (#5526)
Co-authored-by: Mickael Maison <mickael.maison@gmail.com>
Co-authored-by: Gantigmaa Selenge  <tina.selenge@gmail.com>

Reviewers: Sriharsha Chintalapani <sriharsha@apache.org>, Jason Gustafson <jason@confluent.io>
2018-08-24 14:03:58 -07:00
Alex Dunayevsky bf0675ac5f KAFKA-6343 Documentation update in OS-level tuning section: add vm.ma… (#4358)
Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
2018-08-22 10:13:10 -07:00
Manikumar Reddy O 49c5994dd9 KAFKA-7063; Remove references to old producers and consumers in docs (#5240)
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
2018-08-21 14:02:14 -07:00
Mayank Tankhiwale da35cd5114 KAFKA-6312; Document --reset-offsets option for consumer group tool (#4527)
KIP-122 added the ability for kafka-consumer-groups.sh to reset/change consumer offsets, at a fine grained level. This patch adds documentation for this feature.

Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
2018-08-20 13:27:44 -07:00
Manikumar Reddy O 028e80204d KAFKA-6835: Enable topic unclean leader election to be enabled without controller change (#4957)
Reviewers: Jun Rao <junrao@gmail.com>
2018-08-20 09:48:13 -07:00
Vahid Hashemian e6fd99dc62 KAFKA-5638; Improve the Required ACL of ListGroups API (KIP-231) (#5352)
Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
2018-08-12 23:59:31 -07:00
nprad 79a2f892ca KAFKA-6966: Extend TopologyDescription to better represent Source and (#5284)
Implements KIP-321

Reviewers: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-08-10 16:12:03 -07:00
Manikumar Reddy O 92004fa21a KAFKA-6751; Support dynamic configuration of max.connections.per.ip/max.connections.per.ip.overrides configs (KIP-308) (#5334)
KIP-308 implementation. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=85474993.

Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Jason Gustafson <jason@confluent.io>
2018-08-09 14:40:24 -07:00
okumin cc39e4aa49 KAFKA-7247: Update a link to Apache BookKeeper project (#5460)
Reviewers: Sriharsha Chintalapani <sriharsha@apache.org>
2018-08-08 09:51:19 -07:00
Arjun Satish e876c921b0 MINOR: Add connector configs to site-docs
In AK's documentation, the config props for connectors are not listed (https://kafka.apache.org/documentation/#connectconfigs). This PR adds these sink and source connector configs to the html site-docs.

Signed-off-by: Arjun Satish <arjunconfluent.io>

Author: Arjun Satish <arjun@confluent.io>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #5469 from wicknicks/add-connector-configs-to-docs
2018-08-07 14:34:27 -07:00
Kevin Lu fb8eba3d33 KAFKA-7207: Make rate & total metrics documentation consistent (#5429)
Some sections of the Monitoring metrics documentation list out the -total metrics, and some sections do not list them out. Make them consistent and list out the missing -total metrics.
2018-08-07 08:49:57 +01:00
Matthias J. Sax c1ca53b369 MINOR: fix Streams docs state.dir (#5465)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-08-06 13:39:50 -07:00
Stanislav Kozlovski 518e9d3eee KAFKA-7169: Custom SASL extensions for OAuthBearer authentication mechanism (KIP-342) (#5379)
Reviewers: Ron Dagostino <rndgstn@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
2018-08-06 17:22:04 +01:00
Simon Clark 530d951cbd MINOR: Fixed default streams state dir location. (#5441)
Co-authored-by: Mickael Maison <mickael.maison@gmail.com>
Co-authored-by: Simon Clark <simonc6r@gmail.com>

Reviewers: Sriharsha Chintalapani <sriharsha@apache.org>
2018-08-01 19:27:08 -07:00
Jason Gustafson c3e7c0bcb2
MINOR: Producers should set delivery timeout instead of retries (#5425)
Use delivery timeout instead of retries when possible and remove various TODOs associated with completion of KIP-91.

Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
2018-08-01 11:04:17 -07:00
Yu Yang 7fc7136ffd KAFKA-5886; Introduce delivery.timeout.ms producer config (KIP-91) (#5270)
Co-authored-by: Sumant Tambe <sutambe@yahoo.com>
Co-authored-by: Yu Yang <yuyang@pinterest.com>

Reviewers: Ted Yu <yuzhihong@gmail.com>, Apurva Mehta <apurva@confluent.io>, Jason Gustafson <jason@confluent.io>
2018-07-26 09:13:50 -07:00
Rajini Sivaram 4b60ed3247
KAFKA-7193: Use ZooKeeper IP address in streams tests to avoid timeouts (#5414)
ZooKeeper client from version 3.4.13 doesn't handle connections to localhost very well. If ZooKeeper is started on 127.0.0.1 on a machine that has both ipv4 and ipv6 and a client is created using localhost rather than the IP address in the connection string, ZooKeeper client attempts to connect to ipv4 or ipv6 randomly with a fixed one second backoff if connection fails. Use 127.0.0.1 instead of localhost in streams tests to avoid intermittent test failures due to ZK client connection timeouts if ipv6 is chosen in consecutive address selections. Also add note to upgrade docs for 2.0.0.

Reviewers: Ismael Juma <github@juma.me.uk>, Matthias J. Sax <matthias@confluent.io>
2018-07-23 19:10:33 +01:00
Jon Lee f9cdadfdcf KAFKA-7177; Update 2.0 documentation to reflect changed quota behaviors by KIP-219
Updated the 2.0 document for changed quota behaviors.

Author: Jon Lee <jonlee@linkedin.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Dong Lin <lindong28@gmail.com>

Closes #5384 from jonlee2/KAFKA-7177
2018-07-21 00:04:16 -07:00
Matthias J. Sax c5a4d5cb91 MINOR: fix upgrade docs for Streams (#5394)
Reviewers: Guozhang Wang <guozhang@confluent.io>, Rajini Sivaram <rajini@confluent.io>
2018-07-20 11:01:32 -07:00
Matthias J. Sax 03e788b29b MINOR: improve docs version numbers (#5372)
Reviewer: Guozhang Wang <guozhang@confluent.io>
2018-07-18 13:52:44 -07:00
Rajini Sivaram 359eb4e083
MINOR: Remove references to version 1.2 in docs (#5386) 2018-07-18 18:07:59 +01:00
Mickael Maison 28e29a4c61 MINOR: Updated Quickstart to mention log.dirs (#5361)
The default server.properties file now contains the log.dirs setting and not log.dir anymore.

Co-authored-by: Mickael Maison <mickael.maison@gmail.com>
Co-authored-by: Katherine Farmer <kfarme3@uk.ibm.com>
Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Sriharsha Chintalapani <sriharsha@apache.org>
2018-07-17 22:24:08 -07:00
Guozhang Wang 2d877631cd
MINOR: web docs fixes on message header (#5381)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2018-07-17 17:31:32 -07:00
Guozhang Wang 6bfaf4dc60
MINOR: Store metrics scope, total metrics (#5290)
1. Rename metrics scope of rocksDB window and session stores; also modify the store metrics accordingly with guidance on its correlations to metricsScope.

2. Add the missing total metrics for per-thread, per-task, per-node and per-store sensors.


Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-06-28 14:32:02 -07:00
Guozhang Wang 7ea8a25a4c HOTFIX: remove old security suggestions 2018-06-27 09:35:34 -07:00
Guozhang Wang a20555102b HOTFIX: update Streams security docs 2018-06-26 18:53:06 -07:00
Jim Galasyn 581adb5013 MINOR: Add note about num.standby.replicas (#5271)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-06-26 08:24:44 -07:00
Vahid Hashemian 9bc9a37e50 MINOR: KIP-211 Follow-up (#5272)
Updates the description of `offsets.retention.minutes` config, and fixes an upgrade note.
2018-06-26 07:57:28 -07:00
Vahid Hashemian c32f7e6ad8 MINOR: Fix expected output in Streams quickstart (#5280)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-06-22 17:18:10 -07:00
Vahid Hashemian 418a91b5d4 KAFKA-4682; Revise expiration semantics of consumer group offsets (KIP-211 - Part 1) (#4896)
This patch contains the improved offset expiration semantics proposed in KIP-211. Committed offsets will not be expired as long as a group is active. Once all members have left the group, then offsets will be expired after the timeout configured by `offsets.retention.minutes`. Note that the optimization for early expiration of unsubscribed topics will be implemented in a separate patch.
2018-06-21 17:19:24 -07:00
Guozhang Wang d3e264e773
MINOR: update web docs and examples of Streams with Java8 syntax (#5249)
Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>, Damian Guy <damian@confluent.io>
2018-06-21 10:02:58 -07:00
Rajini Sivaram bb68b9c046
MINOR: Update version for doc to 2.0.0 (#5262)
Reviewers: Damian Guy <damian.guy@gmail.com>
2018-06-21 11:18:31 +01:00
Guozhang Wang f8cb3c3e1d
MINOR: add headers support in new api (#5252)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2018-06-19 14:23:04 -07:00
Ismael Juma cc4dce94af
KAFKA-2983: Remove Scala consumers and related code (#5230)
- Removed Scala consumers (`SimpleConsumer` and `ZooKeeperConsumerConnector`)
and their tests.
- Removed Scala request/response/message classes.
- Removed any mention of new consumer or new producer in the code
with the exception of MirrorMaker where the new.consumer option was
never deprecated so we have to keep it for now. The non-code
documentation has not been updated either, that will be done
separately.
- Removed a number of tools that only made sense in the context
of the Scala consumers (see upgrade notes).
- Updated some tools that worked with both Scala and Java consumers
so that they only support the latter (see upgrade notes).
- Removed `BaseConsumer` and related classes apart from `BaseRecord`
which is used in `MirrorMakerMessageHandler`. The latter is a pluggable
interface so effectively public API.
- Removed `ZkUtils` methods that were only used by the old consumers.
- Removed `ZkUtils.registerBroker` and `ZKCheckedEphemeral` since
the broker now uses the methods in `KafkaZkClient` and no-one else
should be using that method.
- Updated system tests so that they don't use the Scala consumers except
for multi-version tests.
- Updated LogDirFailureTest so that the consumer offsets topic would
continue to be available after all the failures. This was necessary for it
to work with the Java consumer.
- Some multi-version system tests had not been updated to include
recently released Kafka versions, fixed it.
- Updated findBugs and checkstyle configs not to refer to deleted
classes and packages.

Reviewers: Dong Lin <lindong28@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
2018-06-19 07:32:54 -07:00
Rajini Sivaram d504e85011
KAFKA-6805: Enable broker configs to be stored in ZK before broker start (#4898)
Support configuration of dynamic broker configs in ZooKeeper before starting brokers using ConfigCommand. This will allow password configs to be encrypted and stored in ZooKeeper, without requiring clear passwords in server.properties to bootstrap the broker first.

Reviewers: Jason Gustafson <jason@confluent.io>
2018-06-18 18:28:08 +01:00