Adds new metrics to support health checks:
1. Error rates for each request type, per-error code
2. Request size and temporary memory size
3. Message conversion rate and time
4. Successful and failed authentication rates
5. ZooKeeper latency and status
6. Client version
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3705 from rajinisivaram/KAFKA-5746-new-metrics
This is the polished version.
1. The old merge() method in StreamsBuilder has been removed,
2. The merge() method in KStreamBuilder was changed so that it would use the single variable argument
rather than several variable arguments in the KStreamImpl implementation
3. The merge() method in KStream has been declared as final and tests have been added to test correctness.
Author: Richard Yu <richardyu@Richards-Air.attlocal.net>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3916 from ConcurrencyPractitioner/trunk
Make documentation consistent across methods and throw
IllegalStateException instead of IllegalArgumentException in
some cases.
Also include a couple of minor fixes in upgrade.html.
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3781 from lindong28/minor-admin-client-comment
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3949 from mjsax/minor-update-streams-quickstart
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3036 from mimaison/KAFKA-3356
Author: Manjula K <manjula@kafka-summit.org>
Author: manjuapu <manjula@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3950 from manjuapu/customer-logo-stream
Author: Manjula K <manjula@kafka-summit.org>
Author: manjuapu <manjula@confluent.io>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3940 from manjuapu/customer-logo-stream
Developed with edoardocomar
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Edoardo Comar <ecomar@uk.ibm.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#3906 from mimaison/KAFKA-5735
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3921 from dguy/minor-kip-173-docs
Author: Florian Hussonnois <florian.hussonnois@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3811 from fhussonnois/KAFKA-5839
minor fixes
I found several minor issues with the Kafka Streams tutorial:
* Some typos
* "As shown above, it illustrate that the constructed ..." instead of "As shown above, it illustrate_s_ that the constructed ..."
* "same as Pipe.java below" instead of "same as Pipe.java _above_"
* Wrong class name in the `LineSplit` example
* Incorrect imports for the code examples
* Missing `import org.apache.kafka.streams.kstream.KStream;` in `LineSplit` and `WordCount` example
* Unnecessary (and potentially confusing) split by whitespaces in the `WorkCount` class (the split into words happened already in `LineSplit`)
Author: Jakub Scholz <www@scholzj.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3883 from scholzj/stream-tutorial-typos
Add overloads of `filter`, `filterNot`, `mapValues` that take `Materialized` as a param to `KTable`. Deprecate overloads using `storeName` and `storeSupplier`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3807 from dguy/ktable-filter-map
Part of KIP-182
- Add `Printed` class and `KStream#print(Printed)`
- deprecate all other `print` and `writeAsText` methods
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3768 from dguy/kafka-5652-printed
Add the `WindowedKStream` interface and implementation of methods that don't require `Materialized`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3809 from dguy/kgrouped-stream-windowed-by
Part of KIP-182
- Add `StateStoreBuilder` interface and `WindowStateStoreBuilder`, `KeyValueStateStoreBuilder`, and `SessionStateStoreBuilder` implementations
- Add `StoreSupplier`, `WindowBytesStoreSupplier`, `KeyValueBytesStoreSupplier`, `SessionBytesStoreSupplier` interfaces and implementations
- Add new methods to `Stores` to create the newly added `StoreSupplier` and `StateStoreBuilder` implementations
- Update `Topology` and `InternalTopology` to use the interfaces
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3767 from dguy/kafka-5650
Add the `Produced` class and `KStream` overloads that use it:
`KStream#to(String, Produced)`
`KStream#through(String, Produced)`
Deprecate all other to and through methods accept the single param methods that take a topic param
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3770 from dguy/kafka-5652-produced
The iterator interface usage has some examples missing explicit close operation after usage. We should remind the user to do so because un-closed iterator will leave the underlying file descriptor open, thus eating up memory.
guozhangwang Ishiihara
Author: Boyang Chen <bychen@pinterest.com>
Author: Boyang Chen <bchen11@outlook.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3714 from abbccdda/add_iterator_close_on_doc
minor fixes
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3796 from guozhangwang/kip-138-minor-renames
Part of KIP-182
- Add the `Serialized` class
- implement overloads of `KStream#groupByKey` and KStream#groupBy`
- deprecate existing methods that have more than default arguments
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3772 from dguy/kafka-5817
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Sharon Liu <sharonliu.cup@gmail.com>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3787 from mjsax/kafka-5823-kip120-docs
Added a simple table of contents for the developer section.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3760 from enothereska/minor-docs-toc
1. Core concepts (added the stream time definition), upgrade guide and developer guide.
2. Related Java docs changes.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3732 from guozhangwang/KMinor-kip138-docs
1. Add upgrade section for 1.0.0, including Streams API changes section.
2. Add metrics name changes section.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3687 from guozhangwang/KMinor-metrics-upgrade-guide
Needs to come after https://github.com/apache/kafka/pull/3701
Originally reviewed as part of #3490.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3711 from enothereska/minor-docs-stateless
Originally reviewed as part of https://github.com/apache/kafka/pull/3490.
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3701 from enothereska/minor-docs-create-source-streams
0. Minor fixes on the existing examples to merge all on a single input topic; also do not use `common.utils.Exit` as it is for internal usage only.
1. Add the archetype project for the quickstart. Steps to try it out:
a. `mvn install` on the quickstart directory.
b. `mvn archetype:generate \
-DarchetypeGroupId=org.apache.kafka \
-DarchetypeArtifactId=streams-quickstart-java \
-DarchetypeVersion=1.0.0-SNAPSHOT \
-DgroupId=streams-quickstart \
-DartifactId=streams-quickstart \
-Dversion=0.1 \
-Dpackage=StreamsQuickstart \
-DinteractiveMode=false` at any directory to create the project.
c. build the streams jar with version `1.0.0-SNAPSHOT` to local maven repository with `./gradlew installAll`; `cd streams-quickstart; mvn clean package`
d. create the input / output topics, start the console producer and consumer.
e. start the program: `mvn exec:java -Dexec.mainClass=StreamsQuickstart.Pipe/LineSplit/WordCount`.
f. type data on console producer and observe data on console consumer.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Ewen Cheslack-Postava <me@ewencp.org>, Eno Thereska <eno.thereska@gmail.com>
Closes#3630 from guozhangwang/KMinor-streams-quickstart-tutorial
Author: Bill Bejeck <bill@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3629 from bbejeck/docs-updates-for-kip-167
These handlers were previously used on the broker to handle uncaught exceptions, but now the broker users the new Java request objects exclusively.
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3646 from hachikuji/remove-old-request-error-handlers
This is important because Bits.reserveMemory calls System.gc() hoping to free native
memory in order to avoid throwing an OutOfMemoryException. This call is currently
a no-op due to -XX:+DisableExplicitGC.
It's worth mentioning that -XX:MaxDirectMemorySize can be used to increase the
amount of native memory available for allocation of direct byte buffers.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Apurva Mehta <apurva@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#3371 from ijuma/kafka-5470-explicit-gc-invokes-concurrent
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Eno Thereska <eno.thereska@gmail.com>
Closes#3604 from dguy/memory-management-docs
A couple of fixes to metric names to match the KIP
- Removed extra strings in the metric names that are already in the tags
- add a separate metric for "all"
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3491 from enothereska/hotfix-metric-names
1. Remove rest deprecation warnings in streams:jar.
2. Consolidate all unit test classes' reflections to access internal topology builder from packages other than `o.a.k.streams`. We need to refactor the hierarchies of StreamTask, StreamThread and KafkaStreams to remove these hacky reflections.
3. Minor fixes such as reference path, etc.
4. Minor edits on web docs for the describe function under developer-guide.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>
Closes#3603 from guozhangwang/K5671-followup-comments
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3602 from mjsax/kafka-5671-add-streamsbuilder
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3590 from mjsax/kafka-3856-replace-topology-builder-by-topology
1. Make the WordCountDemo application to not stop automatically but via "ctrl-C".
2. Update the quickstart html file to let users type input messages one-by-one, and observe added output in an interactive manner.
3. Some minor fixes on the parent documentation page pointing to streams sub-pages, added a new recommended Scala version number.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>
Closes#3515 from guozhangwang/KMinor-interactive-quickstart
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Onur Karaman <okaraman@linkedin.com>
Closes#2929 from lindong28/KAFKA-4763
Content and assets for the updated Streams API landing page
Author: Derrick Or <derrickor@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3540 from derrickdoo/streams-landing-page
Allow users to specify default topic configs for streams internal topics by supplying properties from `TopicConfig` with a prefix.
Supplied defaults are used when creating the internal topics. They are overridden by the configs supplied along with the `InternalTopicConfig`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3459 from dguy/kafka-3741
For topics that support timestamp search, if no offset is found for a partition, the partition should still be included in the result with a `null` offset value. This `KafkaConsumer` method currently excludes such partitions from the result.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#3460 from vahidhashemian/KAFKA-5534
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Jeff Klukas <jeff@klukas.net>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3249 from jklukas/docs-no-more-yammer-metrics-in-client
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3415 from vahidhashemian/doc/make_topic_config_a_section
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3510 from guozhangwang/KHotfix-streams-quickstart
Author: Eno Thereska <eno.thereska@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#3494 from enothereska/minor-quickstart-docs
rename section name and bold font for section names
Also fix quickstart.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3501 from ijuma/introduce-full-version-variable-in-docs
Operations documentation should guide user to employ `kafka-configs.sh` to add/remove configs for a topic.
Author: huxihx <huxi_2b@hotmail.com>
Reviewers: Damian Guy <damian.guy@gmail.com>
Closes#3429 from huxihx/KAFKA-5508_Doc_for_altering_topics
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Closes#3484 from dguy/iq-doc
Also fixed a bunch of broken links (details can be found in 34f8ecea0d)
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3473 from guozhangwang/KMinor-streams-doc-breakdown
Implementation for KIP-138: Change punctuate semantics
Author: Michal Borowiecki <michal.borowiecki@openbet.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bbejeck@gmail.com>, Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3055 from mihbor/KIP-138
It describes the old deprecated clients and it's better to just
remove it.
The contribution is my original work and I license the work to the
project under the project's open source license.
Author: Tom Bentley <tbentley@redhat.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#3385 from tombentley/KAFKA-4059
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>
Closes#3378 from mjsax/minor-producer-naming
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#3388 from hachikuji/KAFKA-5021
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Apurva Mehta <apurva@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3345 from guozhangwang/KMinor-streams-eos-docs
based on conversations with vahidhashemian rajinisivaram apurvam
The docs didn't make clear that what gets committed and what gets not may depend on the producer acks.
Author: Edoardo Comar <ecomar@uk.ibm.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Apurva Mehta <apurva@confluent.io>, Jason Gustafson <jason@confluent.io>
Closes#3035 from edoardocomar/DOC-clarification-on-committed
This is contributed by Hao Chen.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3366 from guozhangwang/KMinor-fix-streams-link-in-uses
“ as opposed to " don't render consistently across browsers. On current Kafka website they render correctly in Firefox but not Chrome (“runsâ€) - charset issue?
Author: Michal Borowiecki <michal.borowiecki@openbet.com>
Author: mihbor <mbor81@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#3227 from mihbor/patch-7
- Show AdminClient configs in the docs.
- Update Javadoc config so that public classes exposed by
the AdminClient are included.
- Version and table of contents fixes.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Colin Mccabe, Gwen Shapira
Closes#3271 from ijuma/kafka-5411-admin-client-javadoc-configs
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Derrick Or <derrickor@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#3214 from guozhangwang/KMinor-doc-java-brush
Add a section in the streams docs about the broker config introduced in KIP-134
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll, Bill Bejeck, Guozhang Wang
Closes#3179 from dguy/kip134-doc
We had originally increased Snappy’s block size as part of KAFKA-3704. However,
we had some issues with excessive memory usage in the producer and we reverted
it in 7c6ee8d5e.
After more investigation, we fixed the underlying reason why memory usage seemed
to grow much more than expected via KAFKA-3747 (included in 0.10.0.1).
In 0.10.2, we changed the broker to use the same classes as the producer and the
broker’s block size for Snappy was changed from 32 KB to 1KB. As reported in
KAFKA-5236, the on disk size is, in some cases, 50% larger when the data is compressed
with 1 KB instead of 32 KB as the block size.
As discussed in KAFKA-3704, it may be worth making this configurable and/or allocate
the compression buffers from the producer pool. However, for 0.11.0.0, I think the
simplest thing to do is to default to 32 KB for Snappy (the default if no block size
is provided).
I also increased the Gzip buffer size. 1 KB is too small and the default is smaller
still (512 bytes). 8 KB (which is the default buffer size for BufferedOutputStream)
seemed like a reasonable default.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#3205 from ijuma/kafka-5236-snappy-block-size
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma, Ewen Chesklack-Postava, Bill Bejeck, Guozhang Wang
Closes#3194 from mjsax/minor-update-docs-for-kip-123
Add a new entry in upgrade.html for `group.initial.rebalance.delay.ms`
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#3189 from dguy/relabance-delay
Adjust "importance level" and add explanation to the docs.
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Eno Thereska <eno.thereska@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Closes#2855 from mjsax/minor-improve-streams-config-parameters
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#3169 from rajinisivaram/MINOR-producer-metrics
KAFKA-4603 the command parsed error
Using "new OptionParser" might result in parse error
Change all the OptionParser constructor in Kafka into "new OptionParser(false)"
Author: xinlihua <xin.lihua1@zte.com.cn>
Author: unknown <00067310@A23338408.zte.intra>
Author: auroraxlh <xin.lihua1@zte.com.cn>
Author: xin <xin.lihua1@zte.com.cn>
Reviewers: Damian Guy, Guozhang Wang
Closes#2349 from auroraxlh/fix_OptionParser_bug
Autogenerate docs for the Consumer Fetcher's metrics. This is a smaller subset of the original PR https://github.com/apache/kafka/pull/1202.
CC ijuma benstopford hachikuji
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
Closes#2993 from wushujames/fetcher_metrics_docs
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Apurva Mehta <apurva@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#3123 from hachikuji/KAFKA-4935
Update upgrade.html
Raising this now, as KIP-118 is pulled from release as such submitting this without java 8 changes.
As per remaining review comment from https://github.com/apache/kafka/pull/2772, updating the upgrade notes.
Author: Michael André Pearce <michael.andre.pearce@me.com>
Author: Michael Andre Pearce <Michael.Andre.Pearce@me.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2991 from michaelandrepearce/KIP-82
This only removes deprecated methods,
fields and constructors in a small number of classes.
Deprecated producer configs is tracked via KAFKA-3353
and the old clients and related (tools, etc.) won't
be removed in 0.11.0.0.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2995 from ijuma/kafka-3763-remove-deprecated-0.11
These configs have been deprecated since 0.9.0.0:
block.on.buffer.full, metadata.fetch.timeout.ms and timeout.ms
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2987 from ijuma/kafka-3353-remove-deprecated-producer-configs
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Ewen Cheslack-Postava <me@ewencp.org>, Ismael Juma <ismael@juma.me.uk>
Closes#2958 from cmccabe/KAFKA-5162
Add a default log retention policy to keep GC logs from growing too large
Author: Ryan P <ryan.n.pridgeon@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1431 from rnpridgeon/KAFKA-3754
Introduced in PR #2824. Already fixed in the
website github.
Author: Gwen Shapira <cshapi@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#2844 from gwenshap/docs-hotfix
(cherry picked from commit 5f728532ac)
Signed-off-by: Ismael Juma <ismael@juma.me.uk>
Author: Dong Lin <lindong28@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com>
Closes#2476 from lindong28/KAFKA-4586
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Jun Rao <junrao@gmail.com>, Apurva Mehta <apurva@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2614 from hachikuji/exactly-once-message-format
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll, Eno Thereska, Matthias J. Sax, Elias Levy, Guozhang Wang
Closes#2725 from dguy/minor-processor-java-doc
KAFKA-4567 - Connect Producer and Consumer ignore ssl parameters configured for worker
Added brief explanation to the docs about parameter inheritance of Kafka consumers and producers from the worker config.
Author: Sönke Liebau <soenke.liebau@opencore.com>
Reviewers: Gwen Shapira
Closes#2511 from soenkeliebau/KAFKA-4567
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2544 from becketqin/KAFKA-4340_follow_up
Reverting some of the recent changes to quickstart doc. Further explanation is provided inline.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2539 from vahidhashemian/doc/fix_quickstart_issues
Update streams.html with GlobalKTable docs
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Michael G. Noll, Matthias J. Sax, Guozhang Wang
Closes#2516 from dguy/global-tables-doc
1. Added an architecture section.
2. Added a configuration / execution sub-section to developer guide.
Minor tweaks and a bunch of missing fixes from `kafka-site` repo.
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Derrick Or <derrickor@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2488 from guozhangwang/KMinor-streams-docs-second-pass
Author: Grant Henke <ghenke@cloudera.com>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2246 from granthenke/truststore-password
1. Update value for queued.max.requests to 500
2. Removed invalid config 'controller.message.queue.size'
3. Removed flush configs including 'log.flush.interval.messages', 'log.flush.interval.ms' and 'log.flush.scheduler.interval.ms'
Author: huxi <huxi@zhenrongbao.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#2490 from amethystic/kafka4727_server_config_doc_update
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Ismael Juma, Will Marshall, Damian Guy, Guozhang Wang, Michael G. Noll
Closes#2461 from mjsax/addStreamsUpdateSecton
Kafka brokers have a config called "offsets.topic.replication.factor" that specify the replication factor for the "__consumer_offsets" topic. The problem is that this config isn't being enforced. If an attempt to create the internal topic is made when there are fewer brokers than "offsets.topic.replication.factor", the topic ends up getting created anyway with the current number of live brokers. The current behavior is pretty surprising when you have clients or tooling running as the cluster is getting setup. Even if your cluster ends up being huge, you'll find out much later that __consumer_offsets was setup with no replication.
The cluster not meeting the "offsets.topic.replication.factor" requirement on the internal topic is another way of saying the cluster isn't fully setup yet.
The right behavior should be for "offsets.topic.replication.factor" to be enforced. Topic creation of the internal topic should fail with GROUP_COORDINATOR_NOT_AVAILABLE until the "offsets.topic.replication.factor" requirement is met. This closely resembles the behavior of regular topic creation when the requested replication factor exceeds the current size of the cluster, as the request fails with error INVALID_REPLICATION_FACTOR.
Author: Onur Karaman <okaraman@linkedin.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2177 from onurkaraman/KAFKA-3959
Author: Colin P. Mccabe <cmccabe@confluent.io>
Reviewers: Gwen Shapira <cshapi@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2387 from cmccabe/KAFKA-4664
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Michael G. Noll
Closes#2445 from ijuma/kafka-4578-upgrade-notes-0.10.2
This was missing from [an earlier PR](https://github.com/apache/kafka/pull/2247) that escaped these symbols in another section of the doc.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2453 from vahidhashemian/doc/escape_lt_gt_in_streams_code
Updated the docs with changes in KAFKA-4060.
Author: Hojjat Jafarpour <hojjat@Hojjat-Jafarpours-MBP.local>
Author: Hojjat Jafarpour <hojjat@HojjatJpoursMBP.attlocal.net>
Reviewers: Ismael Juma, Matthias J. Sax, Guozhang Wang
Closes#2377 from hjafarpour/KAFKA-4060-docs-update
Renames `HoistToStruct` SMT to `HoistField`.
Adds the following SMTs:
`ExtractField`
`MaskField`
`RegexRouter`
`ReplaceField`
`SetSchemaMetadata`
`ValueToKey`
Adds HTML doc generation and updates to `connect.html`.
Author: Shikhar Bhushan <shikhar@confluent.io>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#2374 from shikhar/more-smt
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#2366 from rajinisivaram/KAFKA-4626
This makes it consistent with MirrorMaker with the old consumer.
Author: huxi <huxi@zhenrongbao.com>
Author: amethystic <huxi_2b@hotmail.com>
Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk>
Closes#2072 from amethystic/kafka-4351_Regex_behavior_change_for_new_consumer
4 release cycles (0.9.0.0, 0.10.0.0, 0.10.1.0, 0.10.2.0) should be enough
to remove the beta label.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2286 from ijuma/kafka-3284-security-beta-label
Based on: https://github.com/apache/kafka-site/pull/27
I recommend also merging into 10.1.0.0 branch to avoid mishaps when re-publishing docs.
Author: Gwen Shapira <cshapi@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2077 from gwenshap/protocol_doc_fix
- Seperate Streams documentation out to a standalone page.
- Setup templates to use handlebars.js
- Create template variables to swap in frequently updated values like version number from a single file templateData.js
Author: Derrick Or <derrickor@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#2245 from derrickdoo/docTemplates
Performed minor cleanup and escaped `<` and `>` so code examples are shown correctly in the browser.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Matthias J. Sax, Guozhang Wang
Closes#2247 from vahidhashemian/doc/fix_streams_doc
Author: Matthias J. Sax <matthias@confluent.io>
Reviewers: Michael G. Noll, Eno Thereska, Damian Guy, Guozhang Wang
Closes#2117 from mjsax/kafka-4393-improveInvalidTsHandling
Removed a duplicate line and also cleaned up some of the language around compaction guarantees.
Author: Apurva Mehta <apurva.1618@gmail.com>
Reviewers: Gwen Shapira
Closes#2089 from apurvam/fix-documentation-of-compaction and squashes the following commits:
03c5bdd [Apurva Mehta] Fix line length to be consistent with the rest of the file
0af1a86 [Apurva Mehta] MINOR: fix duplicate line in docs for compaction.
Removed a non-existing reference in table of contents and fixed some section numbers.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2004 from vahidhashemian/doc/fix_toc
And improve readability by adding proper punctuations.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2002 from vahidhashemian/doc/fix_typos
Make the latest version of our docs follow the latest site template structure.
Author: Derrick Or <derrickor@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#2003 from derrickdoo/docs-updates
Author: Jakub Dziworski <jakub.dziworski@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1996 from JakubDziworski/doc_fix_streams_link
Author: Jakub Dziworski <jakub.dziworski@gmail.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1994 from JakubDziworski/doc_fix
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1990 from vahidhashemian/doc/quickstart_update_windows
This is a minor change to fix the most glaring issues. We have another JIRA to revamp the upgrade docs.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jeff Klukas <jeff@klukas.net>, Jiangjie Qin <becket.qin@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1971 from ijuma/kafka-4257-upgrade-docs-inconsitencies
In doc it stays:
_"Our topic is divided into a set of totally ordered partitions, each of which is consumed by one consumer at any given time."_
And consumer is described as:
_"We'll call **processes** that subscribe to topics and process the feed of published messages **consumers**."_
Which might lead to a wrong conclusion - that each partition can be read by one process at any given time.
I think this statements misses information about **consumer groups**, so i propose:
_"Our topic is divided into a set of totally ordered partitions, each of which is consumed by exactly one consumer (from each subscribed consumer groups) at any given time"_
This contribution is my original work and I license the work to the project under the project's open source license.
Author: pilo <jakub.pilimon@4finance.com>
Reviewers: Jiangjie Qin <becket.qin@gmail.com>, Jason Gustafson <jason@confluent.io>
Closes#1900 from pilloPl/minor/doc-fix
Reopening of https://github.com/apache/kafka/pull/1428
Author: Edoardo Comar <ecomar@uk.ibm.com>
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#1908 from edoardocomar/KAFKA-3396
This is to imply that the Java consumer/producer are the recommended consumer/producer now.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1921 from vahidhashemian/KAFKA-3697
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#1897 from becketqin/KAFKA-4194
- Updated implementation docs with details on Cluster Id generation.
- Mention cluster id in "noteworthy changes for 0.10.1.0" in upgrade docs.
Author: Sumit Arrawatia <sumit.arrawatia@gmail.com>
Author: arrawatia <sumit.arrawatia@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1895 from arrawatia/kip-78-docs
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
Closes#1852 from becketqin/KAFKA-4148
Include a few clean-ups (also in producer section), mention deprecation plans and reorder so that the new consumer documentation is before the old consumers.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jason Gustafson <jason@confluent.io>
Closes#1880 from ijuma/remove-beta-from-new-consumer-documentation
Details in KIP-55.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1847 from rajinisivaram/KAFKA-4079
Now uses LogSegment.largestTimestamp to determine age of segment's messages.
Author: Eric Wasserman <eric.wasserman@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#1794 from ewasserman/feat-1981
Update documentation for Kafka Connect distributed’s config.storage.topic, offset.storage.topic, and status.storage.topic configuration values to indicate that all three should refer to compacted topics.
Author: Mathieu Fenniak <mathieu.fenniak@replicon.com>
Reviewers: Jason Gustafson
Closes#1832 from mfenniak/kafka-connect-topic-docs
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1809 from becketqin/KAFKA-4099
Rephrase 'alpha quality' wording in Streams section of api.html.
Couple of other minor fixes in streams.html
Author: Damian Guy <damian.guy@gmail.com>
Reviewers: Guozhang Wang, Ismael Juma, Michael G. Noll
Closes#1811 from dguy/kstreams-312
1. I think the instructions in step 2 of the security section which describe adding the CA to server/client truststores are swapped. That is, the instruction that says to add the CA to the server truststore adds it to the client truststore (and vice versa).
2. "clients keys" should be possessive ("clients' keys").
This contribution is my original work, and I license the work to the project under the project's open source license.
Author: Samuel Taylor <staylor@square-root.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1651 from ssaamm/trunk
Add an optional configuration for the SecureRandom PRNG implementation, with the default behavior being the same (use the default implementation in the JDK/JRE).
Author: Todd Palino <Todd Palino>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy@gmail.com>, Jiangjie Qin <becket.qin@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Closes#1747 from toddpalino/trunk
added new consumer metrics section
refactored common metrics into new section
updated TOC
Author: Kaufman Ng <kaufman@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1361 from coughman/KAFKA-3479-consumer-metrics-doc
ijuma said that it would make sense to split out this work from KAFKA-3234, since KAFKA-3234 had both a mechanical change (generating docs) as well as a change requiring discussion (deprecating/renaming config options).
jjkoshy, I hope you don't mind that I took over this work. It's been 3 months since the last activity on KAFKA-3234, so I thought it would be okay to take over.
This work is essentially is the first 5-6 commits from Joel's https://github.com/apache/kafka/pull/907. However, since I'm not very experienced with git, I didn't do a direct merge/rebase, but instead largely hand-merged it. I did some minor cleanup. All credit goes to Joel, all blame goes to me. :)
For reference, I attached the auto-generated configuration.html file (as a PDF, because github won't let me attache html).
[configuration.pdf](https://github.com/apache/kafka/files/323901/configuration.pdf)
This is my first time writing Scala, so let me know if there are any changes needed.
I don't know who is the right person to review this. ijuma, can you help me redirect this to the appropriate person? Thanks.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1527 from wushujames/generate_topic_docs
By default Kafka is configured to allow ssl communication without hostname verification. This docs has been amended to include instructions on how to set that up in the event clients would like to take a more conservative approach.
Author: Ryan P <ryan.n.pridgeon@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1384 from rnpridgeon/KAFKA-3667
This contribution is my original work, and I license the work to the project under the project's open source license.
Author: Samuel Taylor <staylor@square-root.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1630 from ssaamm/trunk
I've updated the ops documentation with information on using the XFS filesystem, based on LinkedIn's testing (and subsequent switch from EXT4).
I've also added some information to clarify the potential risk to the suggested EXT4 options (again, based on my experience with a multiple broker failure situation).
Author: Todd Palino <tpalino@linkedin.com>
Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Dana Powers <dana.powers@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1605 from toddpalino/trunk
Since 0.9.0.1 Configuration parameter log.cleaner.enable is now true by default.
Author: Nihed MBAREK <nihedmm@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1592 from nihed/patch-1
The reasons to remove it are:
1. It's currently broken. The purpose of the [JIRA](https://issues.apache.org/jira/browse/KAFKA-3761) was to report that the RunningAsController state gets overwritten back to "RunningAsBroker".
2. It's not a useful state.
a. If clients want to use this metric to know whether a broker is ready to receive requests or not, they do not care whether or not the broker is the controller
b. there is already a separate boolean property, KafkaController.isActive which contains this information.
Author: Roger Hoover <roger.hoover@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1437 from theduderog/KAFKA-3761-broker-state
We had mentioned this step in the performance impact section in the middle of a long paragraph, which made it easy to miss. I also tweaked the reason for setting `log.message.format.version` as it could be misinterpreted previously.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1514 from ijuma/tweak-upgrade-notes
To be consistent with `ConfigCommand` and `TopicCommand`.
No release includes this option yet, so we can simply change it.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Mickael Maison, Grant Henke
Closes#1430 from ijuma/use-force-instead-of-yes-in-acl-command and squashes the following commits:
bdf3a57 [Ismael Juma] Update `AclCommandTest`
78b8467 [Ismael Juma] Change variable name to `forceOpt`
0bb27af [Ismael Juma] Use `--force` instead of `--yes` in `AclCommand`
Added a new argument to AclCommand: --yes. When set, automatically answer yes to prompts
Author: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Gwen Shapira
Closes#1406 from mimaison/KAFKA-3732
Adding sizing recommendations for file descriptors to the ops guide.
Author: Dustin Cote <dustin@confluent.io>
Author: Dustin Cote <dustin@dustins-mbp.attlocal.net>
Reviewers: Gwen Shapira
Closes#1353 from cotedm/KAFKA-3683 and squashes the following commits:
8120318 [Dustin Cote] Adding file descriptor sizing recommendations
0908aa9 [Dustin Cote] Merge https://github.com/apache/kafka into trunk
32315e4 [Dustin Cote] Merge branch 'trunk' of https://github.com/cotedm/kafka into trunk
13309ed [Dustin Cote] Update links for new consumer API
4dcffc1 [Dustin Cote] Update links for new consumer API
Process grep command has been updated. Previous "ps | grep server-1.properties" command is showing nothing.
Author: Satendra Kumar <satendra@knoldus.com>
Reviewers: Gwen Shapira
Closes#1386 from satendrakumar06/patch-1
It was previously in the SASL section (probably by mistake).
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>
Closes#1405 from ijuma/fix-security-upgrade-location-in-docs
Author: MayureshGharat <gharatmayuresh15@gmail.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1060 from MayureshGharat/KAFKA-3393
This is not an exact revert as the code changed a bit since the
original commit. We also include a note in `upgrade.html`.
The original commit is 1182d61deb.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira, Guozhang Wang
Closes#1391 from ijuma/kafka-3704-revert and squashes the following commits:
7891b67 [Ismael Juma] Tweak upgrade note based on Gwen's feedback
1673cd0 [Ismael Juma] Revert "KAFKA-3704: Remove hard-coded block size in KafkaProducer"
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1375 from becketqin/KAFKA-3698
If you're pushing close to the network capacity, 0.10's additional 8 bytes per message can lead to overload of your network. We (Heroku Kafka) ran into this issue whilst benchmarking 0.10 RC and the ijuma suggested it belonged in the update note.
Comments/suggestions welcome.
Author: Tom Crayford <tcrayford@googlemail.com>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1389 from tcrayford/upgrade_note_about_increased_network_bandwidth
…e throughput drop due to the addition of timestamp field.
Author: Jiangjie Qin <becket.qin@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#1372 from becketqin/KAFKA-3565
This fixes test_producer_throughput with compression_type=snappy.
Also: added heap dump on out of memory error to `producer_performance.py` and corrected the upgrade note related to the change in buffer size for compression streams.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Gwen Shapira
Closes#1385 from ijuma/kafka-3713-test_producer_throughput-snappy-fail and squashes the following commits:
54c7962 [Ismael Juma] Correct upgrade note about buffer size for compression stream
515040b [Ismael Juma] Call `compressor.close()` to fix memory leak
5311e5b [Ismael Juma] Dump heap on out of memory error when running `producer_performance.py`
This is a follow up of KAKFA-3421 to update the connect developer guide to include the configuration validation. Also includes a couple of minor fixes.
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1366 from Ishiihara/connect-dev-doc
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#1373 from ijuma/fix-producer-buffer-size-upgrade-note
Author: Jason Gustafson <jason@confluent.io>
Reviewers: Roger Hoover <roger.hoover@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1358 from hachikuji/KAFKA-3583
Author: Guozhang Wang <wangguoz@gmail.com>
Reviewers: Michael G. Noll <michael@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Closes#1321 from guozhangwang/KStreamsJavaDoc
This contribution is my original work and I license the work under Apache 2.0.
Author: Dana Powers <dana.powers@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
Closes#1212 from dpkp/KAFKA-3160
Pull request to update the consumer API links in the docs.
Author: Dustin Cote <dustin@dustins-mbp.attlocal.net>
Reviewers: Ismael Juma <ismael@juma.me.uk>
Closes#1331 from cotedm/KAFKA-3666
Documentation corresponding to KIP-43 - SASL/PLAIN and multiple mechanism support.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Magnus Edenhill <apache_m@edenhill.se>, Jun Rao <junrao@gmail.com>
Closes#1232 from rajinisivaram/KAFKA-3517
This patch fixes all occurances of two consecutive 'the's in the code comments.
Author: Ishita Mandhan (imandhaus.ibm.com)
Author: Ishita Mandhan <imandha@us.ibm.com>
Reviewers: Guozhang Wang <wangguoz@gmail.com>
Closes#1240 from imandhan/typofixes
ewencp gwenshap Docs. I also tried to clean up some typos. However, it seems that the we don't have two words without space in between in the source yet they showed up as no space in between in the generated doc.
Author: Liquan Pei <liquanpei@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1227 from Ishiihara/config-doc
This PR fixes 8 typos in HTML files of `docs` module. I wrote explicitly here since Github sometimes does not highlight the corrections on long lines correctly.
- docs/api.html: compatability => compatibility
- docs/connect.html: simultaneoulsy => simultaneously
- docs/implementation.html: LATIEST_TIME => LATEST_TIME, nPartions => nPartitions
- docs/migration.html: Decomission => Decommission
- docs/ops.html: stoping => stopping, ConumserGroupCommand => ConsumerGroupCommand, youre => you're
Author: Dongjoon Hyun <dongjoon@apache.org>
Reviewers: Ismael Juma
Closes#1138 from dongjoon-hyun/KAFKA-3461
Sentence was missing "as", minor grammar clean up.
Author: Paul Cavallaro <paulcavallaro@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1151 from paulcavallaro/docs-fix
This fixes an issue with long topic names by considering, during topic
validation, the '-' and the partition id that is appended to the log
folder created for each topic partition.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Gwen Shapira, Grant Henke
Closes#898 from vahidhashemian/KAFKA-3219
This will need to be double-committed.
Author: Gwen Shapira <cshapi@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#1107 from gwenshap/fix-doc-version
Please see https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment for the overall design.
The update to TopicMetadataRequest/TopicMetadataResponse will be done in a different PR.
Author: Allen Wang <awang@netflix.com>
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#132 from allenxwang/KAFKA-1215
Author: Manikumar reddy O <manikumar.reddy@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <mail@harsha.io>
Closes#1054 from omkreddy/KAFKA-2551
Because protocol.html is going to be in its own page it needs the header and footer included.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira
Closes#1043 from granthenke/protocol-docs-style
- Moves all generated docs under /docs/generated
- Generates docs for Protocol, Errors, and ApiKeys
- Adds new protocol.html page
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira
Closes#970 from granthenke/protocol-doc-wip
https://issues.apache.org/jira/browse/KAFKA-1476
Let me know if these kind of contributions should have their own requisite JIRA opened in advance.
Cheers..
Author: Christian Posta <christian.posta@gmail.com>
Reviewers: Gwen Shapira
Closes#945 from christian-posta/ceposta-tidy-up-consumer-groups-describe
Adjust the listeners property rather than the port. Following the original instructions would result in all of the brokers being started with the same listeners setting, and so fail to work.
Author: Duncan Sands <baldrick@free.fr>
Reviewers: Ismael Juma, Gwen Shapira
Closes#1002 from CunningBaldrick/quickstart
* Change `MessageFormat.writeTo` to take a `ConsumerRecord`
* Change `MessageReader.readMessage()` to use `ProducerRecord`
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#972 from ijuma/kafka-3273-message-formatter-and-reader-resilient
…list is not supported for new consumers
Author: Ashish Singh <asingh@cloudera.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#923 from SinghAsDev/KAFKA-3243
This PR includes a number of clean-ups:
* Code style
* Documentation wording improvements
* Efficiency improvements
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Jun Rao <junrao@gmail.com>
Closes#943 from ijuma/kafka-3259-kip-31-32-clean-ups
See KIP-31 and KIP-32 for details.
A few notes on the patch:
1. This patch implements KIP-31 and KIP-32. The patch includes features in both KAFKA-3025, KAFKA-3026 and KAFKA-3036
2. All unit tests passed.
3. The unit tests were run with new and old message format.
4. When message format conversion occurs during consumption, the consumer will not be able to detect the message size too large situation. I did not try to fix this because the situation seems rare and only happen during migration phase.
Author: Jiangjie Qin <becket.qin@gmail.com>
Author: Ismael Juma <ismael@juma.me.uk>
Author: Jiangjie (Becket) Qin <becket.qin@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>, Anna Povzner <anna@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>, Jun Rao <junrao@gmail.com>
Closes#764 from becketqin/KAFKA-3025
There are multi-byte characters In quickstart.html and security.html.
This PR will fix it.
Author: Sasaki Toru <sasakitoa@nttdata.co.jp>
Reviewers: Grant Henke
Closes#897 from sasakitoa/remove_multi_byte_character
Correct the text that said that you can't change the replication factor of a topic.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ismael Juma
Closes#881 from wushujames/KAFKA-3216
… is not specified correctly
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#876 from granthenke/rebalance-doc
Set default to 1 instead of 1024, this matches the existing doc and feels like a better default value. Have run the unit tests with the change.
Author: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson, Ismael Juma, Grant Henke, Guozhang Wang
Closes#832 from rajinisivaram/KAFKA-3170
And added info about the krb5.conf file as we don't appear to mention that in the current docs
Author: Ben Stopford <benstopford@gmail.com>
Reviewers: Ismael Juma
Closes#625 from benstopford/security_docs
Simple fix, but important because the incorrect syntax causes the server not to start.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Grant Henke, Guozhang Wang
Closes#819 from ijuma/fix-jaas-comment-syntax
Log compaction is supported on compressed topics as of 0.9.0, so update the docs to reflect that.
Author: James Cheng <jylcheng@yahoo.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#807 from wushujames/KAFKA-3138
Improve the documentation by fixing typos, punctuations, and correcting the content.
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io>
Closes#778 from vahidhashemian/typo05/fix_documentation_typos
Provides a configuration to opt out of broker id generation.
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Gwen Shapira
Closes#762 from granthenke/id-generation
Add some basic documentation about the format, a link to get more detailed information and an example usage. I didn't want to make a huge section on the format since it documented elsewhere but I can expand is folks want.
https://issues.apache.org/jira/browse/KAFKA-3095
Author: Tom Graves <tgraves@yahoo-inc.com>
Reviewers: Gwen Shapira
Closes#776 from tgravescs/KAFKA-3095
topic level config delete config options
use --delete-config instead of --deleteConfig
Author: eamon <zhang.eamon@hotmail.com>
Reviewers: Ismael Juma, Grant Henke, Guozhang Wang
Closes#612 from EamonZhang/trunk
Simple fixes that have tripped users.
Author: Ismael Juma <ismael@juma.me.uk>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>
Closes#745 from ijuma/security-doc-improvements
Fix two minor typos in Kafka official website page.
Author: Gabriel Zhang <smalldirector@yahoo.com>
Reviewers: Ismael Juma, Guozhang Wang
Closes#742 from smalldirector/kafka-document-typos-fix
Author: Grant Henke <granthenke@gmail.com>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Closes#686 from granthenke/compaction
This commit contains minor grammatical fixes. Some of the changes are just removing rogue commas, which can be hard to see in the diff.
This contribution is my original work and I license the work to the project under the project's open source license.
Author: Samuel Julius Hecht <samjhecht@gmail.com>
Reviewers: Gwen Shapira
Closes#721 from samjhecht/minor-docs-edits
sasl.kerberos.service.name surround by double quote didn't work, have to remove.
Author: BINLEI XUE <kongpo0412@gmail.com>
Reviewers: Gwen Shapira
Closes#720 from secjex/patch-1
In Kafka 0.9, request.required.acks=-1 which configration of producer is replaced by acks=all,
but this old config is remained in docs.
Author: Sasaki Toru <sasakitoa@nttdata.co.jp>
Reviewers: Gwen Shapira
Closes#716 from sasakitoa/acks_doc
Rajini Sivaram
Manikumar Reddy
Manjula K
Richard Yu
Dong Lin
Matthias J. Sax
Mickael Maison
Damian Guy
manjuapu
Florian Hussonnois
Jakub Scholz
Guozhang Wang
Boyang Chen
James Cheng
Eno Thereska
Michal Borowiecki
Bill Bejeck
Jason Gustafson
Ismael Juma
Robin Moffatt
Ivan Babrou
Derrick Or
Vahid Hashemian
ppatierno
Jeff Klukas
Gwen Shapira
huxihx
Apurva Mehta
Kevin Sweeney
Tom Bentley
Edoardo Comar
Hao Chen
Michal Borowiecki
xinlihua
Jiangjie Qin
Michael André Pearce
Colin P. Mccabe
Ryan P
sharad-develop
Eno Thereska
sunnykrgupta
shuguo zheng
Michael G. Noll
Sönke Liebau
Grant Henke
Alex Loddengaard
huxi
Onur Karaman
Ewen Cheslack-Postava
Hojjat Jafarpour
Shikhar Bhushan
dasl
Scott Ferguson
Hikiko Murakami
Stig Rohde Døssing
Apurva Mehta
Ben Stopford
Andrea Cosentino
Jakub Dziworski
pilo
Jay Kreps
Sumit Arrawatia
Eric Wasserman
Mathieu Fenniak
Andrew Otto
Ashish Singh
Samuel Taylor
Todd Palino
sahil kharb
Kaufman Ng
Todd Palino
Nihed Bbarek
Alex Glikson
Thanasis Katsadas
Roger Hoover
Filipe Azevedo
uchan-nos
Dustin Cote
Satendra Kumar
Sriharsha Chintalapani
MayureshGharat
Tom Crayford
Liquan Pei
Dana Powers
Dustin Cote
Dionysis Grigoropoulos
Ishita Mandhan
Dongjoon Hyun
Paul Cavallaro
Flavio Junqueira
Grant Henke
Allen Wang
Christian Posta
Duncan Sands
Sasaki Toru
Jun Rao
Jakub Nowak
Tom Graves
David Jacot
Eamon Zhang
Ján Koščo
Gabriel Zhang
Samuel Julius Hecht
BINLEI XUE