Commit Graph

425 Commits

Author SHA1 Message Date
Victoria Bialas dd8131499f KAFKA-8227 DOCS Fixed missing links duality of streams tables (#6625)
Fixed missing links duality of streams tables

Reviewers: Jim Galasyn <jim.galasyn@confluent.io> Bill Bejeck <bbejeck@gmail.com>
2019-04-24 14:54:29 -07:00
Ted Yu e56ebbffca [KAFKA-3729] Auto-configure non-default SerDes passed alongside the topology builder (#6461)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2019-04-20 11:30:20 -07:00
John Roesler 99ce7d76ce KAFKA-7778: document scala suppress API (#6586)
Document the minor API change.

Reviewers: Casey Green <greenc421@gmail.com>, Guozhang Wang <wangguoz@gmail.com>,  Bill Bejeck <bbejeck@gmail.com>
2019-04-16 13:48:58 -07:00
Bill Bejeck 312e55bf56
KAFKA-8208: Change paper link directly to ASM (#6572)
Reviewers: Matthias J. Sax <mjsax@apache.org>, Victoria Bialas <vicky@confluent.io>
2019-04-13 18:59:40 -04:00
Bill Bejeck 5e2d062267
KAFKA-8210: Fix link for streams table duality (#6573)
Reviewers: Victoria Bialas <vicky@confluent.io>
2019-04-13 18:54:06 -04:00
Bill Bejeck 75dc7e2b6b
KAFKA-8209: Wrong link for KStreams DSL in core concepts doc (#6564)
Reviewers Matthias J. Sax <mjsax@apache.org>, Michael Drogalis <michael.drogalis@confluent.io>, Victoria Bialas <vicky@confluent.io>
2019-04-13 18:50:27 -04:00
Victoria Bialas ad0f7aead7 KAFKA-8212 DOCS (kafka) - Fix Maven artifacts table from cutting off text (#6576)
Reviewer: Matthias J. Sax <matthias@confluent.io>
2019-04-13 15:43:51 -07:00
Victoria Bialas 8bd66eb870 KAFKA-8213 - Fix typo in Streams Dev Guide (#6574)
Reviewers: Jim Galasyn <jim.galasyn@confluent.io>, Joel Hamill <joel@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2019-04-13 15:35:47 -07:00
Victoria Bialas e5a131d9bf KAFKA-8181: Removed Avro topic from TOC on kafka (#6529)
Removed TOC entry in Streams Developer Guide for Avro, since we have no content for this

PR on kafka-site: apache/kafka-site#195

Reviewers: Guozhang Wang <wangguoz@gmail.com>
2019-04-01 16:32:14 -07:00
Jarrod Urban acb5188810 fix compile error for example (#6526)
Reviewers: Prashant Sabnekar, Bill Bejeck <bbejeck@gmail.com>
2019-04-01 11:39:05 -04:00
Tcsalist 17f73b42ad MINOR: Streams input topic corrected (#6513)
Though out the tutorial, the name of the input topic that was created is `streams-plaintext-input`. However, this was mistaken at some point in the tutorial and changed to `streams-wordcount-input`.

This patch is to adjust that. Thanks.

Reviewers: Guozhang Wang <wangguoz@gmail.com>
2019-03-28 21:02:41 -07:00
Matthias J. Sax 82615256ff MINOR: add MacOS requirement to Streams docs
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*

*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Bill Bejeck

Closes #6490 from mjsax/minor-streams-docs-rocksdb
2019-03-23 15:48:24 -07:00
Matthias J. Sax 4cae4523fc MINOR: Add verification step for Streams archetype to Jenkins build (#6431)
Updates ./jenkins.sh to build stream archetype and install it in local maven cache. Afterward, archetype is used to create a new maven project and maven project is compiled for verification.

Reviewers: Guozhang Wang <wangguoz@gmail.com>, John Roesler <john@confluent.io>,  Bill Bejeck <bbejeck@gmail.com>
2019-03-21 15:51:18 -04:00
Victoria Bialas 70ee72491f MINOR: updated names for deprecated streams constants (#6466)
* updated names for deprecated streams constants
* add DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG in place of deprecated

Reviewers: Jim Galasyn <jim.galasyn@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2019-03-19 16:33:41 -07:00
Victoria Bialas 5092b26393 MINOR: update docs JSON serde links (#6465)
Reviewers: Joel Mamill <joel@confluent.io>, Matthias J. Sax <mjsax@apache.org>
2019-03-19 18:01:59 -04:00
A. Sophie Blee-Goldman e62e23099b Fixed docs regarding caching default (#6375)
The current docs are out of date regarding the default caching state.

Reviewers: Guozhang Wang <wangguoz@gmail.com>,  Bill Bejeck <bbejeck@gmail.com>
2019-03-06 13:54:36 -05:00
cwildman abbd992a4a MINOR: state.cleanup.delay.ms default is 600,000 ms (10 minutes). (#6345)
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2019-02-28 09:22:54 -08:00
Vito Jeng 342c336bf9 MINOR: Missing punctuation marks in quickstart (#5755)
Minor fix for missing punctuation marks in the quickstart.

Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bbejeck@gmail.com>
2019-02-21 19:41:03 -05:00
Chia-Ping Tsai 35a0de32ee KAFKA-6161 Add default implementation to close() and configure() for Serdes (#5348)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2019-02-21 09:05:13 -08:00
A. Sophie Blee-Goldman ff603c63bb KAFKA-4730: Streams does not have an in-memory windowed store (#6239)
Implemented an in-memory window store allowing for range queries. A finite retention period defines how long records will be kept, ie the window of time for fetching, and the grace period defines the window within which late-arriving data may still be written to the store.

Unit tests were written to test the functionality of the window store, including its insert/update/delete and fetch operations. Single-record, all records, and range fetch were tested, for both time ranges and key ranges. The logging and metrics for late-arriving (dropped)records were tested as well as the ability to restore from a changelog.

Reviewers: John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2019-02-20 19:09:50 -08:00
Viktor Somogyi-Vass 776041db11 KAFKA-7804: Update docs for topic-command related KIP-377
This PR adds a upgrade notes and changes examples to use the bootstrap-server.

Author: Viktor Somogyi-Vass <viktorsomogyi@gmail.com>

Reviewers: Srinivas <srinivas96alluri@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>

Closes #6118 from viktorsomogyi/topiccommand-adminclient-doc
2019-02-04 13:46:33 -08:00
Guozhang Wang 68a6a7ae29
MINOR: Streams upgrade guide section for newly merged KIPs (#6108)
Reviewers: Bill Bejeck <bbejeck@gmail.com>, Matthias J. Sax <matthias@confluent.io>
2019-01-15 10:25:29 -08:00
John Roesler 8ae985705f KAFKA-7768: Use absolute paths for javadoc (#6100)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2019-01-07 13:30:49 -08:00
slim 6a20e3cd4c KAFKA-7768: Add version to java html urls (#6094)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2019-01-07 09:21:33 -08:00
Guozhang Wang 5c549b2a89
MINOR: Replace tbd with the actual link for out-of-ordering data (#6035)
Reviewers: Jason Gustafson <jason@confluent.io>
2018-12-14 09:37:43 -08:00
Matthias J. Sax a1807d4914 MINOR: Improve GlobalKTable docs (#5996)
Reviewers: Jim Galasyn, Michael G. Noll, John Roesler, Bill Bejeck, Guozhang Wang
2018-12-04 16:21:18 -08:00
cadonna 808dc0a96b MINOR: Update docs with out-dated context.schedule(...) examples (#5924)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-11-21 09:48:01 -08:00
Ismael Juma b4722cc16f
KAFKA-7524: Recommend Scala 2.12 and use it for development (#5530)
Scala 2.12 has better support for newer Java versions and includes additional
compiler warnings that are helpful during development. In addition, Scala 2.11
hasn't been supported by the Scala community for a long time, the soon to be
released Spark 2.4.0 will finally support Scala 2.12 (this was the main reason
preventing many from upgrading to Scala 2.12) and Scala 2.13 is at the RC stage.
It's time to start recommending the Scala 2.12 build as we prepare support for
Scala 2.13 and start thinking about removing support for Scala 2.11.

In the meantime, Jenkins will continue to build all supported Scala versions (including
Scala 2.11) so the PR and trunk jobs will fail if people accidentally use methods
introduced in Scala 2.12.

Reviewers: Ewen Cheslack-Postava <me@ewencp.org>
2018-10-28 11:31:39 -07:00
Matthias J. Sax ff3c4e22b8
MINOR: fix docs typo (#5827)
Reviewers: Bill Bejeck <bill@confluent.io>, John Roesler <john@confluent.io>
2018-10-23 16:47:06 -07:00
Bill Bejeck 86b1150e18 MINOR: Update Streams Scala API for addition of Grouped (#5793)
While working on the documentation updates I realized the Streams Scala API needs
to get updated for the addition of Grouped

Added a test for Grouped.scala ran all streams-scala tests and streams tests

Reviewers: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-10-16 07:24:50 -07:00
Bill Bejeck 90d0fd51d1 MINOR: Doc changes for KIP-312 (#5789)
Documentation changes for adding overloaded StreamsBuilder(java.util.Properties props) method in KIP-312

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-10-15 21:26:18 -07:00
John Roesler 4b7148a5b6 KAFKA-7223: Suppression documentation (#5787)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-10-15 21:04:01 -07:00
Matthias J. Sax 2646781d32
KAFKA-7080 and KAFKA-7222: Cleanup overlapping KIP changes (#5804)
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-10-15 20:56:30 -07:00
Bill Bejeck 8182c4fc7d MINOR: doc changes for KIP-372 (#5790)
Reviewers: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>
2018-10-15 19:01:21 -07:00
Bill Bejeck 763b72c9a5 MINOR: Doc changes for KIP-324 (#5788)
Reviewers: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>
2018-10-15 18:55:39 -07:00
Matthias J. Sax 0b417b8331
MINOR: updates docs for KIP-358 (#5796)
Reviewers: Guozhang Wang <guozhang@confluent.io>, Jim Galasyn <jim.galasyn@confluent.io>
2018-10-15 17:22:03 -07:00
Guozhang Wang 6a3382c9d3
MINOR: Streams Update for KIP-330 / KIP-356 (#5794)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-10-12 14:48:28 -07:00
Matthias J. Sax 349b5ad56c
KAFKA-4932: Update docs for KIP-206 (#5769)
Reviewers: Bill Bejeck <bill@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-10-10 12:09:09 -07:00
Bill Bejeck 862e09f994 HOTFIX: Fix broken links (#5676)
Reviewers: Joel Hamill <11722533+joel-hamill@users.noreply.github.com>, Guozhang Wang <wangguoz@gmail.com>
2018-10-03 18:27:15 -07:00
Guozhang Wang 7bd8ada8e2
MINOR: Docs on state store instantiation (#5698)
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-10-03 11:20:47 -07:00
Guozhang Wang 08dfab06e7
KAFKA-3514: Upgrade Documentation (#5714)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2018-09-30 12:01:11 -07:00
Lee Dongjin e7b6d65713 KAFKA-6620: Fix documentation about 'exactly_once'
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2018-09-28 14:41:50 -07:00
Guozhang Wang 68b2f49ea7
KAFKA-3514, Documentations: Add out of ordering in concepts. (#5458)
Reviewers: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-09-11 16:28:52 -07:00
huxi aa7358e8cc KAFKA-7326: KStream.print() should flush on each line for PrintStream (#5579)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>
2018-09-01 09:48:50 -07:00
lucapette 2514a42e84 KAFKA-7269: Add docs for KStream.merge (#5512)
Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-08-29 09:34:54 -07:00
nprad 79a2f892ca KAFKA-6966: Extend TopologyDescription to better represent Source and (#5284)
Implements KIP-321

Reviewers: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-08-10 16:12:03 -07:00
Matthias J. Sax c1ca53b369 MINOR: fix Streams docs state.dir (#5465)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-08-06 13:39:50 -07:00
Simon Clark 530d951cbd MINOR: Fixed default streams state dir location. (#5441)
Co-authored-by: Mickael Maison <mickael.maison@gmail.com>
Co-authored-by: Simon Clark <simonc6r@gmail.com>

Reviewers: Sriharsha Chintalapani <sriharsha@apache.org>
2018-08-01 19:27:08 -07:00
Guozhang Wang 2d877631cd
MINOR: web docs fixes on message header (#5381)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2018-07-17 17:31:32 -07:00
Guozhang Wang 6bfaf4dc60
MINOR: Store metrics scope, total metrics (#5290)
1. Rename metrics scope of rocksDB window and session stores; also modify the store metrics accordingly with guidance on its correlations to metricsScope.

2. Add the missing total metrics for per-thread, per-task, per-node and per-store sensors.


Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-06-28 14:32:02 -07:00
Guozhang Wang 7ea8a25a4c HOTFIX: remove old security suggestions 2018-06-27 09:35:34 -07:00
Guozhang Wang a20555102b HOTFIX: update Streams security docs 2018-06-26 18:53:06 -07:00
Jim Galasyn 581adb5013 MINOR: Add note about num.standby.replicas (#5271)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-06-26 08:24:44 -07:00
Vahid Hashemian c32f7e6ad8 MINOR: Fix expected output in Streams quickstart (#5280)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-06-22 17:18:10 -07:00
Guozhang Wang d3e264e773
MINOR: update web docs and examples of Streams with Java8 syntax (#5249)
Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>, Damian Guy <damian@confluent.io>
2018-06-21 10:02:58 -07:00
Guozhang Wang f8cb3c3e1d
MINOR: add headers support in new api (#5252)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2018-06-19 14:23:04 -07:00
Guozhang Wang 1546bcd877
MINOR: provide an example for deserialization exception handler (#5231)
Also added a paragraph from data types to link to the example code.

Reviewers: Matthias J. Sax <mjsax@apache.org>
2018-06-17 17:31:30 -07:00
Guozhang Wang dad19ac00f
KAFKA-7021: Update upgrade guide section for reusing source topic (#5195)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2018-06-15 08:49:45 -07:00
Guozhang Wang 86410e14d7 MINOR: Add missing configs for resilience settings
Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-06-05 11:41:03 -07:00
Jorge Quilcate Otoya e8ddb76573 KAFKA-6938: Add documentation for accessing Headers on Kafka Streams Processor API (#5128)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-06-04 15:39:20 -07:00
Guozhang Wang 718d6f2475
MINOR: Remove deprecated KafkaStreams constructors in docs (#5118)
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-06-04 13:43:20 -07:00
Vahid Hashemian 0cacbcf30e MINOR: Remove usages of JavaConversions and fix some typos (#5115)
Reviewers: Ismael Juma <ismael@juma.me.uk>
2018-06-02 23:45:44 -07:00
Guozhang Wang f33e9a346e KAFKA-4936: Add dynamic routing in Streams (#5018)
implements KIP-303

Reviewers: Bill Bejeck <bill@confluent.io>, John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-05-30 11:54:53 -07:00
Joan Goyeau 1e0793c4ce Update doc of curried joins and aggregates (#5053)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-05-30 11:03:58 -07:00
Andy Coates 4e1c8ffd0d KAFKA-6849: add transformValues methods to KTable. (#4959)
See the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-292%3A+Add+transformValues%28%29+method+to+KTable

This PR adds the transformValues method to the KTable interface. The semantics of the call are the same as the methods of the same name on the KStream interface.

Fixes KAFKA-6849

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2018-05-18 16:06:50 -07:00
Guozhang Wang 6b8e79b137
HOTFIX: move Conusmed to o.a.k.streams.kstream (#5033)
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-05-17 11:56:07 -07:00
Matthias J. Sax 0b3712d8a5
MINOR: add missing parameter `processing.guaratees` to Streams docs (#5023)
Reviewers: Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-05-16 13:58:25 -07:00
Guozhang Wang d4204e8b14
MINOR: fix broken links in streams doc (#5025)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-05-16 13:23:20 -07:00
David Glasser e9154b7960 KAFKA-6905: Document that Processors may be re-used by Streams (#5022)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-05-16 13:10:21 -07:00
Guozhang Wang c9161afda9
MINOR: doc change for deprecate removal (#5006)
Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-05-15 20:26:19 -07:00
Joel Hamill c14b0ad9ee MINOR - Fix typo in Streams Dev Guide (#4972)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-05-11 10:27:07 -07:00
Joan Goyeau b88d70b532 MINOR: Make Serdes less confusing in Scala (#4963)
Serdes are confusing in the Scala wrapper:

* We have wrappers around Serializer, Deserializer and Serde which are not very useful.
* We have Serdes in 2 places org.apache.kafka.common.serialization.Serde and in DefaultSerdes, instead we should be having only one place where to find all the Serdes.

I wanted to do this PR before the release as this is a breaking change.
This shouldn't add more so the current tests should be enough.

Reviewers: Debasish Ghosh <dghosh@acm.org>, Guozhang Wang <guozhang@confluent.io>
2018-05-08 09:15:31 -07:00
Guozhang Wang 32e97b1d9d
MINOR: Remove deprecated parameter in ProcessorContext#register (#4911)
Updated the upgrade doc as well since we do not have an overloaded function without the deprecated parameter before. Also renamed the 1.2 release version to 2.0.

Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-05-07 09:22:26 -07:00
Sean Glover 893e044515 MINOR: Build and code sample updates for Kafka Streams DSL for Scala (#4949)
Several build and documentation updates were required after the merge of KAFKA-6670: Implement a Scala wrapper library for Kafka Streams.

Encode Scala major version into streams-scala artifacts.
To differentiate versions of the kafka-streams-scala artifact across Scala major versions it's required to encode the version into the artifact name before its published to a maven repository. This is accomplished by following a similar release process as kafka core, which encodes the Scala major version and then runs the build for each major version of Scala supported. This is considered standard practice when releasing Scala libraries, but is not handled for us automatically with the basic Scala for Gradle support.

After this change you can generate and install the kafka-streams-scala artifact into the local maven repository:

$ ./gradlew -PscalaVersion=2.11 install
$ ./gradlew -PscalaVersion=2.12 install

Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
2018-05-06 20:55:12 -07:00
Guozhang Wang de8a67b565 HOTFIX: fix streams tutorial code example 2018-05-04 09:59:36 -07:00
Boyang Chen 1b170df31c KAFKA-6657: Add StreamsConfig prefix for different consumers (#4805)
This pull request is for JIRA 6657, for KIP-276.

Added unit tests for new getGlobalConsumerConfigs API and make sure existing restore consumer tests are passing.

Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2018-05-02 13:24:15 -07:00
khairy 6655a4d75f KAFKA-6535: Set default retention ms for Streams repartition topics to Long.MAX_VALUE (#4730)
Implements KIP-284

Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-04-30 12:18:40 +02:00
manjuapu 0143a65091 Fix streams web doc configs tables (#4943)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-04-27 16:15:55 -07:00
manjuapu 3ce14a84a8 MINOR: Streams web doc table fix (#4942) 2018-04-27 14:20:29 -07:00
John Roesler 12a0f46895 KAFKA-6376: Document skipped records metrics changes (#4922)
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2018-04-24 15:40:16 -07:00
Debasish Ghosh b2e4db01b6 KAFKA-6670: Implement a Scala wrapper library for Kafka Streams
This PR implements a Scala wrapper library for Kafka Streams. The library is implemented as a project under streams, namely `:streams:streams-scala`. The PR contains the following:

* the library implementation of the wrapper abstractions
* the test suite
* the changes in `build.gradle` to build the library jar

The library has been tested running the tests as follows:

```
$ ./gradlew -Dtest.single=StreamToTableJoinScalaIntegrationTestImplicitSerdes streams:streams-scala:test
$ ./gradlew -Dtest.single=StreamToTableJoinScalaIntegrationTestImplicitSerdesWithAvro streams:streams-scala:test
$ ./gradlew -Dtest.single=WordCountTest streams:streams-scala:test
```

Author: Debasish Ghosh <ghosh.debasish@gmail.com>
Author: Sean Glover <seglo@randonom.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk>, John Roesler <john@confluent.io>, Damian Guy <damian@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #4756 from debasishg/scala-streams
2018-04-23 13:33:35 -07:00
taekyung 51db468bf3 MINOR: Fixed deserialization.exception.handler default value of config-streams (#4914)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-04-23 09:49:53 -07:00
ro7m b6f0420350 Kafka-6792: Fix wrong pointer in the link for stream dsl (#4876)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-04-16 10:49:41 -07:00
ro7m 8434d494e6 KAFKA-6790: Fix Streams processor node broken link (#4874)
The page here https://kafka.apache.org/11/documentation/streams/developer-guide/memory-mgmt.html talks about processor nodes and refers to non existing links.

Broken link (appears twice in the same document):

https://kafka.apache.org/11/documentation/streams/concepts.html#streams-concepts-processor

To find this search for the word "processor node" on the page memory-management , the ones which are links are broken.

Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-04-15 09:56:12 -07:00
Matthias J. Sax 0c0d8363e5
KAFKA-6054: Fix upgrade path from Kafka Streams v0.10.0 (#4779)
Reviewers: Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>, John Roesler <john@confluent.io>, Damian Guy <damian@confluent.io>
2018-04-06 17:00:52 -07:00
ro7m ac4374dc24 KAFKA-6732: Fix Streams doc ref link (#4806)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-04-01 18:05:15 -07:00
Boyang Chen 964693e40d KAFKA-6386: Use Properties instead of StreamsConfig in KafkaStreams constructor
This pull request targets https://issues.apache.org/jira/browse/KAFKA-6386
The minor fix to deprecate usage of `StreamsConfig` in favor of `java.util.Properties`.
I created separate public constructors using `Properties` in order to replace the old ones,
and prioritize new functions in the `KafkaStreams.java` file.

Since this is my first time doing open source contribution, I'm very happy to get
any comment or pointer to be more professional and get better next time, thank you Guozhang guozhangwang and Liquan Ishiihara!

testing strategy: existing unit test should be suffice to cover this change.

Author: cs427fa16staff <bchen11@outlook.com>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>

Closes #4354 from abbccdda/starter

github comments
2018-03-27 16:09:34 -07:00
John Roesler adbf31ab1d KAFKA-6473: Add MockProcessorContext to public test-utils (#4736)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>
2018-03-27 14:03:24 -07:00
Matthias J. Sax 394aa74261
KAFKA-6454: Allow timestamp manipulation in Processor API (#4519)
Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-03-16 16:02:11 -07:00
huxi a9c16a8110 KAFKA-6663: Doc for `GlobalKTable` should be corrected. (#4723)
Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Damian Guy <damian@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-03-16 13:24:14 -07:00
huxi d2aca95aeb MINOR: Fix incorrect expression for KTable in stream doc. (#4718)
In dsi-api.html, when reading from Kafka to a KTable, the doc says "In the case of a KStream.." where `Kstream` here is not correct. They should be `KTable`.

Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-03-15 11:34:33 -07:00
Guozhang Wang 925acc9f47
KAFKA-4831: add documentation for KIP-265 (#4686)
Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>
2018-03-14 17:33:54 -07:00
Matthias J. Sax 0c00aa0983
MINOR: Streams doc example should not close store (#4667)
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-03-13 00:42:40 -07:00
Guozhang Wang 8e84961661
KAFKA-6560: Add docs for KIP-261 (#4685)
Reviewers: Matthias J. Sax <mjsax@apache.org>, Bill Bejeck <bill@confluent.io>
2018-03-12 13:11:29 -07:00
Joel Hamill 65aa1a0c3e KAFKA-6472 - Fix WordCount example code error (#4538)
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <mjsax@apache.org>
2018-02-12 16:54:52 -08:00
Matthias J. Sax 77a6104f54
KAFKA-3625: add docs for Kafka Streams test-utils (follow up) (#4493)
Adds web page docs for KIP-247

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>, Joel Hamill <joel@confluent.io>, Damian Guy <damian@confluent.io>
2018-02-08 13:35:57 -08:00
Matthias J. Sax a0fb4db40a
MINOR: improve security docs for Kafka Streams (#4532)
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Bill Bejeck <bill@confluent.io>, Joel Hamill <joel@confluent.io>, Guozhang Wang <guozhang@confluent.io>
2018-02-08 11:49:09 -08:00
Daniel Wojda d8eddc6e16 MINOR: Add missing imports to 'Hello Kafka Streams' examples (#4535)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2018-02-08 09:53:57 -08:00
Guozhang Wang 077fd9ced3
HOTFIX: Fix broken javadoc links on web docs(#4543)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2018-02-08 09:31:58 -08:00
Matthias J. Sax 164fea30ea
MINOR: update upgrade notes for Streams API; message format 0.10 requiered (#4500)
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io.>
2018-02-07 11:43:57 -08:00
Yu f6bbec4af2 KAFKA-6354: Update KStream JavaDoc using new State Store API (#4456)
Updates KStream JavaDoc and web page documentations using new State Store API

Author: Yu Liu <yu.liu003@gmail.com>

Reviewers: Guozhang Wang <guozhang@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2018-02-02 11:24:20 -08:00
Matthias J. Sax a78f66a5ae KAFKA-3625: Add public test utils for Kafka Streams (#4402)
* KAFKA-3625: Add public test utils for Kafka Streams
 - add new artifact test-utils
 - add TopologyTestDriver
 - add MockTime, TestRecord, add TestRecordFactory

Reviewers: Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>
2018-01-29 17:21:48 -08:00
Joel Hamill 6ea53d22e2 MINOR: Fix some streams web doc nits (#4411)
Reviewers: Derrick Or <derrickor@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
2018-01-29 10:09:32 -08:00
Yu-Jhe decfb834fb Add missing backslash (#4474)
There're missing backslash when running the quick start tutorial.

Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-01-26 10:14:10 -08:00
Matthias J. Sax d673c8cf94 MINOR: update upgrade notes with regard to KIP-149 (#4439)
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>
2018-01-26 09:18:22 -08:00
Matthias J. Sax 9ef883e62f MINOR: update docs with regard to improved resilience of Kafka Streams (#4380)
Reviewers: Guozhang Wang <wangguoz@gmail.com>, Joel Hamill, Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>
2018-01-22 14:37:19 -08:00
manjuapu b543bfc18e Streams Use case anchor (#4420)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-01-18 09:59:32 -08:00
Joel Hamill c7e7cc1a5a MINOR: Fix typo (#4426)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-01-16 10:30:43 -08:00
RichardYuSTUG 1d1c857596 4385
Author: RichardYuSTUG <yohan.richard.yu2@gmail.com>
Author: Prasanna Gautam <prasannagautam@gmail.com>
Author: Dong Lin <lindong28@gmail.com>
Author: Vahid Hashemian <vahidhashemian@us.ibm.com>
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #4385 from ConcurrencyPractitioner/doc-kafka-4499
2018-01-12 17:27:52 -08:00
Richard Yu b8aa1761c3 KAFKA-6265: Add #queryableStoreName() to GlobalKTable
A spinoff of original pull request #4340 for resolving conflicts.

Author: RichardYuSTUG <yohan.richard.yu2@gmail.com>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #4413 from ConcurrencyPractitioner/kafka-6265-2
2018-01-11 09:54:02 -08:00
Joel Hamill 06597ba26b MINOR: Menu updates and navigation (#4405)
* Menu updates and navigation

Reviewers: Guozhang Wang <wangguoz@gmail.com>
2018-01-09 16:28:49 -08:00
Jorge Quilcate Otoya 2900af6b33 Updating docs for streams app reset tool (#4401)
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
2018-01-08 12:02:29 -08:00
Joel Hamill 26270983cd MINOR: Fix menu ordering in streams docs
https://issues.apache.org/jira/browse/KAFKA-6419

Related: https://github.com/apache/kafka-site/pull/115

Author: Joel Hamill <joel-hamill@users.noreply.github.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #4390 from joel-hamill/fix-streams-menu-order
2018-01-08 11:43:38 -08:00
Joel Hamill d63f94797c MINOR: fix image 404s in streams doc
Fixes image 404s in streams docs (e.g. https://kafka.apache.org/documentation/streams/developer-guide/interactive-queries.html).

Related https://github.com/apache/kafka-site/pull/114

Author: Joel Hamill <joel-hamill@users.noreply.github.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #4389 from joel-hamill/fix-404-streams
2018-01-08 09:38:58 -08:00
Matt Farmer e1c5d0c119 MINOR: Add documentation for KAFKA-6086 (ProductionExceptionHandler) (#4395)
* Update streams documentation to describe production exception handler

* Add a mention of the ProductionExceptionHandler in the upgrade guide
2018-01-08 11:33:23 +00:00
Guozhang Wang 5995453f4c MINOR: Web docs for KIP-220
1. added functions for KafkaStreams and KafkaClientSupplier.
2. added prefix for admin client in StreamsConfig.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Jason Gustafson <jason@confluent.io>, Matthias J. Sax <matthias@confluent.io>

Closes #4338 from guozhangwang/K6150-doc-changes
2017-12-22 11:53:04 -08:00
Joel Hamill 3e2fe17c08 MINOR: Improve Streams Dev Guide content on web docs
This PR migrates content from CP Streams Dev Guide.

Here is the top-level page:
![image](https://user-images.githubusercontent.com/11722533/33904945-df9cf804-df31-11e7-93aa-52385961522c.png)

Here is a child page:
![image](https://user-images.githubusercontent.com/11722533/33904976-f2eafabe-df31-11e7-918c-fbf95db0f76b.png)

See related: https://github.com/apache/kafka-site/pull/112

Author: Joel Hamill <joel-hamill@users.noreply.github.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #4252 from joel-hamill/20171122-migrate-cp-dev-guide
2017-12-21 11:15:54 -08:00
Manjula K ca7f3dab04 MINOR: Adding secondary nav to Streams webpage
Added secondary navigation to Streams sub-pages:
 - quickstart.html
 - core-concepts.html
 - developer-guide.html
 - tutorial.html

Author: Manjula K <manjula@kafka-summit.org>

Reviewers: Joel Hamill <joel-hamill@users.noreply.github.com>, Guozhang Wang <wangguoz@gmail.com>

Closes #4222 from manjuapu/streams-nav
2017-11-15 14:43:11 -08:00
Guozhang Wang 487436b1a4 MINOR: Update docs for new version
1. Update the Streams hello world examples with the new API.
2. Update the version references in various places.
3. Update version templates to 1.1.x.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>, Derrick Or <derrickor@gmail.com>

Closes #4169 from guozhangwang/KMINOR-streams-docs
2017-11-03 08:51:44 -07:00
Manjula K 5178702715 MINOR: Adding Trivago logo to Streams landing page
guozhangwang Please review

Author: Manjula K <manjula@kafka-summit.org>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #4164 from manjuapu/ny-trivago-logos
2017-10-31 13:22:56 -07:00
Joel Hamill 5f779ca3f3 MINOR: Fix typo dev guide title
related to https://github.com/apache/kafka-site/pull/103

Author: Joel Hamill <git config --global user.email>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #4133 from joel-hamill/dev-guide-title
2017-10-25 10:11:09 -07:00
Guozhang Wang ef49145200 MINOR: a few web doc and javadoc fixes
1. Added missing Javadocs in public interfaces.
2. Added missing upgrade web docs.
3. Minor improvements on exception messages.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Antony Stubbs <antony.stubbs@gmail.com>

Closes #4071 from guozhangwang/KMinor-javadoc-gaps
2017-10-16 16:50:59 -07:00
Manjula K f6e724b127 MINOR: Redesign of Streams page to include video & logos
guozhangwang Please review.

Author: Manjula K <manjula@kafka-summit.org>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #4059 from manjuapu/redesign-streams-page
2017-10-11 15:16:12 -07:00
Bill Bejeck dac990aab3 KAFKA-6025: small fix for streams tutorial
Author: Bill Bejeck <bill@confluent.io>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #4053 from bbejeck/KAFKA_6025_fix_streams_tutorial
2017-10-10 15:07:53 -07:00
Guozhang Wang e2e8d4a57a MINOR: KIP-161 upgrade docs change
Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Damian Guy <damian.guy@gmail.com>

Closes #4036 from guozhangwang/KMinor-kip-161-docs
2017-10-06 17:57:32 -07:00
Joel Hamill ff5fc9dd1e Rename streams tutorial and quickstart
Changed these topic titles:
- Write your own Streams Applications -> Tutorial: Write a Streams Application
- Play with a Streams Application -> Run the Streams Demo Application

Author: Joel Hamill <joel@Joel-Hamill-Confluent.local>
Author: Joel Hamill <11722533+joel-hamill@users.noreply.github.com>

Reviewers: Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #4017 from joel-hamill/joel-hamill/streams-titles
2017-10-05 09:30:38 -07:00
Damian Guy 93b71e7dee MINOR: update streams quickstart for KIP-182
Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Michael G. Noll <michael@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3984 from dguy/quickstart-update
2017-10-04 12:19:40 -07:00
Joel Hamill d985513b22 MINOR: streams dev guide fixup
Author: Joel Hamill <joel@Joel-Hamill-Confluent.local>
Author: Joel Hamill <11722533+joel-hamill@users.noreply.github.com>

Reviewers: Derrick Or <derrickor@gmail.com>, Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3862 from joel-hamill/joel-hamill/streams-dev-guide
2017-10-04 11:28:59 -07:00
Damian Guy cc84686a4a MINOR: additional kip-182 doc updates
Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Michael G. Noll <michael@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk>

Closes #3971 from dguy/kip-182-docs
2017-10-02 13:20:49 -07:00
Bill Bejeck 39d5cdcccf KAFKA-5985; update javadoc regarding closing iterators
Author: Bill Bejeck <bill@confluent.io>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>

Closes #3994 from bbejeck/KAFKA-5985_document_need_to_close_iterators
2017-10-02 11:49:22 -07:00
Manjula K 5b943ca8a9 MINOR:Updated Rabobank description
dguy Please review

Author: Manjula K <manjula@kafka-summit.org>
Author: manjuapu <manjula@confluent.io>

Reviewers: Damian Guy <damian.guy@gmail.com>

Closes #3963 from manjuapu/customer-logo-stream
2017-09-27 09:26:57 +01:00
Richard Yu b8be86b805 KAFKA-5765; Move merge() from StreamsBuilder to KStream
This is the polished version.
1. The old merge() method in StreamsBuilder has been removed,
2. The merge() method in KStreamBuilder was changed so that it would use the single variable argument
rather than several variable arguments in the KStreamImpl implementation
3. The merge() method in KStream has been declared as final and tests have been added to test correctness.

Author: Richard Yu <richardyu@Richards-Air.attlocal.net>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>, Damian Guy <damian.guy@gmail.com>

Closes #3916 from ConcurrencyPractitioner/trunk
2017-09-26 09:42:53 +01:00
Matthias J. Sax a96e28eac1 MINOR: Update Streams quickstart to create output topic with compaction enabled
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3949 from mjsax/minor-update-streams-quickstart
2017-09-25 16:17:01 +08:00
Manjula K 1fd70c7c94 MINOR - Adding New York Times logo to streams page
Author: Manjula K <manjula@kafka-summit.org>
Author: manjuapu <manjula@confluent.io>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #3950 from manjuapu/customer-logo-stream
2017-09-23 10:53:10 +08:00
Manjula K 36afd1095c MINOR: Adding LINE corp logo to streams page
Author: Manjula K <manjula@kafka-summit.org>
Author: manjuapu <manjula@confluent.io>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #3940 from manjuapu/customer-logo-stream
2017-09-22 07:11:06 +08:00
Damian Guy f29391c490 MINOR: add upgrade note for KIP-173 topic configs
Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3921 from dguy/minor-kip-173-docs
2017-09-21 18:19:24 +08:00
manjuapu c82be0f303 MINOR: Adding See how Kafka Streams is being used section to Streams page
Author: manjuapu <manjula@confluent.io>
Author: Manjula K <manjula@kafka-summit.org>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #3914 from manjuapu/customer-logo-stream
2017-09-20 08:59:05 -07:00
Florian Hussonnois 398714b758 KAFKA-5839: Upgrade Guide doc changes for KIP-130
Author: Florian Hussonnois <florian.hussonnois@gmail.com>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3811 from fhussonnois/KAFKA-5839

minor fixes
2017-09-20 16:50:08 +08:00
Jakub Scholz a64fe2ed88 KAFKA-5918: Fix minor typos and errors in the Kafka Streams turotial
I found several minor issues with the Kafka Streams tutorial:
* Some typos
  * "As shown above, it illustrate that the constructed ..." instead of "As shown above, it illustrate_s_ that the constructed ..."
  * "same as Pipe.java below" instead of "same as Pipe.java _above_"
  * Wrong class name in the `LineSplit` example
* Incorrect imports for the code examples
  * Missing `import org.apache.kafka.streams.kstream.KStream;` in `LineSplit` and `WordCount` example
* Unnecessary (and potentially confusing) split by whitespaces in the `WorkCount` class (the split into words happened already in `LineSplit`)

Author: Jakub Scholz <www@scholzj.com>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3883 from scholzj/stream-tutorial-typos
2017-09-19 07:58:06 +08:00
Damian Guy f7b1add686 MINOR: update tutorial doc to match ak-site
Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #3847 from dguy/minor-doc-update
2017-09-13 13:22:06 -07:00
Guozhang Wang 9d2437a464 MINOR: Un-hide the tutorial buttons on web docs
Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Damian Guy <damian.guy@gmail.com>

Closes #3850 from guozhangwang/KMinor-unhide-button
2017-09-13 13:12:09 -07:00
Damian Guy 2db1e4423f KAFKA-5852: Add filter, filterNot, mapValues and Materialized to KTable
Add overloads of `filter`, `filterNot`, `mapValues` that take `Materialized` as a param to `KTable`. Deprecate overloads using `storeName` and `storeSupplier`

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3807 from dguy/ktable-filter-map
2017-09-08 14:01:58 -07:00
Damian Guy 4769e3d92a KAFKA-5815; add Printed class and KStream#print(printed)
Part of KIP-182
- Add `Printed` class and `KStream#print(Printed)`
- deprecate all other `print` and `writeAsText` methods

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3768 from dguy/kafka-5652-printed
2017-09-08 18:22:04 +01:00
Damian Guy e16b9143df KAFKA-5853; implement WindowedKStream
Add the `WindowedKStream` interface and implementation of methods that don't require `Materialized`

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3809 from dguy/kgrouped-stream-windowed-by
2017-09-08 16:49:18 +01:00
Damian Guy d0ee6ed36b KAFKA-5832; add Consumed and change StreamBuilder to use it
Added `Consumed` class.
Updated `StreamBuilder#stream`, `StreamBuilder#table`, `StreamBuilder#globalTable`

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3784 from dguy/kip-182-stream-builder
2017-09-08 08:21:48 +01:00
Damian Guy 9cbb9f0939 KAFKA-5650; add StateStoreBuilder interface and implementations
Part of KIP-182

- Add `StateStoreBuilder` interface and `WindowStateStoreBuilder`, `KeyValueStateStoreBuilder`, and `SessionStateStoreBuilder` implementations
- Add `StoreSupplier`, `WindowBytesStoreSupplier`, `KeyValueBytesStoreSupplier`, `SessionBytesStoreSupplier` interfaces and implementations
- Add new methods to `Stores` to create the newly added `StoreSupplier` and `StateStoreBuilder` implementations
- Update `Topology` and `InternalTopology` to use the interfaces

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3767 from dguy/kafka-5650
2017-09-07 09:39:46 +01:00
Damian Guy 667cd60dc6 KAFKA-5816; add Produced class, KStream#to(topic, Produced), and KStream#through(topic, Produced)
Add the `Produced` class and `KStream` overloads that use it:
`KStream#to(String, Produced)`
`KStream#through(String, Produced)`
Deprecate all other to and through methods accept the single param methods that take a topic param

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3770 from dguy/kafka-5652-produced
2017-09-07 08:54:10 +01:00
Boyang Chen b041c8d87d MINOR: close iterator on doc example
The iterator interface usage has some examples missing explicit close operation after usage. We should remind the user to do so because un-closed iterator will leave the underlying file descriptor open, thus eating up memory.
guozhangwang Ishiihara

Author: Boyang Chen <bychen@pinterest.com>
Author: Boyang Chen <bchen11@outlook.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #3714 from abbccdda/add_iterator_close_on_doc

minor fixes
2017-09-06 22:12:10 -07:00
Guozhang Wang 9b85cf9ed0 MINOR: KIP-138 renaming of string names
Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Damian Guy <damian.guy@gmail.com>

Closes #3796 from guozhangwang/kip-138-minor-renames
2017-09-06 18:39:40 -07:00
Matthias J. Sax 212bce6e3f MINOR: fix typos on web doc's upgrade guide
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #3804 from mjsax/hotfix
2017-09-06 15:42:52 -07:00
Damian Guy b687c06800 KAFKA-5817; Add Serialized class and overloads to KStream#groupBy and KStream#groupByKey
Part of KIP-182
- Add the `Serialized` class
- implement overloads of `KStream#groupByKey` and KStream#groupBy`
- deprecate existing methods that have more than default arguments

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3772 from dguy/kafka-5817
2017-09-06 10:43:14 +01:00
Matthias J. Sax cd59976eea KAFKA-5823: Extend upgrade section for KIP-120
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Sharon Liu <sharonliu.cup@gmail.com>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>

Closes #3787 from mjsax/kafka-5823-kip120-docs
2017-09-05 16:54:00 -07:00
Eno Thereska 493c2aad59 MINOR: add table of contents
Added a simple table of contents for the developer section.

Author: Eno Thereska <eno.thereska@gmail.com>

Reviewers: Damian Guy <damian.guy@gmail.com>

Closes #3760 from enothereska/minor-docs-toc
2017-09-04 14:28:50 +01:00
Guozhang Wang ce04b14358 MINOR: KIP-160 docs
Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Damian Guy <damian.guy@gmail.com>

Closes #3733 from guozhangwang/KMinor-kip160-docs
2017-08-30 11:17:55 -07:00
Guozhang Wang f5310d645c MINOR: doc changes for KIP-138
1. Core concepts (added the stream time definition), upgrade guide and developer guide.
2. Related Java docs changes.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Damian Guy <damian.guy@gmail.com>

Closes #3732 from guozhangwang/KMinor-kip138-docs
2017-08-30 09:45:16 -07:00
Eno Thereska bd54d2e3e0 MINOR: stateful docs for aggregates
Author: Eno Thereska <eno.thereska@gmail.com>

Reviewers: Damian Guy <damian.guy@gmail.com>

Closes #3730 from enothereska/minor-docs-aggregates
2017-08-30 10:09:08 +01:00
Michal Borowiecki 5648dcc3e5 MINOR: Fix doc typos and grammar
This is contributed by mihbor on various doc fixes including:

https://github.com/apache/kafka/pull/3224
https://github.com/apache/kafka/pull/3226
https://github.com/apache/kafka/pull/3229

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>

Closes #3746 from guozhangwang/KMinor-doc-typos
2017-08-26 16:33:33 -07:00
Guozhang Wang 21fea170d8 MINOR: add upgrade section for 1.0.0
1. Add upgrade section for 1.0.0, including Streams API changes section.
2. Add metrics name changes section.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>

Closes #3687 from guozhangwang/KMinor-metrics-upgrade-guide
2017-08-24 09:12:12 -07:00
Eno Thereska 5e65c0da45 MINOR: Stateless transformation documentation
Needs to come after https://github.com/apache/kafka/pull/3701
Originally reviewed as part of #3490.

Author: Eno Thereska <eno.thereska@gmail.com>

Reviewers: Damian Guy <damian.guy@gmail.com>

Closes #3711 from enothereska/minor-docs-stateless
2017-08-24 14:53:39 +01:00
Eno Thereska ecea150bde MINOR: Document how to create source streams and tables
Originally reviewed as part of https://github.com/apache/kafka/pull/3490.

Author: Eno Thereska <eno.thereska@gmail.com>

Reviewers: Damian Guy <damian.guy@gmail.com>

Closes #3701 from enothereska/minor-docs-create-source-streams
2017-08-22 14:14:35 +01:00
Guozhang Wang b6102cf4b2 HOTFIX: missing imports and version on web docs 2017-08-11 13:53:47 -07:00
Guozhang Wang d2774e302f KAFKA-5727: Add Streams quickstart tutorial as an archetype project
0. Minor fixes on the existing examples to merge all on a single input topic; also do not use `common.utils.Exit` as it is for internal usage only.

1. Add the archetype project for the quickstart. Steps to try it out:

  a. `mvn install` on the quickstart directory.
  b. `mvn archetype:generate \
-DarchetypeGroupId=org.apache.kafka \
-DarchetypeArtifactId=streams-quickstart-java \
-DarchetypeVersion=1.0.0-SNAPSHOT \
-DgroupId=streams-quickstart \
-DartifactId=streams-quickstart \
-Dversion=0.1 \
-Dpackage=StreamsQuickstart \
-DinteractiveMode=false` at any directory to create the project.
  c. build the streams jar with version `1.0.0-SNAPSHOT` to local maven repository with `./gradlew installAll`; `cd streams-quickstart; mvn clean package`
  d. create the input / output topics, start the console producer and consumer.
  e. start the program: `mvn exec:java -Dexec.mainClass=StreamsQuickstart.Pipe/LineSplit/WordCount`.
  f. type data on console producer and observe data on console consumer.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Ewen Cheslack-Postava <me@ewencp.org>, Eno Thereska <eno.thereska@gmail.com>

Closes #3630 from guozhangwang/KMinor-streams-quickstart-tutorial
2017-08-11 12:19:28 -07:00
Bill Bejeck b2b529522d MINOR: First cut at porting State Store docs to AK
Author: Bill Bejeck <bill@confluent.io>

Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>

Closes #3629 from bbejeck/docs-updates-for-kip-167
2017-08-11 09:35:26 +01:00
Damian Guy 2010aa067f MINOR: add memory management section to streams docs
Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Guozhang Wang <wangguoz@gmail.com>, Eno Thereska <eno.thereska@gmail.com>

Closes #3604 from dguy/memory-management-docs
2017-08-07 10:17:17 +01:00
Guozhang Wang 125d69caee KAFKA-5671 Followup: Remove reflections in unit test classes
1. Remove rest deprecation warnings in streams:jar.

2. Consolidate all unit test classes' reflections to access internal topology builder from packages other than `o.a.k.streams`. We need to refactor the hierarchies of StreamTask, StreamThread and KafkaStreams to remove these hacky reflections.

3. Minor fixes such as reference path, etc.

4. Minor edits on web docs for the describe function under developer-guide.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Bill Bejeck <bill@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Damian Guy <damian.guy@gmail.com>

Closes #3603 from guozhangwang/K5671-followup-comments
2017-08-02 15:13:02 -07:00
Matthias J. Sax da2205578b KAFKA-5671: Add StreamsBuilder and Deprecate KStreamBuilder
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3602 from mjsax/kafka-5671-add-streamsbuilder
2017-07-31 15:28:59 -07:00
Matthias J. Sax 1844bf2b2f KAFKA-5670: (KIP-120) Add Topology and deprecate TopologyBuilder
Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3590 from mjsax/kafka-3856-replace-topology-builder-by-topology
2017-07-28 16:46:34 -07:00
Guozhang Wang 91c207c2c6 MINOR: Make streams quick start more interactive
1. Make the WordCountDemo application to not stop automatically but via "ctrl-C".
2. Update the quickstart html file to let users type input messages one-by-one, and observe added output in an interactive manner.
3. Some minor fixes on the parent documentation page pointing to streams sub-pages, added a new recommended Scala version number.

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>

Closes #3515 from guozhangwang/KMinor-interactive-quickstart
2017-07-25 11:34:16 -07:00
Derrick Or 300f48018c MINOR: Add Streams landing page
Content and assets for the updated Streams API landing page

Author: Derrick Or <derrickor@gmail.com>

Reviewers: Michael G. Noll <michael@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3540 from derrickdoo/streams-landing-page
2017-07-21 19:16:14 -07:00
Damian Guy 1110f66fa7 KAFKA-3741; allow users to specify default topic configs for internal topics
Allow users to specify default topic configs for streams internal topics by supplying properties from `TopicConfig` with a prefix.
Supplied defaults are used when creating the internal topics. They are overridden by the configs supplied along with the `InternalTopicConfig`

Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3459 from dguy/kafka-3741
2017-07-21 12:15:40 +01:00
Guozhang Wang 6a5cd67b9b HOTFIX: fix a few typos on streams quickstart
Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Eno Thereska <eno.thereska@gmail.com>, Damian Guy <damian.guy@gmail.com>

Closes #3510 from guozhangwang/KHotfix-streams-quickstart
2017-07-10 10:16:02 -07:00
Eno Thereska 7bfe008ae1 MINOR: Move quickstart under streams
Author: Eno Thereska <eno.thereska@gmail.com>

Reviewers: Michael G. Noll <michael@confluent.io>, Damian Guy <damian.guy@gmail.com>, Guozhang Wang <wangguoz@gmail.com>

Closes #3494 from enothereska/minor-quickstart-docs

rename section name and bold font for section names
2017-07-09 17:35:14 -07:00
Damian Guy 75f1b757e1 HOTFIX: fix paths in streams index
Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Ismael Juma <ismael@juma.me.uk>

Closes #3502 from dguy/doc-fixes
2017-07-07 15:10:16 +01:00
Damian Guy 6cae5ec668 MINOR: add IQ docs to streams documentation
Author: Damian Guy <damian.guy@gmail.com>

Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>

Closes #3484 from dguy/iq-doc
2017-07-05 14:15:43 -07:00
Guozhang Wang 2c6e9a9653 MINOR: Follow-up Streams doc changes to break into sub-pages
Also fixed a bunch of broken links (details can be found in 34f8ecea0d)

Author: Guozhang Wang <wangguoz@gmail.com>

Reviewers: Eno Thereska <eno.thereska@gmail.com>, Bill Bejeck <bbejeck@gmail.com>, Damian Guy <damian.guy@gmail.com>

Closes #3473 from guozhangwang/KMinor-streams-doc-breakdown
2017-07-03 15:35:47 +01:00