Commit Graph

1080 Commits

Author SHA1 Message Date
Jason Gustafson 017868d8ac
MINOR: Add section on listener configuration (including kraft) to security docs (#12682)
This patch adds a section in security.html about listener configuration. This includes the basics of how to define the security mapping of each listener as well as the configurations to control inter-cluster traffic.

Reviewers: José Armando García Sancio <jsancio@users.noreply.github.com>, Luke Chen <showuon@gmail.com>
2022-09-27 10:33:32 +08:00
Jason Gustafson 1c0f8f90e2
MINOR: Update design docs to avoid zookeeper-specific assumptions (#12690)
Update a few cases in the documentation which do not make sense for KRaft.

Reviewers: José Armando García Sancio <jsancio@users.noreply.github.com>
2022-09-26 13:01:07 -07:00
José Armando García Sancio 4dec656699
KAFKA-14207; KRaft Operations documentation (#12642)
Reviewers: Colin Patrick McCabe <cmccabe@apache.org>, Chase Thomas <forlack@users.noreply.github.com>
2022-09-26 11:19:48 -07:00
Niket eb8f0bd5e4
MINOR: Adding KRaft Monitoring Related Metrics to docs/ops.html (#12679)
This commit adds KRaft monitoring related metrics to the Kafka docs (docs/ops.html).

Reviewers: Jason Gustafson <jason@confluent.io>, Luke Chen <showuon@gmail.com>
2022-09-26 14:25:36 +08:00
Vicky Papavasileiou cda5da9b65
KAFKA-14209: Change Topology optimization to accept list of rules 1/3 (#12641)
This PR is part of a series implementing the self-join rewriting. As part of it, we decided to clean up the TOPOLOGY_OPTIMIZATION_CONFIG and make it a list of optimization rules. Acceptable values are: NO_OPTIMIZATION, OPTIMIZE which applies all optimization rules or a comma separated list of specific optimizations.

Reviewers: Guozhang Wang <guozhang@apache.org>, John Roesler <vvcephei@apache.org>
2022-09-22 11:20:37 -05:00
Jason Gustafson 3549a5524e
MINOR: Update security docs for kraft Authorizer configuration (#12673)
Update security documentation to describe how to configure the KRaft `Authorizer` implementation and include a note about principal forwarding.

Additionally, this patch renames `KafkaConfig.Defaults.DefaultPrincipalSerde` to `DefaultPrincipalBuilder` since the former is somewhat misleading.

Reviewers: David Arthur <mumrah@gmail.com>
2022-09-21 19:38:59 -07:00
Jason Gustafson 695424fa9d
MINOR: Mention deprecation of authorizer flags in security documentation (#12668)
The following options are deprecated in kafka-acls.sh: `--authorizer`, `--authorizer-properties`, and `--zk-tls-config-file`. This patch updates the security documentation to mention the deprecation and changes examples to use `--bootstrap-server` when possible.

Reviewers: Luke Chen <showuon@gmail.com>
2022-09-21 09:34:17 -07:00
Federico Valeri 6d463c1733
MINOR: Update offset.storage.topic description (#12656)
Reviewers: Chris Egerton <chrise@aiven.io>
2022-09-19 13:21:59 -04:00
Nikolay 51b079dca7
KAFKA-12878: Support --bootstrap-server in kafka-streams-application-reset tool (#12632)
Reviewers: Chris Egerton <chrise@aiven.io>
2022-09-19 13:20:41 -04:00
José Armando García Sancio fc6a814391
MINOR; Add missing li end tag (#12640)
Reviewers: Ismael Juma <ismael@juma.me.uk>
2022-09-14 08:55:28 -07:00
Ismael Juma fcbc9c49d0
MINOR: Mention that kraft is production ready in upgrade notes (#12635)
Reviewers: José Armando García Sancio <jsancio@apache.org>
2022-09-14 08:30:39 -07:00
Artem Livshits 7b49f175b9
MINOR: Add upgrade note regarding the Strictly Uniform Sticky Partitioner (KIP-794) (#12630)
Reviewers: Ismael Juma <ismael@juma.me.uk>, David Jacot <djacot@confluent.io>
2022-09-13 11:56:44 -07:00
José Armando García Sancio bc0b868cd4
KAFKA-14205; Document how to replace the disk for the KRaft Controller (#12597)
Document process for recovering and formatting the metadata log directory for the KRaft controller.

Reviewers: Colin Patrick McCabe <cmccabe@apache.org>, Jason Gustafson <jason@confluent.io>
2022-09-12 16:57:54 -07:00
Colin Patrick McCabe 26e3b813af
KAFKA-14217: app-reset-tool.html should not show --zookeeper flag that no longer exists (#12618)
Reviewers: Luke Chen <showuon@gmail.com>
2022-09-12 08:33:01 -07:00
José Armando García Sancio 12c0ddc9c0
KAFKA-14188; Getting started for Kafka with KRaft (#12604)
Update the quickstart HTML pages for Kafka and Kafka Stream to include how to quickly start and
experiment with a Kafka cluster using KRaft in addition to ZooKeeper.

Reviews: Colin Patrick McCabe <cmccabe@apache.org>,  Chase Thomas <forlack@users.noreply.github.com>, Luke Chen <showuon@gmail.com>
2022-09-08 16:22:09 -07:00
José Armando García Sancio 6397d50396 Remove the html end tag from upgrade.html 2022-09-07 16:04:19 -07:00
Colin Patrick McCabe 28d5a05943
KAFKA-14187: kafka-features.sh: add support for --metadata (#12571)
This PR adds support to kafka-features.sh for the --metadata flag, as specified in KIP-778.  This
flag makes it possible to upgrade to a new metadata version without consulting a table mapping
version names to short integers. Change --feature to use a key=value format.

FeatureCommandTest.scala: make most tests here true unit tests (that don't start brokers) in order
to improve test run time, and allow us to test more cases. For the integration test part, test both
KRaft and ZK-based clusters. Add support for mocking feature operations in MockAdminClient.java.

upgrade.html: add a section describing how the metadata.version should be upgraded in KRaft
clusters.

Add kraft_upgrade_test.py to test upgrades between KRaft versions.

Reviewers: David Arthur <mumrah@gmail.com>, dengziming <dengziming1993@gmail.com>, José Armando García Sancio <jsancio@gmail.com>
2022-08-30 16:56:03 -07:00
José Armando García Sancio 4a870c0335
MINOR; Update upgrade documentation for 3.3 (#12550) 2022-08-25 17:27:14 -07:00
Bill Bejeck 008d1afc4d
MINOR: Update site docs for ASF compliance (#12494)
This PR is a mirror of apache/kafka-site#433 which used placeholder images for the Kafka Streams that users need to click in order to load the iframe with the corresponding video.

Reviewers: Mickael Maison <mimaison@apache.org>
2022-08-24 10:45:32 -04:00
Janik Dotzel ae3f48b699
MINOR: Fix invalid link to plugin.path property docs in quickstart (#12523)
Reviewers: Chris Egerton <fearthecellos@gmail.com>
2022-08-17 13:25:35 -04:00
Yash Mayya 18142bff4b
KAFKA-13809: Propagate full connector configuration to tasks in FileStream connectors (#12450)
Reviewers: Chris Egerton <fearthecellos@gmail.com>
2022-08-15 15:25:29 -04:00
Derek Troy-West 2539be9086
MINOR: Add note on IDEMPOTENT_WRITE ACL to notable changes (#12260)
Update notable changes documentation to mention requiring IDEMPOTENT_WRITE permission
when producing messages with default/idempotent configuration and broker version lower than
2.8.0.

Reviewers: Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>
2022-08-12 17:41:44 -07:00
José Armando García Sancio 6ace67b2de
MINOR; Bump trunk to 3.4.0-SNAPSHOT (#12463)
Version bumps in trunk after the creation of the 3.3 branch.

Reviewers: David Arthur <mumrah@gmail.com>
2022-08-01 09:54:12 -07:00
Divij Vaidya a63f464f5e
KAFKA-13868: Replace YouTube embedded video with links on streams page (#12438)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2022-07-26 16:59:19 +02:00
Mickael Maison df899a2d08
MINOR: Fix broken link to Streams tutorial (#12426)
Also fix Transforming Data Pt. 2 video title

Reviewer: Bruno Cadonna <cadonna@apache.org>
2022-07-21 15:09:31 +02:00
Federico Valeri 74a18bafae
Minor: replace .kafka with .log in implementation documentation (#12401)
replace .kafka with .log in implementation documentation

Reviewers: Luke Chen <showuon@gmail.com>, Liam Clarke-Hutchinson <liam@steelsky.co.nz>
2022-07-20 18:11:51 +08:00
Christo Lolov 94d4fdeb28
KAFKA-14008: Add docs for Streams throughput metrics introduced in KIP-846 (#12377)
Reviewers: Walker Carlson <wcarlson@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2022-07-13 17:47:34 -07:00
RivenSun 7ec759d67c
MINOR: Mention switch to reload4j in Notable changes in 3.1.1 (#12313)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Kvicii
2022-07-12 16:51:18 +02:00
Marco Aurelio Lotz 63a6130af3
KAFKA-12943: update aggregating documentation (#12091)
Reviewers: Luke Chen <showuon@gmail.com>, Andrew Eugene Choi <andrew.choi@uwaterloo.ca>, Matthias J. Sax <matthias@confluent.io>
2022-07-07 14:00:05 -07:00
vamossagar12 5a1bac2608
KAFKA-13846: Follow up PR to address review comments (#12297)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2022-07-07 11:43:38 -07:00
Viktor Somogyi-Vass 277c4c2e97
KAFKA-6945: Add docs about KIP-373 (#12346)
Reviewers: Manikumar Reddy
2022-07-05 17:29:31 +05:30
RivenSun bad475166f
MINOR: Add indent space after hyperlink in `docs/upgrade.html` (#12353)
Reviewers: Kvicii <42023367+Kvicii@users.noreply.github.com>, Divij Vaidya <divijvaidya13@gmail.com>, Jason Gustafson <jason@confluent.io>
2022-07-01 10:30:32 -07:00
RivenSun 35ee09d43f
MINOR: Remove extra commas in upgrade steps documentation (#12311)
Reviewers: Luke Chen <showuon@gmail.com>
2022-06-20 16:57:49 +08:00
Lee Dongjin 89c057397e
KAFKA-13436: Omitted BrokerTopicMetrics metrics in the documentation (#11473)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2022-06-13 16:46:19 +02:00
Mickael Maison 4a06458633
KAFKA-13780: Generate OpenAPI file for Connect REST API (#12067)
New gradle task `connect:runtime:genConnectOpenAPIDocs` that generates `connect_rest.yaml` under `docs/generated`.
This task is executed when `siteDocsTar` runs.
2022-06-10 11:35:22 +02:00
Luke Chen 4f40883d02
MINOR: add java 8/scala 2.12 deprecation info in doc (#12261)
Reviewers: Ismael Juma <mlists@juma.me.uk>
2022-06-09 09:39:23 +08:00
A. Sophie Blee-Goldman a6c5a74fdb
KAFKA-13945: add bytes/records consumed and produced metrics (#12235)
Implementation of KIP-846: Source/sink node metrics for Consumed/Produced throughput in Streams

Adds the following INFO topic-level metrics for the total bytes/records consumed and produced:

    bytes-consumed-total
    records-consumed-total
    bytes-produced-total
    records-produced-total

Reviewers: Kvicii <Karonazaba@gmail.com>, Guozhang Wang <guozhang@apache.org>, Bruno Cadonna <cadonna@apache.org>
2022-06-07 16:02:17 +02:00
Richard Joerger 6ff2bf03a9
KAFKA-13875 Adjusted the output the topic describe output to include TopicID & se… (#12170)
Reviewers: Luke Chen <showuon@gmail.com>
2022-06-05 15:13:02 +08:00
JK-Wang 04f23688e7
MINOR: Fix plugin path in quickstart.html (#12252)
Reviewers: Luke Chen <showuon@gmail.com>
2022-06-05 15:08:37 +08:00
Okada Haruki a39d447677
MINOR: fix streams tutorial (#12251)
Reviewers: Luke Chen <showuon@gmail.com>
2022-06-04 13:23:11 +08:00
RivenSun d8d92f0f80
MINOR: Update the kafka-reassign-partitions script command in documentation (#12237)
Reviewers: Luke Chen <showuon@gmail.com>
2022-06-02 21:30:22 +08:00
Mickael Maison e4f4e50b7b
MINOR: Small fixes in docs/upgrade.html (#12239)
Reviewers: David Jacot <djacot@confluent.io>
2022-06-02 12:05:30 +02:00
Amir M. Saeid 8a05884343
MINOR: Fix typo in processor api developer guide (#12203)
The reference to `changlogConfig` should be `changelogConfig`.

Reviewers: Jason Gustafson <jason@confluent.io>
2022-05-30 15:57:46 -07:00
Milind Mantri 07459d2eb3
Fix DSL typo in streams docs (#12152)
Reviewer: Bruno Cadonna <cadonna@apache.org>
2022-05-17 11:19:26 +02:00
xjin-Confluent 354a4e9470
Update note on upgrade from log4j to reload4j (#12164)
Reviewers: Nikhil Bhatia <rite2nikhil@gmail.com>, Bruno Cadonna <cadonna@apache.org>
2022-05-16 13:23:10 +02:00
Dejan Maric 1c02a764ec
KAFKA-12703; Allow unencrypted private keys when using PEM files (#11916)
Reviewers: David Jacot <djacot@confluent.io>
2022-05-16 09:25:05 +02:00
Bruno Cadonna 816d63b908
MINOR: Fix link to old doc in quickstart (#12129)
In Kafka's quickstart a link points to the 2.5 Kafka Streams demo.
This PR fixes this link.
2022-05-06 11:43:30 +02:00
Bruno Cadonna 3d087244d8
MINOR: Note that slf4j-log4j in version 1.7.35+ should be used (#12114)
Adds a note to the upgrade notes to use slf4j-log4j version
1.7.35+ [1] or slf4j-reload4j to avoid possible compatibility issues
originating from the logging framework [2].

[1] https://www.slf4j.org/manual.html#swapping
[2] https://www.slf4j.org/codes.html#no_tlm

Reviewer: Ismael Juma <ismael@juma.me.uk>
2022-05-03 12:24:53 +02:00
RivenSun 53ad99309d
MINOR: Improve document description in zero-copy (#12099)
Follow up of #12052 to improve the description.

Reviewer: David Jacot <djacot@confluent.io>
2022-04-29 17:44:39 +02:00
Hongten ff3d42a18c
KAFKA-13852: Kafka Acl documentation bug for wildcard '*' (#12090)
The wildcard * in command without wrapped by single quote will be replaced into the file name under the current folder by bash. So we need to wrap with single quote. Update the doc and command option description.

Reviewers: dengziming <dengziming1993@gmail.com>, Luke Chen <showuon@gmail.com>
2022-04-24 16:50:44 +08:00
RivenSun fb66b3668b
KAFKA-13799: Improve documentation for Kafka zero-copy (#12052)
Improve documentation for Kafka zero-copy. Kafka combines pagecache and zero-copy to greatly improve message consumption efficiency. But zero-copy only works in PlaintextTransportLayer.

Reviewers: Divij Vaidya <divijvaidya13@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
2022-04-20 10:31:32 -07:00
Aleksandr Sorokoumov adf5cc5371
KAFKA-13769: Explicitly route FK join results to correct partitions (#11945)
Prior to this commit FK response sink routed FK results to
SubscriptionResolverJoinProcessorSupplier using the primary key.

There are cases, where this behavior is incorrect. For example,
if KTable key serde differs from the data source serde which might
happen without a key changing operation.

Instead of determining the resolver partition by serializing the PK
this patch includes target partition in SubscriptionWrapper payloads.
Default FK response-sink partitioner extracts the correct partition
from the value and routes the message accordingly.

Reviewers: Matthias J. Sax <matthias@confluent.io>
2022-04-15 11:28:43 -07:00
Bruno Cadonna 2d2926cf81
MINOR: Make link in quickstart dynamic (#12057)
Reviewer: Matthias J. Sax <mjsax@apache.org>
2022-04-15 16:22:16 +02:00
RivenSun 19b5853565
MINOR: Improve the description of principal under different mechanisms of sasl (#11947)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2022-04-15 11:09:20 +02:00
Bruno Cadonna b7f2d979d6
[MINOR] Update upgrade documentation for 3.2 (#12055)
Reviewer: Bruno Cadonna <cadonna@apache.org>
2022-04-15 10:21:05 +02:00
Chris Egerton b9fc893546
MINOR: Correct Connect docs on connector/task states (#11914)
The `DESTROYED` state is represented internally as a tombstone record when running in distributed mode and by the removal of the connector/task from the in-memory status map when running in standalone mode. As a result, it will never appear to users of the REST API, and we should remove mention of it from our docs so that developers creating tooling against the REST API don't write unnecessary logic to account for that state.

Reviewers: Mickael Maison <mickael.maison@gmail.com>
2022-04-13 12:51:34 +02:00
Tom Bentley 9b8f6289be
MINOR: Mention KAFKA-13748 in release notes (#11994)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Bruno Cadonna <bruno@confluent.io>
2022-04-06 10:29:44 +01:00
RivenSun 74909e000a
MINOR: Fix wrong configuration in Adding and Removing Listeners docs (#11992)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2022-04-04 17:27:34 +02:00
Mickael Maison dd4afdeb6d
MINOR: Doc updates for Kafka 3.0.1 (#11906)
Reviewers: David Jacot <djacot@confluent.io>
2022-04-04 14:43:31 +02:00
Konstantine Karantasis dd62ef2eda
KAFKA-13748: Do not include file stream connectors in Connect's CLASSPATH and plugin.path by default (#11908)
With this change we stop including the non-production grade connectors that are meant to be used for demos and quick starts by default in the CLASSPATH and plugin.path of Connect deployments. The package of these connector will still be shipped with the Apache Kafka distribution and will be available for explicit inclusion. 

The changes have been tested through the system tests and the existing unit and integration tests. 

Reviewers: Mickael Maison <mickael.maison@gmail.com>, Randall Hauch <rhauch@gmail.com>
2022-03-30 13:15:42 -07:00
Mike Lothian ecb0e8eece
KAFKA-13660: Switch log4j12 to reload4j (#11743)
This bumps the slf4j version to 1.7.36 and swaps out log4j 1.2.17 with
reload4j 1.2.19

Signed-off-by: Mike Lothian <mike@fireburn.co.uk>

Reviewers: Luke Chen <showuon@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Bruno Cadonna <cadonna@apache.org>
2022-03-30 20:54:01 +02:00
Levani Kokhreidze 35ae4f248b
KAFKA-6718: Add documentation for KIP-708 (#11923)
Adds documentation for KIP-708: Rack awareness for Kafka Streams

Co-authored-by: Bruno Cadonna <cadonna@apache.org>

Reviewers: Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>
2022-03-29 14:08:51 +02:00
Luke Chen ca3271e0f3
MINOR: replace kafka-preferred-replica-election.sh with kafka-leader-election.sh (#11954)
Reviewers: Luke Chen <showuon@gmail.com>
2022-03-27 15:47:24 +08:00
Konstantine Karantasis 6ce69021fd
KAFKA-13759: Disable idempotence by default in producers instantiated by Connect (#11933)
With AK 3.0, idempotence was enabled by default in Kafka producers. However, if idempotence is enabled, Connect won't be able to communicate via its producers with Kafka brokers older than version 0.11. Perhaps more importantly, for brokers older than version 2.8 the IDEMPOTENT_WRITE ACL is required to be granted to the principal of the Connect worker.

Therefore this commit disables producer idempotence by default to all the producers instantiated by Connect. Users can still choose to enable producer idempotence by explicitly setting the right worker and/or connector properties.

The changes were tested via existing unit, integration and system tests.

Reviewers: Randall Hauch <rhauch@gmail.com>
2022-03-23 15:03:52 -07:00
Bruno Cadonna 4c8685e701
MINOR: Bump trunk to 3.3.0-SNAPSHOT (#11925)
Version bumps on trunk following the creation of the 3.2 release branch.

Reviewer: David Jacot <djacot@confluent.io>
2022-03-21 21:37:05 +01:00
aSemy 38e3787d76
Minor typo: "result _is_ a" > "result _in_ a" (#11876)
Reviewers Bill Bejeck <bbejeck@apache.org>
2022-03-10 14:03:12 -05:00
Luke Chen 7c280c1d5f
KAFKA-13673: disable idempotence when config conflicts (#11788)
Disable idempotence when conflicting config values for acks, retries
and max.in.flight.requests.per.connection are set by the user. For the
former two configs, we log at info level when we disable idempotence
due to conflicting configs. For the latter, we log at warn level since
it's due to an implementation detail that is likely to be surprising.

This mitigates compatibility impact of enabling idempotence by default.

Added unit tests to verify the change in behavior.

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Mickael Maison <mickael.maison@gmail.com>
2022-03-03 05:40:41 -08:00
Mickael Maison 576496a1ca
MINOR: Improve Connect docs (#11642)
- Fix indendation of code blocks
- Add links to all SMTs and Predicates

Reviewers: Luke Chen <showuon@gmail.com>, Tom Bentley <tbentley@redhat.com>
2022-02-22 20:49:53 +08:00
Mickael Maison 2dc4d5d95e
MINOR: Add links to connector configs in TOC (#11794)
Reviewers: Luke Chen <showuon@gmail.com>, Andrew Eugene Choi <andrew.choi@uwaterloo.ca>
2022-02-22 10:54:53 +08:00
Ismael Juma 71f732e6e5
MINOR: Clarify producer idempotence default in upgrade docs (#11757)
* Mention `acks=1` to `acks=all` change in 3.0.0 upgrade docs
* Have a separate section for 3.0.1 and 3.1.1 as some may skip the
  3.0.0/3.1.0 section when upgrading to a bug fix.
* Move the 3.0.0 note to the top since it's more impactful than the
  other changes.

Reviewers: Jason Gustafson <jason@confluent.io>
2022-02-14 15:17:44 -08:00
Luke Chen e6db0ca48c
KAFKA-13598: enable idempotence producer by default and validate the configs (#11691)
In v3.0, we changed the default value for `enable.idempotence` to true, but we didn't adjust the validator and the `idempotence` enabled check method. So if a user didn't explicitly enable idempotence, this feature won't be turned on. This patch addresses the problem, cleans up associated logic, and fixes tests that broke as a result of properly applying the new default. Specifically it does the following:

1. fix the `ProducerConfig#idempotenceEnabled` method, to make it correctly detect if `idempotence` is enabled or not
2. remove some unnecessary config overridden and checks due to we already default `acks`, `retries` and `enable.idempotence` configs.
3. move the config validator for the idempotent producer from `KafkaProducer` into `ProducerConfig`. The config validation should be the responsibility of `ProducerConfig` class.
4. add an `AbstractConfig#hasKeyInOriginals` method, to avoid `originals` configs get copied and only want to check the existence of the key. 
5. fix many broken tests. As mentioned, we didn't actually enable idempotence in v3.0. After this PR, there are some tests broken due to some different behavior between idempotent and non-idempotent producer.
6. add additional tests to validate configuration behavior

Reviewers: Kirk True <kirk@mustardgrain.com>, Ismael Juma <ismael@juma.me.uk>, Mickael Maison <mimaison@users.noreply.github.com>, Jason Gustafson <jason@confluent.io>
2022-02-05 10:53:27 -08:00
Matthew Wong 17dcb8097c
MINOR: Update documentation and DumpLogSegments tool for addition of `deleteHorizonMs` in batch format (#11694)
This PR updates the documentation and tooling to match https://github.com/apache/kafka/pull/10914, which added support for encoding `deleteHorizonMs` in the record batch schema. The changes include adding the new attribute and updating field names. We have also updated stale references to the old `FirstTimestamp` field in the code and comments. Finally, In the `DumpLogSegments` tool, when record batch information is printed, it will also include the value of `deleteHorizonMs` is (e.g. `OptionalLong.empty` or `OptionalLong[123456]`).

Reviewers: Vincent Jiang <84371940+vincent81jiang@users.noreply.github.com>, Kvicii <42023367+Kvicii@users.noreply.github.com>, Jason Gustafson <jason@confluent.io>
2022-02-04 16:20:37 -08:00
Luke Chen e82df83968
KAFKA-13374: Update doc to mention read from leader/followers (#11408)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2021-12-20 18:34:20 +01:00
David Jacot fd9fdd8dac
MINOR: Update doc for 3.1 (#11539)
* Update main version of main documentation and add link to previous version;
* Update quick start guide (links do not work yet obviously);
* Add upgrade section.

Reviewers: Luke Chen <showuon@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
2021-12-03 09:55:33 +01:00
Luke Chen 908461a07e
MINOR: Update rocksDb memory management doc (#11528)
In the RocksDb memory management doc, we mentioned in the footnote that there's a rocksdb bug caused the strict_capacity_limit boolean parameter in the LRUCache constructor can't be set to true. However, the bug is already fixed in 6.11.4, and we're using 6.22 now, so the note can be removed.

Reviewer: Bruno Cadonna <cadonna@apache.org>
2021-11-25 10:20:32 +01:00
Kate Stanley c0b2afb353
KAFKA-13455: Add steps to run Kafka Connect to quickstart (#11500)
Signed-off-by: Katherine Stanley <11195226+katheris@users.noreply.github.com>
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2021-11-22 14:41:24 +01:00
Luke Chen 1b4cffdcb7
KAFKA-13439: Deprecate eager rebalance protocol in kafka stream (#11490)
Deprecate eager rebalancing protocol in kafka streams and log warning message when upgrade.from is set to 2.3 or lower. Also add a note in upgrade doc to prepare users for the removal of eager rebalancing support

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
2021-11-17 03:05:19 -08:00
Matthias J. Sax 30d1989db1
MINOR: update Kafka Streams standby task config (#11404)
Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Antony Stubbs <antony@confluent.io>, James Galasyn <jim.galasyn@confluent.io>
2021-11-16 17:34:49 -08:00
yasar03 1075e1189a
MINOR: Fix typo in upgrade docs (#11466)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Luke Chen <showuon@gmail.com>
2021-11-15 17:00:49 +01:00
GuoPhilipse 3f8ca44bc6
KAFKA-13441: Fix typo in upgrade.html (#11483)
Reviewers: Mickael Maison <mickael.maison@gmail.com>

Co-authored-by: gf13871 <gf13871@ly.com>
2021-11-15 14:16:41 +01:00
Lee Dongjin 9d305c79d3
KAFKA-13430: Remove broker-wide quota properties from the documentation (#11463)
Reviewers: David Jacot <djacot@confluent.io>
2021-11-04 10:04:02 +01:00
Luizfrf3 252a40ea1f
KAFKA-8941: Add RocksDB Metrics that Could not be Added due to RocksD… (#11441)
This PR adds some RocksDB metrics that could not be added in KIP-471 due to RocksDB version. The new metrics are extracted using Histogram data provided by RocksDB API, and the old ones were extracted using Tickers. The new metrics added are memtable-flush-time-(avg|min|max) and compaction-time-(avg|min|max).

Reviewer: Bruno Cadonnna <cadonna@apache.org>
2021-11-03 12:18:28 +01:00
David Jacot 3aef0a5ceb
MINOR: Bump trunk to 3.2.0-SNAPSHOT (#11458)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
2021-11-02 13:38:54 +01:00
Jongho Jeon 48475e89ca
MINOR: Fix broken link in introduction.html (#11249)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2021-10-11 16:45:36 +02:00
Andy Chambers 1573c707e6
KAFKA-13298: Improve documentation on EOS KStream requirements (#11355)
Reviewer: Matthias J. Sax <matthias@confluent.io>
2021-10-05 19:07:13 -07:00
Matthias J. Sax 0888953e0e
MINOR: remove note on future multi-cluster support of Kafka Streams (#11363)
Reviewer: Bill Bejeck <bill@confluent.io>
2021-09-28 23:09:24 -07:00
David Jacot ad6b7d421c
MINOR: Add missing upgrade doc for 2.8 (#11343)
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
2021-09-20 10:27:36 +02:00
Weisheng Yang db16c516a7
KAFKA-13301 Config documentation optimized for 'request.timeout. ms' and 'max.poll.interval.ms'. (#11329)
Reviewers: Guozhang Wang <wangguoz@gmail.com>
2021-09-18 22:19:02 -07:00
Josep Prat 286126f9a5
KAFKA-13243: KIP-773 Differentiate metric latency measured in ms and ns (#11302)
KAFKA-13243: KIP-773 Differentiate metric latency measured in ms and ns

Implementation of KIP-773

Deprecates inconsistent metrics bufferpool-wait-time-total,
io-waittime-total, and iotime-total.
Introduces new metrics bufferpool-wait-time-ns-total,
io-wait-time-ns-total, and io-time-ns-total with the same semantics as
before.
Adds metrics (old and new) in ops.html.
Adds upgrade guide for these metrics.

Reviewers: Guozhang Wang <wangguoz@gmail.com>, Tom Bentley <tbentley@redhat.com>
2021-09-08 18:00:58 +01:00
Ismael Juma 0118330103
KAFKA-13273: Add support for Java 17 (#11296)
Java 17 is at release candidate stage and it will be a LTS release once
it's out (previous LTS release was Java 11).

Details:
* Replace Java 16 with Java 17 in Jenkins and Readme.
* Replace `--illegal-access=permit` (which was removed from Java 17)
   with  `--add-opens` for the packages we require internal access to.
   Filed KAFKA-13275 for updating the tests not to require `--add-opens`
   (where possible).
* Update `release.py` to use JDK8. and JDK 17 (instead of JDK 8 and JDK 15).
* Removed all but one Streams test from `testsToExclude`. The
   Connect test exclusion list remains the same.
* Add notable change to upgrade.html
* Upgrade to Gradle 7.2 as it's required for proper Java 17 support.
* Upgrade mockito to 3.12.4 for better Java 17 support.
* Adjusted `KafkaRaftClientTest` and `QuorumStateTest` not to require
   private access to `jdk.internal.util.random`.

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2021-09-06 08:55:52 -07:00
Guozhang Wang fb2b131310
KAFKA-13172: Add downgrade guidance note for 3.0 (#11184)
Reviewers: Luke Chen <showuon@gmail.com>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Bruno Cadonna <cadonna@confluent.io>
2021-08-17 17:03:11 -07:00
Luke Chen 2bfd0ae2e9
MINOR: update the branch(split) doc and java doc and tests (#11195)
Reviewers: Ivan Ponomarev <iponomarev@mail.ru>, Matthias J. Sax <matthias@confluent.io>
2021-08-10 13:37:59 -07:00
Matthias J. Sax f6351a9542
MINOR: update stream-stream join docs (#11172)
Reviewers: Guozhang Wang <guozhang@confluent.io>
2021-08-05 11:33:20 -07:00
Ron Dagostino 2edaf9ddd3
MINOR: Update KRaft README.md and upgrade.html for 3.0
Reviewers: Colin P. McCabe <cmccabe@apache.org>
2021-08-02 10:39:32 -07:00
Jim Galasyn d88989d595
KQE-289: Replace SerDe with Serde (#11050)
Right now, we have scattered uses of `SerDes` throughout the docs. These should be updated to be `Serdes`, as that's what we commonly use now.

Reviewers: Boyang Chen <bchen11@outlook.com>
2021-08-01 17:00:57 -07:00
A. Sophie Blee-Goldman 7fbc6b73aa
KAFKA-13021: clarify KIP-633 javadocs and address remaining feedback (#11114)
There were a few followup things to address from #10926, most importantly a number of updates to the javadocs. Also includes a few missing verification checks.

Reviewers: Guozhang Wang <guozhang@confluent.io>, Matthias J. Sax <mjsax@apache.org>, Israel Ekpo
2021-07-23 16:14:37 -07:00
Luke Chen ad59e3b622
MINOR: update doc to reflect the grace period change (#11100)
We removed default 24 hours grace period in KIP-633, and deprecate some grace methods, but we forgot to update the stream docs.

Reviewer: Bruno Cadonna <cadonna@apache.org>
2021-07-21 13:26:41 +02:00
Luke Chen b6159042c0
KAFKA-12598: ConfigCommand should only support communication via ZooKeeper for a reduced set of cases (#10811)
Checked the documentation, we must use `--zookeeper` option in 3 places (alter and describe):
1. user configs where the config is a SCRAM mechanism name (i.e. a SCRAM credential for a user)
2. update broker configs for a particular broker when that broker is down
3. broker default configs when all brokers are down

Reference:
1. [config SCRAM Credentials](https://kafka.apache.org/documentation/#security_sasl_scram_credentials)
2. [Update config before broker started](https://kafka.apache.org/documentation/#dynamicbrokerconfigs)

So, after this PR, we only support `--zookeeper` on `users` and `brokers` entity. Add some argument parse rules and tests. 

Reviewers: Ron Dagostino <rdagostino@confluent.io>, Ismael Juma <ismael@juma.me.uk>
2021-07-19 18:53:14 -07:00
Ismael Juma a46b82bea9
KAFKA-12944: Assume message format version is 3.0 when inter-broker protocol is 3.0 or higher (KIP-724) (#11036)
Also:
* Deprecate `log.message.format.version` and `message.format.version`.
* Log broker warning if the deprecated config values are ignored due to
the inter-broker protocol version.
* Log warning if `message.format.version` is set via `ConfigCommand`.
* Always down-convert if fetch version is v3 or lower.
* Add tests to verify new message format version based on the
inter-broker protocol version.
* Adjust existing tests that create topics with an older message format to
have the inter-broker protocol set to 2.8.
* Add upgrade note.

Note that the log compaction change to always write new segments with
record format v2 if the IBP is 3.0 or higher will be done as part of
KAFKA-13093 (with Kafka 3.1 as the target release version).

Reviewers: David Jacot <djacot@confluent.io>, David Arthur <mumrah@gmail.com>, Jason Gustafson <jason@confluent.io>
2021-07-19 05:37:16 -07:00
Guozhang Wang 3e3264760b
KAFKA-10847: Remove internal config for enabling the fix (#10941)
Also update the upgrade guide indicating about the grace period KIP and its indication on the fix with throughput impact.

Reviewers: Luke Chen <showuon@gmail.com>, Matthias J. Sax <mjsax@apache.org>
2021-07-15 10:58:15 -07:00
Luke Chen ac18bdc183
MINOR: update doc for default assignor change (#11009)
Update the doc and upgrade doc for default assignor change. REF: #10903

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
2021-07-13 13:00:15 -07:00
Ryan Dielhenn f97f36b650
KAFKA-13051; Require principal builders implement `KafkaPrincipalSerde` and set default (#11011)
This patch adds a check to ensure that principal builder implementations implement `KafkaPrincipalSerde` as specified in KIP-590: https://cwiki.apache.org/confluence/display/KAFKA/KIP-590%3A+Redirect+Zookeeper+Mutation+Protocols+to+The+Controller. This patch also changes the default value of `principal.builder.class` to `DefaultKafkaPrincipalBuilder`, which was already the implicit behavior when no principal builder was specified.

Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
2021-07-13 10:54:36 -07:00
John Roesler a08e0cfe65
KAFKA-8410: Update the docs to reference the new PAPI (#10994)
Reviewers: Jim Galasyn <jim.galasyn@confluent.io>, Luke Chen <showuon@gmail.com>, Matthias J. Sax <mjsax@apache.org>
2021-07-13 10:23:50 -05:00
John Roesler bfdef11b97
KAFKA-12360: Document new time semantics (#11003)
Update the docs for task idling, since the semantics have
changed in 3.0.

Reviewers: Jim Galasyn <jim.galasyn@confluent.io>, Luke Chen <showuon@gmail.com>, Boyang Chen <boyang@apache.org>
2021-07-12 16:16:29 -05:00
A. Sophie Blee-Goldman 53f9b04176
MINOR: fix typo in config for eos-v2 docs of the upgrade guide (#11001)
One of the configs has a typo and should have been exactly_once _v2 rather than just exactly_once

Reviewers: John Roesler <vvcephei@apache.org>
2021-07-08 12:32:45 -07:00
leah d6918e427e
KAFKA-9559 add docs for changing default serde to null (#10988)
#10813 changed the default serde from ByteArraySerde as discussed in KIP-741. This adds proper documentation so users know to set a serde through the configs or explicitly pass one in.

Reviewers: Walker Carlson <wcarlson@confluent.io>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>
2021-07-08 11:33:22 -07:00
Konstantine Karantasis d2a05d71c0
Bump trunk to 3.1.0-SNAPSHOT (#10981)
Typical version bumps on trunk following the creation of the 3.0 release branch.

Reviewer: Randall Hauch <rhauch@gmail.com>
2021-07-06 14:28:13 -07:00
kpatelatwork 2e89f40575
KAFKA-13035 updated documentation for connector restart REST API to … (#10975)
Updated documentation for connector restart REST API to include the tasks restart behavior, including calling out that by default the same behavior of previous versions is preserved.

Author: Kalpesh Patel <kpatel@confluent.io>
Reviewer: Randall Hauch <rhauch@gmail.com>
2021-07-06 13:31:40 -05:00
Ryanne Dolan 6d2f563865
KAFKA-12436: Deprecate MirrorMaker v1 (KIP-720) (#10805)
Reviewers: Luke Chen <showuon@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Mickael Maison <mickael.maison@gmail.com>
2021-07-04 15:17:31 +01:00
Chris Egerton cad2f5e120
KAFKA-12717: Remove internal Connect converter properties (KIP-738) (#10854)
Removed Connect Distributed worker's internal converter properties.

Author: Chris Egerton <chrise@confluent.io>
Reviewer: Randall Hauch <rhauch@gmail.com>
2021-07-01 21:02:24 -05:00
Ryanne Dolan 93f57370c6
KAFKA-9726: Add IdentityReplicationPolicy to MirrorMaker2 (#10652)
This new policy enables active/passive, one-way replication without renaming topics, similar to MM1. This implementation is described in KIP-382 (adopted), originally as "LegacyReplicationPolicy".

This enables operators to migrate from MM1 to MM2 without re-architecting their replication flows, and enables some additional use-cases for MM2. For example, operators may wish to "upgrade" their Kafka clusters by mirroring everything to a completely new cluster. Such a migration would have been difficult with either MM1 or MM2 previously.

When using IdentityReplicationPolicy, operators should be aware that MM2 will not be able to detect cycles among replicated topics. A misconfigured topology may result in replicating the same records back-and-forth or in an infinite loop. However, we don't prevent this behavior, as some use-cases involve filtering records (via SMTs) to prevent cycles.

Reviewers: Mickael Maison <mickael.maison@gmail.com>

Co-authored-by: Ryanne Dolan <rdolan@twitter.com>
Co-authored-by: Matthew de Detrich <mdedetrich@gmail.com>
Co-authored-by: Ivan Yurchenko <ivanyu@aiven.io>
2021-07-01 09:21:27 +01:00
Ron Dagostino 4f5b4c868e
KAFKA-12756: Update ZooKeeper to v3.6.3 (#10918)
Update the ZooKeeper version to v3.6.3. This requires adding dropwizard
as a new dependency.

Also, add Kafka v2.8.0 to the ducktape system test image.

Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>
2021-06-30 11:21:33 -07:00
Josep Prat 6655a09e99
KAFKA-12849: KIP-744 TaskMetadata ThreadMetadata StreamsMetadata as API (#10840)
Implementation of KIP-744.

Creates new Interfaces for TaskMetadata, ThreadMetadata, and
StreamsMetadata, providing internal implementations for each of them.

Deprecates current TaskMetadata, ThreadMetadata under o.a.k.s.processor,
and SreamsMetadata under a.o.k.s.state.

Updates references on internal classes from deprecated classes to new interfaces.

Deprecates methods on KafkaStreams returning deprecated ThreadMeatada and
StreamsMetadata, and provides new ones returning the new interfaces.

Update Javadocs referencing to deprecated classes and methods to point
to the right ones.

Co-authored-by: Bruno Cadonna <cadonna@apache.org>

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Bruno Cadonna <cadonna@apache.org>
2021-06-25 18:31:49 +02:00
kpatelatwork 5652ef1af0
KAFKA-12482 Remove deprecated rest.host.name and rest.port configs (#10841)
Remove the `rest.host.name` and `rest.port` Connect worker configs that were deprecated in KIP-208 and AK 1.1.

Author: Kalpesh Patel <kalpeshpatel.india@gmail.com>
Reviewers: Randall Hauch <rhauch@gmail.com>, Konstantine Karantasis <konstantine@confluent.io>, wenbingshen <oliver.shen999@gmail.com>
2021-06-23 09:20:04 -05:00
Josep Prat f5a94d913f
MINOR: Fix indentation for several doc pages (#10766)
Fixes the indentation of the code listings for:

api.html
configuration.html
design.html
implementation.html
toc.html
These changes consist of whitespaces added or removed for consistency. It also contains a couple of fixes on unbalanced html tags.

Reviewers: Luke Chen <showuon@gmail.com>, Bill Bejeck <bbejeck@apache.org>
2021-06-17 15:58:40 -04:00
Ismael Juma d27a84f70c
KAFKA-12945: Remove port, host.name and related configs in 3.0 (#10872)
They have been deprecated since 0.10.0. Full list of removes configs:
* port
* host.name
* advertised.port
* advertised.host.name

Also adjust tests to take the removals into account. Some tests were
no longer relevant and have been removed.

Finally, took the chance to:
* Clean up unnecessary usage of `KafkaConfig$.MODULE$` in
related files.
* Add missing `Test` annotations to `AdvertiseBrokerTest` and
make necessary changes for the tests to pass.

Reviewers: David Jacot <djacot@confluent.io>, Luke Chen <showuon@gmail.com>
2021-06-17 05:32:34 -07:00
Jason Gustafson a75b5c635b
KAFKA-12874; Increase default consumer session timeout to 45s (#10803)
This patch increases the default consumer session timeout to 45s as documented in KIP-735: https://cwiki.apache.org/confluence/display/KAFKA/KIP-735%3A+Increase+default+consumer+session+timeout.

Reviewers: Luke Chen <showuon@gmail.com>, David Arthur <mumrah@gmail.com>, David Jacot <djacot@confluent.io>
2021-06-09 15:09:31 -07:00
Luke Chen ccde334ca9
KAFKA-12597: Remove deprecated --zookeeper option in ReassignPartitionsCommand (#10471)
Also remove zookeeper dependent methods and tests.

Reviewers: Ismael Juma <ismael@juma.me.uk>
2021-06-03 23:31:29 -07:00
Ismael Juma 364bd36771
KAFKA-12880: Remove deprecated `Count` and `SampledTotal` in 3.0 (#10808)
They were both deprecated in Apache Kafka 2.4 and it's a straightforward change
to use the non deprecated variants.

Reviewers: David Jacot <djacot@confluent.io>
2021-06-02 11:31:05 -07:00
wenbingshen 4c1efd3d08
MINOR: Update kafka-topics.sh line command tool upgrade notes with removed option (#10806)
Reviewers: Luke Chen <showuon@gmail.com>, Ismael Juma <ismael@juma.me.uk>

Co-authored-by: shenwenbing <shenwenbing@qianxin.com>
2021-06-02 05:57:04 -07:00
Josep Prat 054d5f9e91
MINOR: fix code listings security.html (#10770)
Fix examples under security.html so they use the right bash icon (`>`
instead of `$`) and also uses the right tool for showing code listings.

Reviewers: Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>
2021-06-01 13:20:35 +02:00
prince-mahajan db288e4a64
Update implementation.html (#10771)
Fixing the link to a cited blog. The existing link now points to a steroid website so need to pull the blog from internet archives.
2021-05-26 17:30:34 -07:00
A. Sophie Blee-Goldman bbe170af70
MINOR: deprecate TaskMetadata constructor and add KIP-740 notes to upgrade guide (#10755)
Quick followup to KIP-740 to actually deprecate this constructor, and update the upgrade guide with what we changed in KIP-740. I also noticed the TaskId#parse method had been modified previously, and should be re-added to the public TaskId class. It had no tests, so now it does

Reviewers: Matthias J. Sax <mjsax@confluent.io>, Luke Chen <showuon@gmail.com>
2021-05-26 10:35:12 -07:00
Josep Prat aa25176e77
MINOR: Kafka Streams code samples formating unification (#10651)
Code samples are now unified and correctly formatted.
Samples under Streams use consistently the prism library.

Reviewers: Bruno Cadonna <cadonna@apache.org>
2021-05-21 17:27:42 +02:00
Daniyar Yeralin 6d1ae8bc00
KAFKA-8326: Introduce List Serde (#6592)
Introduce List serde for primitive types or custom serdes with a serializer and a deserializer according to KIP-466

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Matthias J. Sax <mjsax@conflunet.io>, John Roesler <roesler@confluent.io>, Michael Noll <michael@confluent.io>
2021-05-13 15:54:00 -07:00
Vito Jeng fae0784ce3
KAFKA-5876: KIP-216 Part 4, Apply InvalidStateStorePartitionException for Interactive Queries (#10657)
KIP-216, part 4 - apply InvalidStateStorePartitionException

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
2021-05-10 17:29:58 -07:00
Matthias J. Sax 6a5992a814
KAFKA-8531: Change default replication factor config (#10532)
Implements KIP-733

Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>
2021-05-05 16:16:06 -07:00
Vito Jeng 816f5c3b86
KAFKA-5876: KIP-216 Part 3, Apply StreamsNotStartedException for Interactive Queries (#10597)
KIP-216 Part 3: Throw StreamsNotStartedException if KafkaStreams state is CREATED

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
2021-05-03 13:53:35 -07:00
Guozhang Wang 3ec6317ee6
KAFKA-12683: Remove deprecated UsePreviousTimeOnInvalidTimestamp (#10557)
Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
2021-05-01 08:31:41 -07:00
Lee Dongjin 0a1ef2e982
MINOR: Group KafkaController, ReplicaManager metrics in documentation (#7891)
Some minor edits to the docs

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
2021-04-30 12:50:37 -07:00
A. Sophie Blee-Goldman 3805f3706f
KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2 (#10573)
Deprecates the following 

1. StreamsConfig.EXACTLY_ONCE
2. StreamsConfig.EXACTLY_ONCE_BETA
3. Producer#sendOffsetsToTransaction(Map offsets, String consumerGroupId)

And introduces a new StreamsConfig.EXACTLY_ONCE_V2 config. Additionally, this PR replaces usages of the term "eos-beta" throughout the code with the term "eos-v2"

Reviewers: Matthias J. Sax <mjsax@confluent.io>
2021-04-28 13:22:15 -07:00
JoelWee 0ea440b2af
KAFKA-6435: KIP-623 Add internal topics option to streamResetter (#8923)
Allow user to specify subset of internal topics to clean up with application reset tool

Reviewers: Boyang Chen <boyang@confluent.io>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Walker Carlson <wcarlson@confluent.io>
2021-04-27 15:44:53 -07:00
Vito Jeng c972ac929e
KAFKA-5876: Apply UnknownStateStoreException for Interactive Queries (#9821)
KIP-216: IQ should throw different exceptions for different errors, Part 2

Reviewers: Matthias J. Sax <mjsax@confluent.io>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Bruno Cadonna <cadonna@confluent.io>
2021-04-23 12:09:32 -07:00
Guozhang Wang 87b24025ce HOTFIX: typo in Streams DSL docs 2021-04-21 10:25:54 -07:00
Guozhang Wang 035449eb55
KAFKA-12633: Remove deprecated APIs in TopologyTestDriver (#10508)
As well as related test classes.

Reviewers: John Roesler <vvcephei@apache.org>
2021-04-18 10:46:01 -07:00
Tom Bentley f8f1769256
KIP-145: Add SMTs, HeaderFrom, DropHeaders and InsertHeader (#9549)
These SMTs were originally specified in KIP-145 but never implemented
at the time.

HeaderTo is not included since its original specification doesn't deal with
the fact that there can be >1 header with the same name, but a field can only
have a single value (which could be an array, but not if the headers for
the given name had different schemas).

Reviewers: Chris Egerton <chrise@confluent.io>, Mickael Maison <mickael.maison@gmail.com>
2021-04-16 15:11:25 +01:00
Ismael Juma 89933f21f2
KAFKA-12612: Remove `checksum` from ConsumerRecord/RecordMetadata for 3.0 (#10470)
The methods have been deprecated since 0.11 without replacement since
message format 2 moved the checksum to the record batch (instead of the
record).

Unfortunately, we did not deprecate the constructors that take a checksum
(even though we intended to) so we cannot remove them. I have deprecated
them for removal in 4.0 and added a single non deprecated constructor to
`ConsumerRecord` and `RecordMetadata` that take all remaining parameters.
`ConsumerRecord` could do with one additional convenience constructor, but
that requires a KIP and hence should be done separately.

Also:
* Removed `ChecksumMessageFormatter`, which is technically not public
API, but may have been used with the console consumer.
* Updated all usages of `ConsumerRecord`/`RecordMetadata` constructors
to use the non deprecated ones.
* Added tests for deprecated `ConsumerRecord/`RecordMetadata`
constructors.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, David Jacot <djacot@confluent.io>
2021-04-14 14:38:37 -07:00
David Jacot 23633cd1ec
KAFKA-12584; Remove deprecated `Sum` and `Total` classes (#10511)
`Sum` and `Total` classes were deprecated and replaced by `WindowedSum` and `CumulativeSum` in 2.4. This patch removes them for 3.0.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Ismael Juma <ismael@juma.me.uk>
2021-04-13 11:48:17 +02:00
dengziming 88eb24db40
KAFKA-12637: Remove deprecated PartitionAssignor interface (#10512)
Remove PartitionAssignor and related classes, update docs and move unit test

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
2021-04-12 18:37:01 -07:00
A. Sophie Blee-Goldman c608d8480e
KAFKA-7606: Remove deprecated options from StreamsResetter (#10411)
Remove deprecated --zookeeper and --execute flags

Reviewers: Matthias J. Sax <mjsax@confluent.io>
2021-04-12 18:24:18 -07:00
Lee Dongjin 0b0eaefc60
KAFKA-12408: Document omitted ReplicaManager metrics (#10258)
Reviewers: Tom Bentley <tbentley@redhat.com>
2021-04-12 15:35:15 +01:00
Andrew Lee c9edb8ad84
MINOR: Fix typo in docs (#10423)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2021-04-09 12:12:52 -07:00
Jorge Esteban Quilcate Otoya db0323e9ba
KAFKA-12449: Remove deprecated WindowStore#put (#10293)
Removes `WindowStore#put(K,V)` that was deprecated via KIP-474.

Reviewers: Matthias J. Sax <matthias@confluent.io>
2021-04-09 11:49:37 -07:00
Jim Galasyn e4b6ffae5d
MINOR: Fix left/outer join descriptions in Streams DSL reference topic (#10453)
Reviewer: Matthias J. Sax <matthias@confluent.io>
2021-04-09 11:41:52 -07:00
David Jacot ff3b2d67a1
KAFKA-12591; Remove deprecated `quota.producer.default` and `quota.consumer.default` configurations (#10427)
`quota.producer.default` and `quota.consumer.default` were deprecated in AK 0.11.0.0. Dynamic default quotas must be used instead. This patch removes them for AK 3.0. 

Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>, Ismael Juma <ismael@juma.me.uk>
2021-04-09 18:11:34 +02:00
cyc 8d356b034b
KAFKA-12492: Fix the formatting of example RocksDBConfigSetter (#10486)
Fix the formatting of example RocksDBConfigSetter due to the un-arranged spaces within <pre> tag.

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>
2021-04-08 20:55:52 -07:00
Guozhang Wang d44e01be48 HOTFIX: add missing upgrade docs 2021-04-08 08:49:33 -07:00
Guozhang Wang 3ca5a3bb78
KAFKA-12568: Remove deprecated APIs in KStream, KTable and Joined (#10421)
This is related to KIP-307 / KIP-372 / KIP-479.

Reviewers: John Roesler <vvcephei@apache.org>
2021-04-07 17:38:43 -07:00
Marco Aurelio Lotz 33d0445b84
KAFKA-5146: remove Connect dependency from Streams module (#10131)
Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Ismael Juma <ismael@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2021-04-06 17:47:52 -07:00
Ismael Juma 2f36001987
KAFKA-12579: Remove various deprecated clients classes/methods for 3.0 (#10438)
* Remove `ExtendedSerializer` and `ExtendedDeserializer`, deprecated since 2.1.
The extra functionality was also made available in `Serializer` and `Deserializer`.
* Remove `close(long, TimeUnit)` from the producer, consumer and admin client,
deprecated since 2.0 for the consumer and 2.2 for the rest. The replacement is `close(Duration)`.
* Remove `ConsumerConfig.addDeserializerToConfig` and `ProducerConfig.addSerializerToConfig`,
deprecated since 2.7 with no replacement. These methods were not intended to be public API
and are likely not used much (if at all).
* Remove `NoOffsetForPartitionException.partition()`, deprecated since 0.11. `partitions()`
should be used instead.
* Remove `MessageFormatter.init(Properties)`, deprecated since 2.7. The `configure(Map)`
method should be used instead.
* Remove `kafka.common.MessageFormatter`, deprecated since 2.7.
`org.apache.kafka.common.MessageFormatter` should be used instead.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, David Jacot <djacot@confluent.io>
2021-04-06 08:54:53 -07:00