Part of KIP-572: follow up work to PR #9800. It's not save to retry a TX commit after a timeout, because it's unclear if the commit was successful or not, and thus on retry we might get an IllegalStateException. Instead, we will throw a TaskCorruptedException to retry the TX if the commit failed.
Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>
We recommend users to switch to jemalloc for RocksDB.
Co-authored-by: Jim Galasyn <jim.galasyn@confluent.io>
Reviewers: Jim Galasyn <jim.galasyn@confluent.io>, Rohan Desai <rohan@confluent.io>, A. Sophie Blee-Goldman <sophie@confluent.io>
Implements KIP-418, that deprecated the `branch()` operator in favor of the newly added and type-safe `split()` operator.
Reviewers: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>
Add docs for KIP-663.
Reviewers: Jim Galasyn <jim.galasyn@confluent.io>, Walker Carlson <wcarlson@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Co-authored-by: Bruno Cadonna <bruno@confluent.io>
Reviewers: Jim Galasyn <jim.galasyn@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Bruno Cadonna <bruno@confluent.io>
This adds a new user-facing documentation "Geo-replication (Cross-Cluster Data Mirroring)" section to the Kafka Operations documentation that covers MirrorMaker v2.
Was already merged to kafka-site via apache/kafka-site#324.
Reviewers: Bill Bejeck <bbejeck@apache.org>
Document the new properties-based metrics for RocksDB
Reviewers: Leah Thomas <lthomas@confluent.io>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>
During the AK website upgrade, changes made to kafka-site weren't migrated back to kafka-docs.
This PR is an attempt at porting the streams changes to kafka/docs
For the most part, the bulk of the changes in the PR are cosmetic.
For testing:
I reviewed the PR diffs
Rendered the changes locally
Reviewers: John Roesler <john@confluent.io>
During the AK website upgrade, changes made to kafka-site weren't migrated back to kafka-docs.
This PR is an initial attempt at porting the changes to kafka/docs, but it does not include the streams changes. Those will come in a separate PR.
For the most part, the bulk of the changes in the PR are cosmetic. Only the introduction.html has substantial changes, but it's a direct port from the live documentation.
For testing:
I reviewed the PR diffs
Rendered the changes locally
Reviewers: Matthias J. Sax <mjsax@apache.org>
Adds support for SSL key and trust stores to be specified in PEM format either as files or directly as configuration values.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
The previous dynamic configuration validation did not actually compare new configs to original configs as intended, so the expected exception was not thrown.
Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>, Jason Gustafson <jason@confluent.io>
Currently, the source reference are all pointing to the 1.0 version codes,
which is obviously wrong. Update to the current dotVersion.
Reviewers: John Roesler <vvcephei@apache.org>
Add necessary documentation for KIP-450, adding sliding window aggregations to KStreams
Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Wildcard import of the old org.apache.kafka.streams.scala.Serdes leads
to a name clash because some of implicits has the same names as types
from the scala's std lib. The new oak.streams.scala.serialization.Serdes is
the same as the old Serdes, but without name clashes.
The old one is marked as deprecated.
Also, add missing serdes for UUID, ByteBuffer and Short types in
the new Serdes.
Implements: KIP-616
Reviewers: John Roesler <vvcephei@apache.org>
- part of KIP-572
- removed the usage of `retries` in `GlobalStateManger`
- instead of retries the new `task.timeout.ms` config is used
Reviewers: John Roesler <john@confluent.io>, Boyang Chen <boyang@confluent.io>, Guozhang Wang <guozhang@confluent.io>
Java 11 has been recommended for a while, but the ops section had not been updated.
Also added `-XX:+ExplicitGCInvokesConcurrent` which has been in `kafka-run-class`
for a while. Finally, tweaked the text slightly to read better.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
- part of KIP-572
- deprecates producer config `retries` (still in use)
- deprecates admin config `retries` (still in use)
- deprecates Kafka Streams config `retries` (will be ignored)
- adds new Kafka Streams config `task.timeout.ms` (follow up PRs will leverage this new config)
Reviewers: John Roesler <john@confluent.io>, Jason Gustafson <jason@confluent.io>, Randall Hauch <randall@confluent.io>
Add docs for KIP-441 and KIP-613.
Fixed some miscellaneous unrelated issues in the docs:
* Adds some missing configs to the Streams config docs: max.task.idle.ms,topology.optimization, default.windowed.key.serde.inner.class, and default.windowed.value.serde.inner.class
* Defines the previously-undefined default windowed serde class configs, including choosing a default (null) and giving them a doc string, so the yshould nwo show up in the auto-generated general Kafka config docs
* Adds a note to warn users about the rocksDB bug that prevents setting a strict capacity limit and counting write buffer memory against the block cache
Reviewers: Bruno Cadonna <bruno@confluent.io>, John Roesler <vvcephei@apache.org>
Brokers currently return NOT_LEADER_FOR_PARTITION to producers and REPLICA_NOT_AVAILABLE to consumers if a replica is not available on the broker during reassignments. Non-Java clients treat REPLICA_NOT_AVAILABLE as a non-retriable exception, Java consumers handle this error by explicitly matching the error code even though it is not an InvalidMetadataException. This PR renames NOT_LEADER_FOR_PARTITION to NOT_LEADER_OR_FOLLOWER and uses the same error for producers and consumers. This is compatible with both Java and non-Java clients since all clients handle this error code (6) as retriable exception. The PR also makes ReplicaNotAvailableException a subclass of InvalidMetadataException.
- ALTER_REPLICA_LOG_DIRS continues to return REPLICA_NOT_AVAILABLE. Retained this for compatibility since this request never returned NOT_LEADER_FOR_PARTITION earlier.
- MetadataRequest version 0 also returns REPLICA_NOT_AVAILABLE as topic-level error code for compatibility. Newer versions filter these out and return Errors.NONE, so didn't change this.
- Partition responses in MetadataRequest return REPLICA_NOT_AVAILABLE to indicate that one of the replicas is not available. Did not change this since NOT_LEADER_FOR_PARTITION is not suitable in this case.
Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Bob Barrett <bob.barrett@confluent.io>
Added a section about error reporting in Connect documentation, and another about how to safely use the new errant record reporter in SinkTask implementations.
Author: Aakash Shah <ashah@confluent.io>
Reviewer: Randall Hauch <rhauch@gmail.com>
* Add documentation for using transformation predicates.
* Add `PredicateDoc` for generating predicate config docs, following the style of `TransformationDoc`.
* Fix the header depth mismatch.
* Avoid generating HTML ids based purely on the config name since there
are very likely to conflict (e.g. #name). Instead allow passing a function
which can be used to generate an id from a config key.
The docs have been generated and tested locally.
Reviewer: Konstantine Karantasis <konstantine@confluent.io>
This applies to the producer, consumer, admin client, connect worker
and inter broker communication.
`ClientDnsLookup.DEFAULT` has been deprecated and a warning
will be logged if it's explicitly set in a client config.
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Ismael Juma <ismael@juma.me.uk>
1. Enables `TLSv1.3` by default with Java 11 or newer.
2. Add unit tests that cover the various TLSv1.2 and TLSv1.3 combinations.
3. Extend `benchmark_test.py` and `replication_test.py` to run with 'TLSv1.2'
or 'TLSv1.3'.
Reviewers: Ismael Juma <ismael@juma.me.uk>
Implements KIP-401:
- Add ConnectedStoreProvider interface
- let Processor/[*]Transformer[*]Suppliers extend ConnectedStoreProvider
- allows to add and connect state stores to processors/transformers implicitly
Reviewers: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>
Implemented KIP-437 by adding a new optional configuration property for the `MaskField` transformation that allows users to define a replacement literal for specific fields in matching records.
Author: Valeria Vasylieva <valeria.vasylieva@gmail.com>
Reviewer: Randall Hauch <rhauch@gmail.com>
fix broken links
rephrase a sentence
update the version number
Reviewers: Sophie Blee-Goldman <sophie@confluent.io>, Boyang Chen <boyang@confluent.io>, Bill Bejeck <bbejeck@apache.org>
In the docs, the `baseTimestamp` term is used when `firstTimestamp` is defined in the on-disk format and implementation.
Reviewers: Jason Gustafson <jason@confluent.io>
1. fix typo: `atleast` -> `at least`
2. add missing `--` from `--bootstrap-servers` argument to be consistent
3. rephrase a sentence, to make it more clear:
before: `LinkedIn is currently running JDK 1.8 u5 (looking to upgrade to a newer version) with the G1 collector`
It will misguide the users to use JDK 1.8 u5, while the JDK 1.8 u251 is already released, which will include many important bug fixes. I did some rephrase as below:
after: `At the time this is written, LinkedIn is running JDK 1.8 u5 (looking to upgrade to a newer version) with the G1 collector`
Reviewers: Konstantine Karantasis <konstantine@confluent.io>
The href attribute missed the starting double quote, so the hyperlink is interpreted as https://docs.oracle.com/.../agent.html", with a redundant tailing double quote.
Add the missing starting double quote back to fix this issue.
Reviewers: Konstantine Karantasis <konstantine@confluent.io>
Scala 2.13.2 introduces support for suppressing warnings,
which makes it possible to enable fatal warnings. This is
useful enough from a development perspective to justify
this change.
In addition, Scala 2.13.2 also has a Vector implementation
with significant performance improvements and encoding
of String matches to switches.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
* Minor: Some html fixes Streams DSL documentation:
- duplicate id
- duplicate opening <span> element
- surplus closing </div> tag
* Replaced opening/closing quotation mark codes with " (they caused w3c validation to complain).
* Replaced right arrow that wasn't rendered correctly with →
Reviewers: Mickael Maison <mickael.maison@gmail.com>
Jim Galasyn
Matthias J. Sax
Luke Chen
Justine Olshan
Cheng Tan
John Roesler
Luke Chen
Ivan Ponomarev
leah
Walker Carlson
Michael G. Noll
Bruno Cadonna
high.lee
James Cheng
fml2
Tom Bentley
zhangyue19921010
Bill Bejeck
Igor Soarez
Kengo Seki
Alexander Iskuskov
Okada Haruki
Emre Hasegeli
Matthias Merdes
Rajini Sivaram
Chia-Ping Tsai
Ankit Kumar
A. Sophie Blee-Goldman
Yuriy Badalyantc
John Thomas
Jason Gustafson
Ismael Juma
Gal Margalit
Aakash Shah
Badai Aqrandista
Nikolay
Randall Hauch
feyman2016
Paul
Navinder Pal Singh Brar
Valeria
Levani Kokhreidze
Jorge Esteban Quilcate Otoya
Viktor Somogyi
Sönke Liebau
Liam Clarke-Hutchinson
