Commit Graph

1165 Commits

Author SHA1 Message Date
Xuan-Zhang Gong 45f932819e
KAFKA-18864:remove the Evolving tag from stable public interfaces (#19036)
The purpose of this PR is to remove the `@InterfaceStability.Evolving` from classes that were created over a year ago.

Reviewers: Jun Rao <junrao@gmail.com>
2025-02-28 13:24:24 -08:00
snehashisp 9dc9973c1c
KAFKA-18863: Connect Multiversion Support (Versioned Connector Creation and related changes) (#17743)
Reviewers: Greg Harris <greg.harris@aiven.io>
2025-02-26 15:12:34 -08:00
Mickael Maison c13324fc16
KAFKA-15995: Adding KIP-877 support to Connect (#17804)
Reviewers: Greg Harris <gharris1727@gmail.com>
2025-02-11 18:27:15 +01:00
Peter Lee 0621c0b4de
KAFKA-18275 Restarting broker in testing should use the same port (#18381)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2025-02-07 03:02:36 +08:00
Piotr P. Karwasz 666571216b
KAFKA-18483 Disable `Log4jController` and `Loggers` if Log4j Core absent (#18496)
If Log4j Core is absent, most calls to Log4jController and Loggers will end up with a NoClassDefFoundError.

This changeset:

- Profits from the major version bump to rename k.util.Log4jController to LoggingController.
- Removes o.a.l.l.Level from the signature of public methods of o.a.k.connect.runtime.Loggers and replaces it with String.
- Provides an additional no-op implementation of k.util.LoggingController and o.a.k.connect.runtime.Loggers: if Log4j Core is not present on the runtime classpath the no-op implementation will be used.

Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2025-02-07 00:04:33 +08:00
Kuan-Po Tseng b99be961b8
KAFKA-18206: EmbeddedKafkaCluster must set features (#18189)
related to KAFKA-18206, set features in EmbeddedKafkaCluster in both streams and connect module, note that this PR also fix potential transaction with empty records in sendPrivileged method as transaction version 2 doesn't allow this kind of scenario.

Reviewers: Justine Olshan <jolshan@confluent.io>
2025-02-05 09:14:36 -08:00
Martin Sillence d001b47093
KAFKA-17792: Efficiently parse decimals with large exponents in Connect Values (#17510)
Reviewers: Greg Harris <greg.harris@aiven.io>, Mickael Maison <mickael.maison@gmail.com>
2025-01-27 09:04:29 -08:00
Ken Huang b4674c002f
KAFKA-18585 Fix fail test ValuesTest#shouldConvertDateValues (#18611)
Reviewers: Divij Vaidya <diviv@amazon.com>, Andrew Schofield <aschofield@confluent.io>
2025-01-21 13:51:39 +00:00
Andrew Schofield 42fd4d1d11
KAFKA-7776: Tests for ISO8601 in Connect value parsing (#18374)
Reviewers: Andrew Schofield <aschofield@confluent.io>
2025-01-16 14:26:03 +00:00
snehashisp 00b22b001f
KAFKA-18211: Override class loaders for class graph scanning in connect. (#18403)
Reviewers: Greg Harris <greg.harris@aiven.io>
2025-01-14 09:45:53 -08:00
Thomas Thornton b35c29401a
KAFKA-18073: Prevent dropped records from failed retriable exceptions (#18146)
Reviewers: Greg Harris <greg.harris@aiven.io>
2025-01-09 10:13:11 -08:00
PoAn Yang a97fb662fd
MINOR: add testRestoreCompactedDeletedConnector back to KafkaConfigBackingStoreTest (#18392)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2025-01-08 17:55:48 +08:00
snehashisp ad3369859c
KAFKA-18419: KIP-891 Connect Multiversion Support (Transformation and Predicate Changes) (#17742)
Reviewers: Greg Harris <greg.harris@aiven.io>
2025-01-06 12:18:45 -08:00
Ismael Juma 409a43eff7
MINOR: Collection/Option usage simplification via methods introduced in Java 9 & 11 (#18305)
Relevant methods:
1. `List.of`, `Set.of`, `Map.of` and similar (introduced in Java 9)
2. Optional: `isEmpty` (introduced in Java 11), `stream` (introduced in Java 9).

Reviewers: Mickael Maison <mimaison@users.noreply.github.com>
2025-01-03 16:13:39 -08:00
Ismael Juma d6f24d3665
Use `instanceof` pattern to avoid explicit cast (#18373)
This feature was introduced in Java 16.

Reviewers: David Arthur <mumrah@gmail.com>, Apoorv Mittal <apoorvmittal10@gmail.com>
2025-01-02 09:32:51 -08:00
YunKui Lu af7f403ae3
MINOR: Optimize log printing format for AbstractHerder#validateConverterConfig (#18199)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-12-30 23:29:34 +08:00
TengYao Chi 96527be90d
KAFKA-18243 Fix compatibility of Loggers class between log4j and log4j2 (#18185)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-12-30 01:50:52 +08:00
Kuan-Po Tseng e9d4aa4cfe
KAFKA-18296 Remove deprecated KafkaBasedLog constructor (#18257)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-12-20 15:14:45 +08:00
Nick Guo 0163fa2d06
KAFKA-18294 Remove deprecated SourceTask#commitRecord (#18260)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-12-19 01:06:33 +08:00
TengYao Chi b37b89c668
KAFKA-9366 Upgrade log4j to log4j2 (#17373)
This pull request replaces Log4j with Log4j2 across the entire project, including dependencies, configurations, and code. The notable changes are listed below:

1. Introduce Log4j2 Instead of Log4j
2. Change Configuration File Format from Properties to YAML
3. Adds warnings to notify users if they are still using Log4j properties, encouraging them to transition to Log4j2 configurations

Co-authored-by: Lee Dongjin <dongjin@apache.org>

Reviewers: Luke Chen <showuon@gmail.com>, Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-12-14 01:14:31 +08:00
Logan Zhu 92352a96e8
MINOR: ensure SuppressWarnings annotation is effective for mockValidationIsolation (#18158)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-12-13 07:57:29 +08:00
snehashisp f4fe6064cc
KAFKA-18215: KIP-891 Connect Multiversioning Support (Configs and Validation changes for Connectors and Converters) (#17741)
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-12-11 15:34:21 -08:00
TengYao Chi de2ccb5789
KAFKA-18021: Disabled MirrorCheckpointConnector throws RetriableException on task config generation (#18098)
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-12-11 13:56:38 -08:00
Christopher L. Shannon bd6d0fbf3d
KAFKA-16437 Upgrade to Jakarta and Jetty 12 (KIP-1032) (#16754)
This commit implements the changes for KIP-1032. This updates Kafka to Jakarta specs, JavaEE 10 and Jetty 12. The changes here primarily effect Kafka Connect and MM2.

Todo/Notes:

1) I bumped the connect modules to JDK 17 but I also had to bump a couple other things that had a dependency on conect. The tools project depends on connect so that had to be bumped, and streams depends on tools so that needed to be bumped. This means we may need to separate some things if we don't want to enforce JDK 17 on streams.

2) There is an issue with a test in DedicatedMirrorIntegrationTest that I had to change for now that involves escaping characters and not quite sure what to do about it yet. The cause is the Servlet 6 spec changing what is allowed in the path. See: Jetty 12: 400: Ambiguous URI path encoding for path <%=FOO%>~1 (encoded: %3C%25%3DFOO%25%3E%7E1) jetty/jetty.project#11890

3) I had to configure the idle timeout in Jetty requests to match our request timeout so tests didn't fail. This was needed to fix the ConnectWorkerIntegrationTest#testPollTimeoutExpiry() test

Testing is being done by just using the existing tests for Connect and MM2 which should be sufficient.

Reviewers: Greg Harris <greg.harris@aiven.io>, David Arthur <mumrah@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-12-11 23:24:14 +08:00
Dmitry Werner 45835a0e45
MINOR: Cleanup connect runtime module (#18074)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-12-10 18:25:38 +01:00
snehashisp af0054b502
KAFKA-18182: KIP-891 Add VersionRange to Plugins and DelegatingClassLoader APIs (#16984)
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-12-07 09:20:02 -08:00
Patrik Marton 0bbed823e8
KAFKA-17200: Allow the replication of user internal topics (#17815)
Reviewers: Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
2024-12-06 15:23:58 +01:00
Andrew Schofield e7d986e48c
KAFKA-17550: DescribeGroups v6 exploitation (#17706)
This PR introduces the DescribeGroups v6 API as part of KIP-1043. This adds an error message for the described groups so that it is possible to get some context on the error. It also changes the behaviour for when the group ID cannot be found but returning error code GROUP_ID_NOT_FOUND rather than NONE.

Reviewers: David Jacot <djacot@confluent.io>
2024-12-05 23:12:24 -08:00
Nick Guo 970d8930ac
KAFKA-18159 Remove onPartitionsRevoked and onPartitionsAssigned from SinkTask (#18049)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-12-06 04:28:34 +08:00
Ken Huang 2b43c49f51
KAFKA-18050 Upgrade the checkstyle version to 10.20.2 (#17999)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-12-05 10:59:18 +08:00
TengYao Chi 6fd951a9c0
KAFKA-17610 Drop alterConfigs (#18002)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-12-02 23:26:06 +08:00
TengYao Chi 056a76e2b9
KAFKA-17811 Separate modules to use different JDKs (#17522)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-11-26 23:30:41 +08:00
Manikumar Reddy 3268435fd6
KAFKA-18013: Add AutoOffsetResetStrategy internal class (#17858)
- Deprecates OffsetResetStrategy enum
- Adds new internal class AutoOffsetResetStrategy
- Replaces all OffsetResetStrategy enum usages with AutoOffsetResetStrategy
- Deprecate old/Add new constructors to MockConsumer

 Reviewers: Andrew Schofield <aschofield@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2024-11-25 19:11:12 +05:30
Logan Zhu 337e2e1506
KAFKA-18037 Remove deprecated methods from EmbeddedConnect and EmbeddedConnectCluster (#17857)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-11-23 23:16:41 +08:00
Yash Mayya 4f1688742e
KAFKA-15387: Remove Connect's deprecated task configurations endpoint (#17412)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-11-21 19:43:54 +05:30
Joao Pedro Fonseca Dantas e9ccc2d6f5
KAFKA-16041: Replace Afterburn module with Blackbird (#17884)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-11-21 14:52:45 +01:00
Andrew Schofield 32c887b05e
KAFKA-17949: Introduce GroupState and replace ShareGroupState (#17763)
This PR introduces the unified GroupState enum for all group types from KIP-1043. This PR also removes ShareGroupState and begins the work to replace Admin.listShareGroups with Admin.listGroups. That will complete in a future PR.

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
2024-11-19 21:17:12 +05:30
Mickael Maison 624cd4f7d0
MINOR: Various cleanups in connect:runtime tests (#17827)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-11-19 15:55:54 +08:00
Rajini Sivaram 52d2fa5c8b
KAFKA-17885: Enable clients to rebootstrap based on timeout or error code (KIP-1102) (#17720)
Implementation of https://cwiki.apache.org/confluence/display/KAFKA/KIP-1102%3A+Enable+clients+to+rebootstrap+based+on+timeout+or+error+code
- Introduces rebootstrap trigger interval config metadata.recovery.rebootstrap.trigger.ms, set to 5 minutes by default
- Makes rebootstrap the default for metadata.recovery.strategy
- Adds new error code REBOOTSTRAP_REQUIRED, introduces top-level error code in metadata response. On this error, clients rebootstrap.
- Configs apply to producers, consumers, share consumers, admin clients, Connect and KStreams clients.

Reviewers: Apoorv Mittal <apoorvmittal10@gmail.com>,  Manikumar Reddy <manikumar.reddy@gmail.com>
2024-11-13 13:01:08 +00:00
Linsiyuan9 af53758746
KAFKA-17814 Use `final` declaration to replace the suppression `this-escape` (#17613)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-11-03 15:00:02 +08:00
David Arthur 4fadec527d
MINOR: Quarantine the worst flaky tests (#17639)
Using the last 7 days of data on Oct 30 2024, this patch marks all flaky tests with more than 10% flakiness on trunk.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-10-30 17:02:04 -04:00
Ken Huang 09d76f917c
KAFKA-16564 Apply `Xlint` to java code in core module (#16965)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-10-26 01:07:56 +08:00
Joao Pedro Fonseca Dantas 3856644cd6
KAFKA-17233: MirrorCheckpointConnector should use batched listConsumerGroupOffsets (#17038)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Greg Harris <gharris1727@gmail.com>
2024-10-24 09:38:44 +02:00
Alieh Saeedi 14a098b289
KAFKA-17600: Add nextOffsets to the ConsumerRecords (#17414)
This PR implements KIP-1094.

Reviewers: Andrew Schofield <aschofield@confluent.io>, Kirk True <ktrue@confluent.io>, Lucas Brutschy <lbrutschy@confluent.io>
2024-10-23 16:25:50 +02:00
Dmitry Werner eb897c6ad5
MINOR: Fix potential NPE (#17541)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-10-19 01:05:45 +08:00
Kuan-Po Tseng f7498a40f2
MINOR: fix broken link in javadoc (#17526)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-10-18 00:50:27 +08:00
Anshul Goyal 2353a7c508
KAFKA-17723 Fix "this-escape" compiler warnings (MultiThreadedEventProcessor and DistributedHerder) for JDK 23 (#17417)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-10-11 21:28:27 +08:00
Gaurav Narula b03fe66cfe
KAFKA-17759 Remove Utils.mkSet (#17460)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-10-11 21:20:43 +08:00
Dániel Urbán 6d650ff9e5
KAFKA-17534: Add configuration to disable the heartbeats topic replication (#17413)
Introducing heartbeats.replication.enabled to explicitly disable the default heartbeats topic replication.
This change implements KIP-1089.

Reviewers: Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
2024-10-10 14:21:36 +02:00
Mickael Maison 3815339e05
KAFKA-14158: Remove auto.include.jmx.reporter configuration (#17360)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-10-08 16:17:07 +02:00