eec8fd6a98
KAFKA-9228: Restart tasks on runtime-only connector config changes ( #16053 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-06-10 17:02:08 -04:00
271a1b42cd
KAFKA-16223 Migrate KafkaConfigBackingStoreTest to Mockito ( #16126 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-07 07:02:14 +08:00
52514a89e3
KAFKA-16858: Throw DataException from validateValue on array and map schemas without inner schemas ( #16161 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Chris Egerton <chrise@aiven.io>
2024-06-05 11:35:32 -07:00
0409003c43
KAFKA-16837, KAFKA-16838: Ignore task configs for deleted connectors, and compare raw task configs before publishing them ( #16122 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-06-04 09:36:24 -04:00
5cbc1d616a
KAFKA-16881: InitialState type leaks into the Connect REST API OpenAPI spec ( #16175 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2024-06-03 13:34:58 -04:00
9eb05fc729
KAFKA-16223 Replace EasyMock/PowerMock with Mockito for KafkaConfigBackingStoreTest ( #16164 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-02 20:32:54 +08:00
966f2eb3ef
Minor: Add URL to log for Connect RestClient ( #16166 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2024-06-01 16:25:16 -04:00
e974914ca5
MINOR: Code Cleanup - Connect Module ( #16066 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-31 04:55:00 +08:00
32b2b73f67
KAFKA-16844: Add ByteBuffer support for Connect ByteArrayConverter ( #16101 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2024-05-30 11:26:25 -04:00
3f3f3ac155
MINOR: Delete KafkaSecurityConfigs class ( #16113 )
...
Reviewers: Omnia Ibrahim <o.g.h.ibrahim@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-05-30 05:55:24 +08:00
64f699aeea
KAFKA-15853: Move general configs out of KafkaConfig ( #16040 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-05-28 16:22:54 +02:00
11ad5e8bca
MINOR: Refactor Values class to fix checkstyle, add benchmark, optimize exceptions ( #15469 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-05-23 13:23:18 -07:00
0ba15ad4d1
KAFKA-15905 Restarts of MirrorCheckpointTask should not permanently i… ( #15910 )
...
* KAFKA-15905 Restarts of MirrorCheckpointTask should not permanently interrupt offset translation
MirrorCheckpointTask reloads the last checkpoint at start,
OffsetSyncStore stores OffsetSyncs before reading till end.
If CheckpointTask cannot read checkpoints at startup,
use previous OffsetSyncStore load logic, with
warning log message about degraded offset translation.
Also addresses KAFKA-16622 Mirromaker2 first Checkpoint not emitted until
consumer group fully catches up once because the OffsetSyncStore store
is populated before reading to log end.
Co-Authored-By: Adrian Preston <prestona@uk.ibm.com>
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-05-23 17:17:56 +01:00
476d323f5a
KAFKA-16197: Print Connect worker specific logs on poll timeout expiry ( #15305 )
...
Reviewers: Luke Chen <showuon@gmail.com>, Greg Harris <greg.harris@aiven.io>
2024-05-20 17:00:04 -07:00
6aac009a2d
MINOR: Remove deprecated constructors from Connect's Kafka*BackingStore classes ( #15865 )
...
- These constructors were deprecated over 3 years ago in KAFKA-10021: Changed Kafka backing stores to use shared admin client to get end offsets and create topics #9780 .
- While these classes are not a part of Connect's public API, deprecation was still introduced instead of outright removal because they are useful utility classes that might've been used outside of Connect.
- The KafkaOffsetBackingStore's deprecated constructor was removed in KAFKA-14785: Connect offset read REST API #13434 .
- This patch removes the deprecated constructors for KafkaConfigBackingStore and KafkaStatusBackingStore.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-18 04:09:30 +08:00
93a5efc4b7
KAFKA-16223 Replace EasyMock/PowerMock with Mockito for KafkaConfigBackingStoreTest ( #15933 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-18 04:01:27 +08:00
7453549314
KAFKA-16671 enable test for ensureInternalEndpointIsSecured ( #15868 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-15 12:49:36 +08:00
e1771448cf
MINOR: Add coverage for MM2 heartbeats topic detection ( #15932 )
...
Reviewers: Greg Harris <gharris1727@gmail.com>
2024-05-14 10:11:34 +02:00
0cfc2983a5
KAFKA-16684: Fix flaky DedicatedMirrorIntegrationTest ( #15906 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2024-05-10 14:42:26 -04:00
5a9ccb6b77
KAFKA-16445: Add PATCH method for connector config ( #6934 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2024-05-09 21:59:09 +02:00
5f933ac840
MINOR: Correct connector scheduled rebalance logs ( #15875 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-08 12:32:47 +08:00
525b9b1d76
KAFKA-15018: Write connector tombstone offsets to secondary store before primary store ( #13801 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2024-05-07 14:29:02 -04:00
a4c6cefd10
KAFKA-14226: Introduce FieldPath abstraction and nested path support for ExtractField SMT ( #15379 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2024-05-07 14:07:18 -04:00
05df10449e
KAFKA-13328, KAFKA-13329 (2): Add custom preflight validation support for connector header, key, and value converters ( #14309 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-05-07 11:30:57 -04:00
55a00be4e9
MINOR: Replaced Utils.join() with JDK API. ( #15823 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-06 15:13:01 +08:00
41f5bf844d
KAFKA-16223 Replace EasyMock/PowerMock with Mockito for KafkaConfigBackingStoreTest (2/3) ( #15841 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-06 08:47:17 +08:00
b8b2415d5e
KAFKA-16592 Add a new constructor which invokes the existing constructor with default value for alternativeString ( #15762 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-24 20:58:29 +08:00
1b301b3020
KAFKA-15853 Move socket configs into org.apache.kafka.network.SocketServerConfigs ( #15772 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-23 17:39:36 +08:00
ecb2dd4cdc
KAFKA-15853 Move KafkaConfig log properties and docs out of core ( #15569 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Nikolay <nizhikov@apache.org>, Federico Valeri <fvaleri@redhat.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-04-20 04:14:23 +08:00
53ff1a5a58
KAFKA-15585: DescribeTopicPartitions client side change. ( #15470 )
...
Add the support for DescribeTopicPartitions API to AdminClient. For this initial implementation, we are simply loading all of the results into memory on the client side.
Reviewers: Andrew Schofield <aschofield@confluent.io>, Kirk True <ktrue@confluent.io>, David Jacot <djacot@confluent.io>, Artem Livshits <alivshits@confluent.io>, David Arthur <mumrah@gmail.com>
2024-04-18 12:09:14 -04:00
aee9724ee1
MINOR: Remove unneeded explicit type arguments ( #15736 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-17 21:55:58 +02:00
7b1c777b33
MINOR: Various cleanups in connect ( #15734 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-17 20:47:25 +08:00
363f4d2847
KAFKA-15853 Move consumer group and group coordinator configs out of core ( #15684 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-17 20:41:22 +08:00
a1ca788c99
KAFKA-15517: Improve MirrorMaker logging in case of authorization errors ( #15558 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-04-16 12:09:14 +02:00
8c0458861c
KAFKA-15853 Move KafkaConfig Replication properties and docs out of … ( #15575 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-16 15:28:35 +08:00
2118d85805
KAFKA-16223 Replace EasyMock/PowerMock with Mockito for KafkaConfigBackingStoreTest (1/3) ( #15520 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-03-29 15:39:36 -07:00
71bcac3b6a
MINOR: AbstractConfig cleanup ( #15597 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Chris Egerton <chrise@aiven.io>, Mickael Maison <mickael.maison@gmail.com>, Omnia G H Ibrahim <o.g.h.ibrahim@gmail.com>, Matthias J. Sax <matthias@confluent.io>
2024-03-28 13:27:41 -07:00
355873aa54
MINOR: Use CONFIG suffix in ZkConfigs ( #15614 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>, Omnia Ibrahim <o.g.h.ibrahim@gmail.com>
Co-authored-by: n.izhikov <n.izhikov@vk.team>
2024-03-28 15:52:34 +01:00
6f38fe5e0a
KAFKA-14588 ZK configuration moved to ZkConfig ( #15075 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-03-27 22:37:01 +08:00
4099774da9
KAFKA-16084: Simplify and deduplicate standalone herder test mocking ( #15389 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-03-26 14:24:37 -07:00
f1d741a9c1
KAFKA-16392: Stop emitting warning log message when parsing source connector offsets with null partitions ( #15562 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>
2024-03-20 15:54:22 +00:00
12a1d85362
KAFKA-12187 replace assertTrue(obj instanceof X) with assertInstanceOf ( #15512 )
...
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-03-20 10:36:25 +08:00
7b9f31e35c
KAFKA-16358: Sort transformations by name in documentation; add missing transformations to documentation; add hyperlinks ( #15499 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>
2024-03-15 16:26:36 +05:30
178761eb36
KAFKA-14683 Cleanup WorkerSinkTaskTest ( #15506 )
...
1) Rename WorkerSinkTaskMockitoTest back to WorkerSinkTaskTest
2) Tidy up the code a bit
3) rewrite "fail" by "assertThrow"
Reviewers: Omnia Ibrahim <o.g.h.ibrahim@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-03-15 03:50:57 +08:00
aa7bef414e
MINOR: Resolve SSLContextFactory.getNeedClientAuth deprecation ( #15468 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-03-13 11:45:38 -07:00
62998b7264
KAFKA-14683: Migrate WorkerSinkTaskTest to Mockito (3/3) ( #15316 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-03-06 10:31:33 -08:00
eea369af94
KAFKA-14588 Log cleaner configuration move to CleanerConfig ( #15387 )
...
In order to move ConfigCommand to tools we must move all it's dependencies which includes KafkaConfig and other core classes to java. This PR moves log cleaner configuration to CleanerConfig class of storage module.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-03-05 18:11:56 +08:00
99e511c706
KAFKA-16288, KAFKA-16289: Fix Values convertToDecimal exception and parseString corruption ( #15399 )
...
* KAFKA-16288: Prevent ClassCastExceptions for strings in Values.convertToDecimal
* KAFKA-16289: Values inferred schemas for map and arrays should ignore element order
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Chris Egerton <chrise@aiven.io>
2024-03-04 15:59:47 -08:00
553f45bca8
MINOR: Fix toString method of IsolationLevel ( #14782 )
...
Reviewers: Matthias J. Sax <matthias@confluent.io>, Ashwin Pankaj <apankaj@confluent.io>
2024-02-15 19:07:18 -08:00
794c52802c
KAFKA-14683: Migrate WorkerSinkTaskTest to Mockito (2/3) ( #15313 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-02-12 11:15:27 -08:00
d233eb98f7
KAFKA-14957: Update-Description-String ( #13909 )
...
HTML code for configs is auto-generated and for Kafka Streams config `state.dir` produces a confusing default value.
This PR adds a new property `alternativeString` to set a "default" value which should be rendered in HTML instead of the actual default value.
Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, @eziosudo <eziosudo@gmail.com>, Matthias J. Sax <matthias@confluent.io>
2024-02-10 12:46:51 -08:00
24a79c2613
KAFKA-14683: Migrate WorkerSinkTaskTest to Mockito (1/3) ( #14663 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-02-05 10:55:57 -08:00
4f0a405908
KAFKA-15575: Begin enforcing 'tasks.max' property for connectors ( #15180 )
...
Reviewers: Ashwin Pankaj <apankaj@confluent.io>, Greg Harris <greg.harris@aiven.io>
2024-02-01 11:33:04 -05:00
7bc4afee11
KAFKA-15675: Improve worker liveness check during Connect integration tests ( #15249 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>, Yash Mayya <yash.mayya@gmail.com>
2024-02-01 09:20:24 -05:00
4e6920cc5f
KAFKA-16196: Handle invalid whole value casts in the Cast transform gracefully ( #15267 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2024-01-26 19:04:39 +05:30
d1d6b5096f
KAFKA-16166: Generify RetryWithToleranceOperator, ErrorReporter, and WorkerTask ( #15233 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Chris Egerton <chrise@aiven.io>
2024-01-22 16:56:52 -08:00
93971465b6
KAFKA-15827: Prevent KafkaBasedLog subclasses from leaking passed-in clients ( #14763 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Chris Egerton <chrise@aiven.io>
2024-01-19 12:50:01 -08:00
e45e032b8d
KAFKA-16087: Avoid data race which drops wrong record when errors.tolerance=all ( #15154 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Chris Egerton <chrise@aiven.io>
2024-01-18 13:26:26 -08:00
72b70288eb
KAFKA-5863: Avoid NPE when RestClient calls expecting no-content receive content. ( #13294 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Hector Geraldino <hgeraldino@gmail.com>, Yash Mayya <yash.mayya@gmail.com>
2024-01-18 11:41:27 -08:00
a989329ee3
MINOR: Stop leaking threads in BlockingConnectorTest ( #12290 )
...
Reviewers: Kvicii <kvicii.yu@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>, Greg Harris <greg.harris@aiven.io>
2024-01-18 11:11:32 -05:00
988aea3f58
KAFKA-16051: Fixed deadlock in StandaloneHerder ( #15080 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>, Sagar Rao <sagarmeansocean@gmail.com>
2024-01-12 15:46:17 -08:00
65424ab484
MINOR: New year code cleanup - include final keyword ( #15072 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Sagar Rao <sagarmeansocean@gmail.com>
2024-01-11 17:53:35 +01:00
dbf00bcf45
KAFKA-16093: Fix spurious REST-related warnings on Connect startup ( #15149 )
...
Reviewers: Sagar Rao <sagarmeansocean@gmail.com>, Greg Harris <greg.harris@aiven.io>
2024-01-10 09:03:23 -05:00
c7e1fdca64
KAFKA-15988: Reuse embedded clusters across test cases in Connect OffsetsApiIntegrationTest suite ( #14966 )
...
Reviewers: Sudesh Wasnik <swasnik@confluent.io>, Sagar Rao <sagarmeansocean@gmail.com>, Yash Mayya <yash.mayya@gmail.com>, Greg Harris <greg.harris@aiven.io>
2024-01-09 10:32:39 -05:00
eccfb03e9a
MINOR: Small cleanups in Connect ( #15128 )
...
Reviewers: Divij Vaidya <diviv@amazon.com>
2024-01-05 11:00:10 +01:00
cd1eb634c5
KAFKA-15816: Fix leaked sockets in runtime tests ( #14764 )
...
This commit fixes to prevent socket leaks, mostly through clients that are left open.
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-01-01 11:38:11 +01:00
314de9f23c
KAFKA-15996: Improve JsonConverter performance ( #14992 )
...
Improve JsonConverter performance by using afterBurnModule of Jackson library.
Reviewers: Divij Vaidya <diviv@amazon.com>, Mickael Maison <mickael.maison@gmail.com>
2023-12-24 21:47:12 +01:00
05014badf2
KAFKA-15208: Upgrade Jackson dependencies to version 2.16.0 ( #13662 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com>
2023-12-19 15:31:24 +01:00
3f7eadaa00
KAFKA-15372: Reconfigure dedicated MM2 connectors after leadership change ( #14293 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Chris Egerton <chrise@aiven.io>
2023-12-12 13:57:56 -08:00
75ad884161
KAFKA-15906: Emit latest MM2 offset syncs every offset.flush.interval.ms ( #14967 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Chris Egerton <chrise@aiven.io>
2023-12-11 16:39:54 -08:00
2a5fbf2882
KAFKA-15563: Provide informative error messages when Connect REST requests time out ( #14562 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2023-12-11 16:48:16 -05:00
dc857fb6bf
KAFKA-13328, KAFKA-13329 (1): Add preflight validations for key, value, and header converter classes ( #14304 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Greg Harris <greg.harris@aiven.io>
2023-12-08 15:00:44 -05:00
964e73178b
KAFKA-14132; Replace EasyMock with Mockito in StandaloneHerderTest ( #12728 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-12-07 10:01:17 -05:00
a2d596c428
KAFKA-15816: Fix leaked sockets in mirror tests ( #14761 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-12-06 10:44:32 -08:00
a83bc2d977
KAFKA-13988: Enable replicating from latest offset with MirrorMaker 2 ( #14567 )
...
Reviewers: hudeqi <1217150961@qq.com>, Federico Valeri <fedevaleri@gmail.com>, Greg Harris <gharris1727@gmail.com>
2023-12-04 16:37:37 -05:00
e8edc279f4
MINOR: Correcting Javadoc for ConnectAssertions ( #14827 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>
---------
Co-authored-by: Yash Mayya <yash.mayya@gmail.com>
2023-11-27 15:29:08 +05:30
ebf435e718
KAFKA-15888: DistributedHerder log context should not use the same client ID for each Connect worker by default ( #14825 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-11-23 15:31:39 +05:30
5d717f2343
MINOR: Fix broken method link in DistributedHerder::writeToConfigTopicAsLeader Javadoc ( #14824 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-11-23 03:22:56 +05:30
1bc4de75a2
KAFKA-15470: Allow creating connectors in a stopped state ( #14704 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-11-15 11:37:50 +05:30
989618a56e
KAFKA-15800: Prevent DataExceptions from corrupting KafkaOffsetBackingStore ( #14718 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Yash Mayya <yash.mayya@gmail.com>
2023-11-10 08:40:16 -08:00
39c6170aa9
KAFKA-15693: Immediately reassign lost connectors and tasks when scheduled rebalance delay is disabled ( #14647 )
...
Reviewers: Sagar Rao <sagarmeansocean@gmail.com>, Yash Mayya <yash.mayya@gmail.com>
2023-11-09 10:48:43 -05:00
ca05da95c3
KAFKA-15680: Fix sink task partition-count metric when cooperative consumer protocol is used ( #14630 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-11-06 11:32:05 -05:00
4083cd627e
KAFKA-15607: Fix NPE in MirrorCheckpointTask::syncGroupOffset ( #14587 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-10-20 12:17:51 -04:00
091eb9b349
KAFKA-15428: Cluster-wide dynamic log adjustments for Connect ( #14538 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>, Yang Yang <yayang@uber.com>, Yash Mayya <yash.mayya@gmail.com>
2023-10-20 09:52:37 -04:00
9b468fb278
MINOR: Do not end Javadoc comments with `**/` ( #14540 )
...
Reviewers: Bruno Cadonna <bruno@confluent.io>, Bill Bejeck <bill@confluent.io>, Hao Li <hli@confluent.io>, Josep Prat <josep.prat@aiven.io>
2023-10-17 21:11:04 -07:00
4150595b0a
KAFKA-14684: Replace EasyMock/PowerMock with Mockito in WorkerSinkTaskThreadedTest ( #14505 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Christo Lolov <christololov@gmail.com>
2023-10-16 11:24:52 +02:00
1c8bb61a43
KAFKA-15387: Deprecate Connect's redundant task configurations endpoint ( #14361 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Sagar Rao <sagarmeansocean@gmail.com>
2023-10-14 14:46:50 +05:30
4bad90835b
KAFKA-15465: Don't throw if MirrorMaker not authorized to create internal topics. ( #14388 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Ahmed Hibot
2023-10-13 12:53:09 +02:00
41c0d44402
KAFKA-15570: Add unit tests for MemoryConfigBackingStore ( #14518 )
...
Reviewers: Chris Egerton <chrise@aiven.io>, Kalpesh Patel <kpatel@confluent.io>
2023-10-12 14:25:40 +05:30
fbc39bcdbe
KAFKA-15547: Fix thread leak in MirrorMakerConfigTest ( #14487 )
...
Reviewers: Akhilesh Chaganti <akhileshchg@users.noreply.github.com>, Rajini Sivaram <rajinisivaram@googlemail.com>, Yash Mayya <yash.mayya@gmail.com>
2023-10-05 10:19:15 +05:30
9c2e5daf60
MINOR: Revert log level changes in LogCaptureAppender ( #14436 )
...
LogCaptureAppender sets the log level in various tests to check if a certain log message is produced. The log level is however never reverted, changing the log level across the board and introducing flakiness due to non-determinism since the log level depends on execution order. Some log messages change the timing inside tests significantly.
Reviewer: Bruno Cadonna <cadonna@apache.org>
2023-09-26 10:49:41 +02:00
7107a750ba
KAFKA-15248 (KIP-959): Add BooleanConverter to Kafka Connect ( #14093 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Sagar Rao <sagarmeansocean@gmail.com>, Qichao Chu <5326144+ex172000@users.noreply.github.com>
2023-09-26 12:42:16 +05:30
98febb989a
KAFKA-15485: Fix "this-escape" compiler warnings introduced by JDK 21 (1/N) ( #14427 )
...
This is one of the steps required for kafka to compile with Java 21.
For each case, one of the following fixes were applied:
1. Suppress warning if fixing would potentially result in an incompatible change (for public classes)
2. Add final to one or more methods so that the escape is not possible
3. Replace method calls with direct field access.
In addition, we also fix a couple of compiler warnings related to deprecated references in the `core` module.
See the following for more details regarding the new lint warning:
https://www.oracle.com/java/technologies/javase/21-relnote-issues.html#JDK-8015831
Reviewers: Divij Vaidya <diviv@amazon.com>, Satish Duggana <satishd@apache.org>, Chris Egerton <chrise@aiven.io>
2023-09-24 05:59:29 -07:00
b088307612
KAFKA-15473: Hide duplicate plugins in /connector-plugins ( #14398 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Sagar Rao <sagarmeansocean@gmail.com>, Hector Geraldino <hgeraldino@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-09-19 12:37:21 -04:00
7872a1ff5b
KAFKA-14855: Harden integration testing logic for asserting that a connector is deleted ( #14371 )
...
Reviewers: Sagar Rao <sagarmeansocean@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-09-19 11:39:39 -04:00
54ab5b29e4
KAFKA-15416: Fix flaky TopicAdminTest::retryEndOffsetsShouldRetryWhenTopicNotFound test case ( #14313 )
...
Reviewers: Philip Nee <pnee@confluent.io>, Greg Harris <greg.harris@aiven.io>
2023-09-07 19:24:17 -04:00
88b554fdbd
KAFKA-15179: Add integration tests for the file sink and source connectors ( #14279 )
...
Reviewers: Ashwin Pankaj <apankaj@confluent.io>, Chris Egerton <chrise@aiven.io>
2023-09-07 12:24:13 -04:00
b49013b73e
KAFKA-9800: Exponential backoff for Kafka clients - KIP-580 ( #14111 )
...
Implementation of KIP-580 to add exponential back-off to situations in which retry.backoff.ms
is used to delay backoff attempts. This KIP adds exponential backoff behavior with a maximum
controlled by a new config retry.backoff.max.ms, together with a +/- 20% of jitter to spread the
retry attempts of the client fleet.
Reviewers: Mayank Shekhar Narula <mayanks.narula@gmail.com>, Milind Luthra <i.milind.luthra@gmail.com>, Kirk True <kirk@mustardgrain.com>, Jun Rao<junrao@gmail.com>
2023-09-05 11:57:51 -07:00
1c5020e142
KAFKA-13327: Gracefully report connector validation errors instead of returning 500 responses ( #14303 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Greg Harris <greg.harris@aiven.io>
2023-08-29 15:44:22 -04:00
8611d28b2e
KAFKA-15392: Prevent shadowing RestServer shutdown exceptions ( #14277 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-29 12:25:30 -07:00
880288879e
KAFKA-15377: Don't expose externalized secret values in tasks-config API endpoint ( #14244 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2023-08-24 15:44:53 -07:00
a3303b6112
KAFKA-15393: Improve shutdown behavior in MM2 integration tests ( #14278 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-08-24 12:24:34 -07:00
064a14795c
MINOR: Update OpenAPI summary documentation for Connect's /tasks endpoint ( #14267 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-08-22 18:26:29 +02:00
7802c264c9
MINOR: Allow writing tombstone offsets for arbitrary partitions in the FileStreamSourceConnector ( #14234 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-17 14:13:53 -04:00
de409b389d
KAFKA-15177: Implement KIP-875 SourceConnector::alterOffset API in MirrorMaker 2 connectors ( #14005 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Greg Harris <greg.harris@aiven.io>
2023-08-17 09:33:59 -04:00
6bd17419b7
KAFKA-15228: Add sync-manifests command to connect-plugin-path (KIP-898) ( #14195 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-16 11:37:33 -07:00
a9efca0bf6
KAFKA-14759: Move Mock, Schema, and Verifiable connectors to new test-plugins module ( #13302 )
...
Reviewers: Hector Geraldino <hgeraldino@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-08-16 10:30:24 -07:00
35e925f353
KAFKA-15102: Add replication.policy.internal.topic.separator.enabled property to MirrorMaker 2 (KIP-949) ( #14082 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-15 19:58:52 -04:00
28858f3a3e
MINOR: Fix SynchronizationTest classloaders sometimes not being parallel capable ( #14177 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-15 14:24:48 -07:00
1a001c1e88
KAFKA-15336: Add ServiceLoader Javadocs for Connect plugins ( #14194 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-15 13:21:45 -07:00
f5655d31d3
KAFKA-15030: Add connect-plugin-path command-line tool ( #14064 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-11 12:05:51 -07:00
f137da04fa
KAFKA-14132: Replace Easymock & Powermock with Mockito in KafkaBasedLogTest ( #14153 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Divij Vaidya <diviv@amazon.com>
2023-08-11 10:50:37 +02:00
8de3e0436a
KAFKA-15239: Fix system tests using producer performance service ( #14092 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2023-08-10 14:23:43 -07:00
f2ebd333e8
KAFKA-13187: Replace EasyMock / PowerMock with Mockito in DistributedHerderTest ( #14102 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-10 16:47:30 -04:00
0ee26640e5
KAFKA-15202: Fix MM2 offset translation when syncs are variably spaced ( #14156 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-10 13:25:50 -07:00
db34f8b9a1
KAFKA-15291: Connect plugins should declare a version ( #14159 )
...
Signed-off-by: Aindriu Lavelle <aindriu.lavelle@aiven.io>
Reviewers: Andrew Schofield, Greg Harris <greg.harris@aiven.io>
2023-08-10 10:45:12 -07:00
f23394336a
MINOR: Use Mockito's strict stubs functionality for Connect tests and cleanup unused stubs ( #14166 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-08-09 11:13:28 +02:00
ff4fed5cbe
KAFKA-15031: Add plugin.discovery to Connect worker configuration (KIP-898) ( #14055 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-08 10:06:35 -07:00
b9936d6292
KAFKA-7438: Replace PowerMockRunner with MockitoJUnitRunner in RetryUtilTest ( #14143 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-03 13:07:35 -04:00
b9a45546a7
KAFKA-15244: Remove PluginType.from(Class) ( #14089 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-01 10:05:46 -07:00
3ba718e159
MINOR: Remove duplicate instantiation of MockConnectMetrics in AbstractWorkerSourceTaskTest ( #14091 )
...
Reviewers: Christo Lolov <christololov@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Greg Harris <greg.harris@aiven.io>
2023-07-28 11:35:42 -07:00
08b3820d5e
KAFKA-15238: Move DLQ reporter setup from the DistributedHerder's tick thread to the sink task thread ( #14079 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-25 09:03:29 -04:00
58b8c5c7b1
MINOR: Downgrade log level for conflicting Connect plugin aliases ( #14081 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>
2023-07-25 08:12:46 -04:00
4daeb2714c
KAFKA-13431 (KIP-793): Expose the original pre-transform topic partition and offset in sink records ( #14024 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>, Chris Egerton <chrise@aiven.io>
2023-07-21 12:06:01 -04:00
125dbb9286
KAFKA-14760: Move ThroughputThrottler from tools to clients, remove tools dependency from connect-runtime ( #13313 )
...
Reviewers: Ismael Juma <ismael@juma.me.uk>
2023-07-20 12:58:48 -07:00
ea6e100ec3
KAFKA-15216: InternalSinkRecord::newRecord should not ignore new headers ( #14044 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-20 10:22:35 -04:00
f6e7aa3763
KAFKA-15150: Add ServiceLoaderScanner implementation ( #13971 )
...
Reviewers: Chris Egerton <chris.egerton@aiven.io>
2023-07-19 16:21:58 -07:00
844469311a
KAFKA-10579: Upgrade reflections from 0.9.12 to 0.10.2 ( #14029 )
...
Reviewers: Divij Vaidya <diviv@amazon.com>, Chris Egerton <chris.egerton@aiven.io>
2023-07-19 14:00:15 -07:00
d5a00cca74
KAFKA-15162: Reflectively find plugins in parent ClassLoaders that aren't on the classpath ( #13977 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewed-by: Chris Egerton <chris.egerton@aiven.io>
2023-07-19 12:49:25 -07:00
a68281bf3c
MINOR: Fix Connect compatibility javadocs to use NoSuchMethodError instead of NoSuchMethodException ( #14036 )
...
Reviewed-by: Greg Harris <greg.harris@aiven.io>
2023-07-19 10:23:02 -07:00
1f7f6ceebf
KAFKA-15211: Mock InvalidParameterException in DistributedConfigTest ( #14039 )
...
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewed-by: Chris Egerton <chris.egerton@aiven.io>
2023-07-19 09:02:44 -07:00
d6aaddf6ee
KAFKA-15091: Fix misleading Javadoc for SourceTask::commit ( #13948 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-18 16:38:30 -04:00
5cb2504c49
KAFKA-14669: Use the generated docs for MirrorMaker configs in the doc ( #13658 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-18 11:27:19 -04:00
0b025b7711
KAFKA-15121: Implement the alterOffsets method in the FileStreamSourceConnector and the FileStreamSinkConnector ( #13945 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-18 10:37:51 -04:00
2b19b7325a
KAFKA-15182: Normalize source connector offsets before invoking SourceConnector::alterOffsets ( #14003 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-14 12:14:03 -04:00
8d24716f27
MINOR: Avoid slow Set.removeAll(List) in MirrorSourceConnector ( #13992 )
...
Reviewed-by: Greg Harris <greg.harris@aiven.io>
2023-07-12 12:19:35 -07:00
d114d8e29c
KAFKA-14938: Fixing flaky test testConnectorBoundary ( #13646 )
...
Reviewers: Sagar Rao <sagarmeansocean@gmail.com>, Yash Mayya <yash.mayya@gmail.com>,
Sudesh Wasnik <swasnik@confluent.io>, Chris Egerton <chrise@aiven.io>
2023-07-12 11:45:49 -04:00
170f5f4ed0
KAFKA-15148: Mark tests correctly as integration tests where they running as unit tests ( #13973 )
...
Reviewers: Divij Vaidya <diviv@amazon.com>
2023-07-12 13:41:58 +02:00
adacfea2d6
KAFKA-12525: Ignoring stale status statuses when reading from Connect status topic ( #13453 )
...
During fast consecutive rebalances where a task is revoked from one worker and assigned to another one, it has been observed that there is a small time window and thus a race condition during which a RUNNING status record in the new generation is produced and is immediately followed by a delayed UNASSIGNED status record belonging to the same or a previous generation before the worker that sends this message reads the RUNNING status record that corresponds to the latest generation.
Although this doesn't inhibit the actual execution of tasks, it reports an incorrect status for those tasks(i.e UNASSIGNED). If the users have setup some kind of monitoring on tasks status then this could lead to false alarms for example.
This fix addresses this problem by checking if a status message is stale after reading it and updates it's status only when it is safe to.
Reviewers: Lucent-Wong <manchesterfans@live.cn>, Chris Egerton <chrise@aiven.io>, Yash Mayya <yash.mayya@gmail.com>, Konstantine Karantasis <k.karantasis@gmail.com>
2023-07-11 08:05:10 -07:00
51bc41031b
KAFKA-15139: Avoid slow Set.removeAll(List) in MirrorCheckpointConnector ( #13946 )
...
Reviewed-by: Greg Harris <greg.harris@aiven.io>
2023-07-10 14:35:46 -07:00
9ee28d1fe6
KAFKA-15145: Don't re-process records filtered out by SMTs on Kafka client retriable exceptions in AbstractWorkerSourceTask ( #13955 )
...
Reviewers: Sagar Rao <sagarmeansocean@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-07-10 13:26:58 -04:00
6368d14a1d
KAFKA-14059 Replace PowerMock with Mockito in WorkerSourceTaskTest ( #13383 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-10 12:58:54 -04:00
d9a3e60dcc
KAFKA-14718: Wait for MirrorMaker to start before executing test ( #13284 )
2023-07-10 12:53:01 -04:00
1b925e9ee7
KAFKA-15069: Refactor plugin scanning logic into ReflectionScanner ( #13821 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-06 13:22:28 -04:00
a1a3ec0bcb
MINOR: Update connector status metric description to include 'stopped' as a potential value ( #13967 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-07-06 15:28:07 +02:00
96e59d7bfd
[MINOR] Correcting few WARN log lines in DistributedHerder#handleRebalance ( #13939 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-30 12:39:37 -04:00
1f4cbc5d53
MINOR: Add JDK 20 CI build and remove some branch builds ( #12948 )
...
It's good for us to add support for Java 20 in preparation for Java 21 - the next LTS.
Given that Scala 2.12 support has been deprecated, a Scala 2.12 variant is not included.
Also remove some branch builds that add load to the CI, but have
low value: JDK 8 & Scala 2.13 (JDK 8 support has been deprecated),
JDK 11 & Scala 2.12 (Scala 2.12 support has been deprecated) and
JDK 17 & Scala 2.12 (Scala 2.12 support has been deprecated).
A newer version of Mockito (4.9.0 -> 4.11.0) is required for Java 20 support, but we
only use it with Scala 2.13+ since it causes compilation errors with Scala 2.12. Similarly,
we upgrade easymock when the Java version is 16 or newer as it's incompatible
with powermock (which doesn't support Java 16 or newer).
Filed KAFKA-15117 for a test that fails with Java 20 (SslTransportLayerTest.testValidEndpointIdentificationCN).
Finally, fixed some lossy conversions that were added after #13582 was submitted.
Reviewers: Ismael Juma <ismael@juma.me.uk>
2023-06-30 01:12:00 -07:00
32bcdac6a1
MINOR: Replace synchronization with atomic update in Connect's StateTracker::changeState method ( #13934 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-29 15:05:06 -04:00
005416879e
KAFKA-15053: Use case insensitive validator for security.protocol config ( #13831 )
...
Fixed a regression described in KAFKA-15053 that security.protocol only allows uppercase values like PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. With this fix, both lower case and upper case values will be supported (e.g. PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL, plaintext, ssl, sasl_plaintext, sasl_ssl)
Reviewers: Chris Egerton <chrise@aiven.io>, Divij Vaidya <diviv@amazon.com>
2023-06-29 10:13:21 +02:00
3978684126
MINOR: Silence error logs for faulty plugins in integration tests ( #13912 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-23 14:10:18 -04:00
6e72986949
KAFKA-14784: Connect offset reset REST API ( #13818 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-23 13:27:46 -04:00
3c059133d3
MINOR: Fix generated client ids for Connect ( #13896 )
...
Reviewers: Chris Egerton <fearthecellos@gmail.com>
2023-06-21 21:44:14 +02:00
3b72b0abb1
MINOR: Optimize runtime of MM2 integration tests by batching transactions ( #13816 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-21 14:51:54 -04:00
d751c13950
MINOR: Fix typos for connect ( #13885 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-20 16:16:16 -04:00
73dd51e137
Revert "MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits ( #13838 )" ( #13864 )
...
Reviewers: Josep Prat <josep.prat@aiven.io>
Reverts commit 505c7b6487
2023-06-16 12:10:26 -04:00
e1d59920f4
KAFKA-15059: Remove pending rebalance check when fencing zombie source connector tasks ( #13819 )
...
Discovered while researching KAFKA-14718
Currently, we perform a check during zombie fencing that causes the round of zombie fencing to fail when a rebalance is pending (i.e., when we've detected from a background poll of the config topic that a new connector has been created, that an existing connector has been deleted, or that a new set of connector tasks has been generated).
It's possible but not especially likely that this check causes issues when running vanilla Kafka Connect. Even when it does, it's easy enough to restart failed tasks via the REST API.
However, when running MirrorMaker 2 in dedicated mode, this check is more likely to cause issues as we write three connector configs to the config topic in rapid succession on startup. And in that mode, there is no API to restart failed tasks aside from restarting the worker that they are hosted on.
In either case, this check can lead to test flakiness in integration tests for MirrorMaker 2 both in dedicated mode and when deployed onto a vanilla Kafka Connect cluster.
This check is not actually necessary, and we can safely remove it. Copied from Jira:
>If the worker that we forward the zombie fencing request to is a zombie leader (i.e., a worker that believes it is the leader but in reality is not), it will fail to finish the round of zombie fencing because it won't be able to write to the config topic with a transactional producer.
>If the connector has just been deleted, we'll still fail the request since we force a read-to-end of the config topic and refresh our snapshot of its contents before checking to see if the connector exists.
>And regardless, the worker that owns the task will still do a read-to-end of the config topic and verify that (1) no new task configs have been generated for the connector and (2) the worker is still assigned the connector, before allowing the task to process any data.
In addition, while waiting on a fix for KAFKA-14718 that adds more granularity for diagnosing failures in the DedicatedMirrorIntegrationTest suite (#13284 ), some of the timeouts in that test are bumped to work better on our CI infrastructure.
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
2023-06-16 11:58:36 +02:00
505c7b6487
MINOR: Reduce MM2 integration test flakiness due to missing dummy offset commits ( #13838 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-06-15 12:02:32 +02:00
930744c3a8
KAFKA-14709: Move content in connect/mirror/README.md to the docs ( #13650 )
...
Most of the contents in the README.md was already covered in the docs therefore only had to add the section for Exactly Once support.
Reviewers: Luke Chen <showuon@gmail.com>
2023-06-15 10:16:52 +08:00
6b128d7e30
KAFKA-14006: Parameterize WorkerConnectorTest suite ( #12307 )
...
Reviewers: Sagar Rao <sagarmeansocean@gmail.com>, Christo Lolov <lolovc@amazon.com>, Kvicii <kvicii.yu@gmail.com>, Mickael Maison <mickael.maison@gmail.com>
2023-06-08 11:20:35 -04:00
c8cb85274e
MINOR: Refactor DelegatingClassLoader to emit immutable PluginScanResult ( #13771 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-06 09:37:57 -04:00
17fd30e6b4
MINOR: Fix flaky DistributedHerderTest cases related to zombie fencing ( #13806 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-06-05 15:50:54 -04:00
383a8d6114
MINOR: Handle the config topic read timeout edge case in DistributedHerder's stopConnector method ( #13750 )
...
Reviewers: Chris Egerton
2023-06-05 11:10:59 -04:00
fca7ee7270
MINOR: Remove reference to 'offset backing store' from exception message in KafkaBasedLog ( #13810 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-05 10:04:17 -04:00
8e60368d90
MINOR: Re-introduce Transformation import to fix TransformationChain Javadoc ( #13808 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-05 09:43:03 -04:00
6678f1b66a
KAFKA-14863: Fix failing tests for invalid plugins that are no longer visible ( #13805 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-03 12:55:20 -04:00
02fb4b882b
KAFKA-15012: Allow leading zeros in numeric fields while deserializing JSON messages using the JsonConverter ( #13800 )
...
Reviewers: Sagar Rao <sagarmeansocean@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-06-02 11:20:37 -04:00
89581a738f
KAFKA-14863: Hide plugins with invalid constructors during plugin discovery ( #13467 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-02 11:09:43 -04:00
9bb2f78d53
KAFKA-15034: Improve performance of the ReplaceField SMT; add JMH benchmark ( #13776 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-01 15:14:31 -04:00
6846f51445
MINOR: Remove spurious warning about plugin.path config provider usage when null ( #13783 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-05-31 12:57:58 -04:00
03ab563206
MINOR: Fixing typo in ExactlyOnceWorkerSourceTask#committableRecords ( #13755 )
...
Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
2023-05-31 16:26:31 +02:00
1957be19d9
KAFKA-8713: JsonConverter replace.null.with.default should prevent emitting default for Struct fields ( #13781 )
...
Co-authored-by: GeunJae Jeon <krespo>
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-05-31 15:48:03 +02:00
9aac5ff1fe
MINOR: Move plugin path parsing from DelegatingClassLoader to PluginUtils ( #13334 )
...
Reviewers: Chaitanya Mukka <chaitanya.mvs2007@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-05-26 12:36:00 -04:00
2ec6b5e1e2
KAFKA-14789: Prevent mis-attributing classpath plugins, allow discovery of classpath RestExtension and ConfigProvider ( #13356 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-05-26 12:17:07 -04:00
7ff2dbb107
KAFKA-14368: Connect offset write REST API ( #13465 )
...
Reviewers: Greg Harris <greg.harris@aiven.io>, Chris Egerton <chrise@aiven.io>
2023-05-26 12:08:06 -04:00
dc00832b96
KAFKA-14654: Connector classes should statically initialize with plugin classloader ( #13165 )
...
Reviewers: Chaitanya Mukka <chaitanya.mvs2007@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-05-25 10:22:44 -04:00
916e801010
MINOR: Fix whitespace typo in Connect Config docs ( #13752 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Yash Mayya <yash.mayya@gmail.com>, Divij Vaidya <diviv@amazon.com>
2023-05-24 21:48:47 +02:00
5b3b385881
KAFKA-13504: Retry connect internal topics' creation in case of InvalidReplicationFactorException ( #11565 )
...
In case the Kafka Broker cluster and the Kafka Connect cluster is started together and Connect would want to create its topics, there's a high chance to fail the creation with InvalidReplicationFactorException.
---------
Co-authored-by: Daniel Urban <durban@cloudera.com>
Reviewers: Daniel Urban <durban@cloudera.com>, Mickael Maison <mickael.maison@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>, Chris Egerton <chrise@aiven.io>, Laszlo Hunyadi <laszlo.istvan.hunyady@gmail.com>
2023-05-24 11:05:02 +02:00
a14e73a036
KAFKA-14980: Fix MirrorSourceConnector source consumer configuration ( #13723 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Josep Prat <josep.prat@aiven.io>
2023-05-19 14:45:01 +02:00
f17fb75b2d
KAFKA-14978 ExactlyOnceWorkerSourceTask should remove parent metrics ( #13690 )
...
Reviewers: Chris Egerton <chrise@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
2023-05-10 16:41:52 +02:00
59ba9dbbc9
KAFKA-14974: Restore backward compatibility in KafkaBasedLog ( #13688 )
...
`KafkaBasedLog` is a widely used utility class that provides a generic implementation of a shared, compacted log of records in a Kafka topic. It isn't in Connect's public API, but has been used outside of Connect and we try to preserve backward compatibility whenever possible. KAFKA-14455 modified the two overloaded void `KafkaBasedLog::send` methods to return a `Future`. While this change is source compatible, it isn't binary compatible. We can restore backward compatibility simply by renaming the new Future returning send methods, and reinstating the older send methods to delegate to the newer methods.
This refactoring changes no functionality other than restoring the older methods.
Reviewers: Randall Hauch <rhauch@gmail.com>
2023-05-09 07:28:45 -05:00
86daf8ce65
KAFKA-14913: Using ThreadUtils.shutdownExecutorServiceQuietly to close executors in Connect Runtime ( #13594 )
...
#13557 introduced a utils method to close executors silently. This PR leverages that method to close executors in connect runtime. There was duplicate code while closing the executors which isn't the case with this PR.
Note that there are a few more executors used in Connect runtime but their close methods don't follow this pattern of shutdown, await and shutdown. Some of them have some logic like executor like Worker, so not changing at such places.
---------
Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>
Reviewers: Daniel Urban <durban@cloudera.com>, Yash Mayya <yash.mayya@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
2023-05-08 16:39:47 +02:00
f44ee4fab7
MINOR: Remove unnecessary code in client/connect ( #13259 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-05-02 17:39:31 +02:00
4773961a44
MINOR: Fix Javadoc for configureAdminResources in Connect's RestServer ( #13635 )
...
Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-05-01 12:21:09 -04:00
007c0d375a
KAFKA-14929: Fixing flaky test putTopicStateRetriableFailure ( #13634 )
...
Co-authored-by: Sagar Rao <sagarrao@Sagars-MacBook-Pro.local>
Reviewers: Daniel Urban <durban@cloudera.com>, Justine Olshan <jolshan@confluent.io>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
2023-04-26 14:23:56 +02:00
baf127a663
KAFKA-14666: Add MM2 in-memory offset translation index for offsets behind replication ( #13429 )
...
Reviewers: Daniel Urban <durban@cloudera.com>, Chris Egerton <chrise@aiven.io>
2023-04-26 03:30:13 -04:00
4780dc773f
KAFKA-14933: Document Connect's log level REST APIs from KIP-495 ( #13636 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Manyanda Chitimbo <manyanda.chitimbo@gmail.com>
2023-04-25 17:50:35 +02:00
7061475445
KAFKA-14905: Reduce flakiness in MM2 ForwardingAdmin test due to admin timeouts ( #13575 )
...
Reduce flakiness of `MirrorConnectorsWithCustomForwardingAdminIntegrationTest`
Reviewers: Josep Prat <jlprat@apache.org>
2023-04-21 21:55:41 +02:00
e14dd8024a
KAFKA-14821 Implement the listOffsets API with AdminApiDriver ( #13432 )
...
We are handling complex workflows ListOffsets by chaining together MetadataCall instances and ListOffsetsCall instances, there are many complex and error-prone logic. In this PR we rewrote it with the `AdminApiDriver` infra, notable changes better than old logic:
1. Retry lookup stage on receiving `NOT_LEADER_OR_FOLLOWER` and `LEADER_NOT_AVAILABLE`, whereas in the past we failed the partition directly without retry.
2. Removing class field `supportsMaxTimestamp` and calculating it on the fly to avoid the mutable state, this won't change any behavior of the client.
3. Retry fulfillment stage on `RetriableException`, whereas in the past we just retry fulfillment stage on `InvalidMetadataException`, this means we will retry on `TimeoutException` and other `RetriableException`.
We also `handleUnsupportedVersionException` to `AdminApiHandler` and `AdminApiLookupStrategy`, they are used to keep consistency with old logic, and we can continue improvise them.
Reviewers: Ziming Deng <dengziming1993@gmail.com>, David Jacot <djacot@confluent.io>
2023-04-20 11:29:27 +08:00
454b72161a
KAFKA-14902: KafkaStatusBackingStore retries on a dedicated background thread to avoid stack overflows ( #13557 )
...
KafkaStatusBackingStore uses an infinite retry logic on producer send, which can lead to a stack overflow.
To avoid the problem, a background thread was added, and the callback submits the retry onto the background thread.
2023-04-18 09:40:14 +02:00
f252c75bf3
MINOR: Fix regression in MM2 task forwarding introduced by KAFKA-14783 ( #13548 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-04-13 09:50:02 -04:00
9c0caca660
MINOR: update the connect task metrics documentation ( #13494 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-04-12 12:08:07 -04:00
e49a5a265f
KAFKA-14783 (KIP-875): New STOPPED state for connectors ( #13424 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Yash Mayya <yash.mayya@gmail.com>, Greg Harris <gharris1727@gmail.com>
2023-04-11 09:37:26 -04:00
751a8af1f0
KAFKA-14420: Use incrementalAlterConfigs API for syncing topic configurations in MirrorMaker 2 (KIP-894) ( #13373 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-04-10 11:55:49 -04:00
5e820571de
MINOR: Fix base ConfigDef in AbstractHerder::connectorPluginConfig ( #13466 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Greg Harris <gharris1727@gmail.com>
2023-04-04 11:57:42 +02:00
970dea60e8
KAFKA-14785 (KIP-875): Connect offset read REST API ( #13434 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-04-02 13:09:33 -04:00
0aa365add8
KAFKA-14838: Add flow/connector/task/role information to MM2 Kafka client.id configs ( #13458 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-03-31 10:50:11 -04:00
5afedd9ac3
KAFKA-14843: Include Connect framework properties when retrieving connector config definitions ( #13445 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Greg Harris <greg.harris@aiven.io>, Chris Egerton <chrise@aiven.io>
2023-03-28 11:26:23 -04:00
f7ea9cfb50
KAFKA-14837/14842:Avoid the rebalance caused by the addition and deletion of irrelevant groups for MirrorCheckPointConnector ( #13446 )
...
Reviewers: Chris Egerton <chrise@aiven.io>
2023-03-28 09:19:52 -04:00
3c4fb01ba8
MINOR: Refactor Mirror integration tests to reduce duplication ( #13428 )
...
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-03-24 16:18:26 +01:00
3c25b311cb
KAFKA-14814: Skip Connect target state updates when the configs store has same state ( #13426 )
...
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-03-23 11:23:38 -04:00
Chris Egerton
Johnny Hsu
Greg Harris
Ken Huang
Chia Chuan Yu
Farbod Ahmadian
Sanskar Jhajharia
Fan Yang
Mickael Maison
Omnia Ibrahim
Edoardo Comar
vamossagar12
Yash Mayya
PoAn Yang
Ivan Yurchenko
yuz10
Jorge Esteban Quilcate Otoya
Hector Geraldino
Calvin Liu
Dmitry Werner
Nikolay
Ahmed Sobeh
Kuan-Po (Cooper) Tseng
Minha, Jeong
Owen Leung
Octavian Ciubotaru
Divij Vaidya
Fiore Mario Vitale
Said Boudjelda
Matthew de Detrich
kumarpritam863
hudeqi
Matthias J. Sax
kpatelatwork
Lucas Brutschy
Ismael Juma
Andrew Schofield
bachmanity1
Federico Valeri
aindriu-aiven
Gantigmaa Selenge
Sambhav Jain
ezio
Bo Gao
minjian.cai
Sungyun Hur
Andras Katona
Dániel Urbán
Christo Lolov
Dimitar Dimitrov
dorwi
Jorge Esteban Quilcate Otoya
Chaitanya Mukka