Commit Graph

750 Commits

Author SHA1 Message Date
Kuan-Po Tseng 84add30ea5
KAFKA-16154: Broker returns offset for LATEST_TIERED_TIMESTAMP (#16783)
This pr support EarliestLocalSpec LatestTierSpec in GetOffsetShell, and add integration tests.

Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>, PoAn Yang <payang@apache.org>
2024-08-05 10:41:14 +08:00
Ken Huang fa682623b9
KAFKA-16666 Migrate OffsetMessageFormatter to tools module (#16689)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-07-31 15:18:14 +08:00
Josep Prat e299a006c8
KAFKA-17214: Add 3.8.0 version to streams system tests (#16728)
* KAFKA-17214: Add 3.8.0 version to streams system tests

Reviewers: Bill Bejeck <bbejeck@gmail.com>
2024-07-30 19:04:38 +02:00
Chris Egerton 61f61d6240
KAFKA-14569: Migrate Connect's integration test EmbeddedKafkaCluster from ZK to KRaft mode (#16599)
Reviewers: Omnia Ibrahim <o.g.h.ibrahim@gmail.com>, Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-07-29 10:43:55 -04:00
Ken Huang ee03c5f98a
KAFKA-17195 remove generated code path from test scrope of sourceSets (#16688)
The generated code path added to test scope of sourceSets will make `Gradle` assume there are tests in module, and then it results in failed test task since the module does not define test engine.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-07-25 18:43:46 +08:00
Ken Huang ee68f3215f
KAFKA-16666 Migrate `TransactionLogMessageFormatter` to tools module (#16019)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-07-25 00:42:25 +08:00
Ken Huang a012af5fb4
KAFKA-17149 Move ProducerStateManagerTest to storage module (#16645)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-07-23 20:08:08 +08:00
xijiu bc031c0410
KAFKA-17121 junit-platform.properties files in published artifacts pollute the test classpath of consumers (#16624)
Reviewers: Andy Wilkinson <ankinson@mastodon.org.uk>, Chia-Ping Tsai <chia7712@gmail.com>
2024-07-19 18:18:27 +08:00
Mickael Maison dceda5e647
MINOR: Don't include checker-qual in distribution (#16578)
Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>, Satish Duggana <satishd@apache.org>
2024-07-15 17:39:38 +02:00
Xuan-Zhang Gong 0ada8fac68
KAFKA-17096 Fix kafka_log4j_appender.py (#16559)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-07-12 22:35:55 +08:00
Vincent Rose 43676f7612
MINOR: Generate javadocs on all source files for streams:test-utils (#16556)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2024-07-09 11:32:16 -07:00
TingIāu "Ting" Kì 07133cebd2
KAFKA-16806 Explicitly declare JUnit dependencies for all test modules (#16301)
Reviewers: Greg Harris <gharris1727@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-07-06 06:39:11 +08:00
Greg Harris a33cc3d351
KAFKA-17052: Downgrade ShadowJarPlugin to 8.1.3 (#16489)
This version of the ShadowJarPlugin uses the incorrect classifier for the published archive. This is a temporary measure to fix publishing prior to the upcoming release.

Signed-off-by: Greg Harris <greg.harris@aiven.io>


Reviewers:  Manikumar Reddy <manikumar.reddy@gmail.com>
2024-06-29 11:26:39 +05:30
gongxuanzhang 1040d78372
KAFKA-10787 Apply spotless to all module (#16467)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-27 20:50:06 +08:00
Ken Huang 9b4f13efbc
KAFKA-15623 Remove junit 4 from stream module (#16447)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-27 15:11:32 +08:00
TingIāu "Ting" Kì 0e346d3103
KAFKA-15623 (4/N) Migrate streams tests (processor) module to JUnit 5 (#16396)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-24 13:06:35 +08:00
Murali Basani 4194e3d163
MINOR: Rename log4j variables to reload4j (#16324)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>, Greg Harris <gharris1727@gmail.com>
2024-06-21 10:43:07 +02:00
gongxuanzhang 8d92535382
KAFKA-10787 apply spotless to connect:runtime module (#16412)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-21 16:04:51 +08:00
Nancy 391778b8d7
KAFKA-16803: Change fork, update ShadowJavaPlugin to 8.1.7 (#16295)
Reviewers: Greg Harris <greg.harris@aiven.io>, Divij Vaidya <diviv@amazon.com>
2024-06-20 12:29:57 -07:00
gongxuanzhang 80f31224aa
KAFKA-10787 Apply spotless to `clients` module (#16393)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-20 17:43:25 +08:00
Ken Huang 4de6a90801
KAFKA-16921 [6/N] Remove junit 4 dependency from connect:runtime module (#16383)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-19 14:39:16 +08:00
gongxuanzhang 96989e4b64
KAFKA-10787 Apply spotless to core module (#16392)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-19 14:24:42 +08:00
gongxuanzhang 5a331acad0
KAFKA-10787 apply spotless to `streams:examples` and `streams-scala` (#16378)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-18 18:56:46 +08:00
gongxuanzhang 09b4ef416a
KAFKA-10787 Apply spotless to `stream:test-utils` and `:streams:upgrade-system-tests-xxxx` (#16357)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-17 19:48:44 +08:00
gongxuanzhang 4e846038a6
KAFKA-10787 Apply spotless to `metadata` and `server` and `storage` module (#16297)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-16 05:28:50 +08:00
gongxuanzhang d239dde8f6
KAFKA-10787 Apply spotless to raft module (#16278)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-15 11:28:36 +08:00
Omnia Ibrahim e99da2446c
KAFKA-15853: Move KafkaConfig.configDef out of core (#16116)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-06-14 17:26:00 +02:00
gongxuanzhang 6d9ef0e12a
KAFKA-10787 Apply spotless to `group-coordinator` and `group-coordinator-api` (#16298)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-14 12:46:28 +08:00
Ken Huang ea60666513
KAFKA-16921 [1/N] Migrate all junit 4 code to junit 5 for connect module (#16253)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-13 16:11:37 +08:00
gongxuanzhang 46eb0814f6
KAFKA-10787 Apply spotless to log4j-appender, trogdor, jmh-benchmarks, examples, shell and generator (#16296)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-12 22:23:39 +08:00
gongxuanzhang 79b9c44acd
KAFKA-10787 Apply spotless to connect module (#16299)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-12 22:19:47 +08:00
gongxuanzhang af86e56fcd
KAFKA-10787 Apply spotless to tools and tools-api module (#16262)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-11 14:48:14 +08:00
Luke Chen 31c55b630b
add slf4jlog4j in tool (#16260)
In #12148 , we removed log4jAppender dependency, and add testImplementation dependency for slf4jlog4j lib. However, we need this runtime dependency in tools module to output logs. (ref) Adding this dependency back.

Note: The slf4jlog4j lib was added in log4j-appender dependency. Since it's removed, we need to explicitly declare it.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Edoardo Comar <ecomar@uk.ibm.com>
2024-06-11 08:43:35 +08:00
gongxuanzhang 816209d187
KAFKA-10787 Apply spotless to transaction-coordinator and server-common (#16172)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-09 05:36:17 +08:00
PoAn Yang 3d5d1504f7
KAFKA-16878 Remove powermock and easymock from code base (#16236)
Reviewers: TaiJuWu <tjwu1217@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-06-09 00:17:43 +08:00
Ismael Juma e41e54d0a0
MINOR: Remove unnecessary log4j-appender dependency and tweak explicit log4j dependency (#12148)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-06-07 16:41:19 +02:00
Johnny Hsu 271a1b42cd
KAFKA-16223 Migrate KafkaConfigBackingStoreTest to Mockito (#16126)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-07 07:02:14 +08:00
David Jacot 7d832cf74f
KAFKA-14701; Move `PartitionAssignor` to new `group-coordinator-api` module (#16198)
This patch moves the `PartitionAssignor` interface and all the related classes to a newly created `group-coordinator/api` module, following the pattern used by the storage and tools modules.

Reviewers: Ritika Reddy <rreddy@confluent.io>, Jeff Kim <jeff.kim@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
2024-06-06 12:19:20 -07:00
gongxuanzhang 62e5cce184
KAFKA-10787 Update spotless version and remove support JDK8 (#16176)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-05 18:57:32 +08:00
Chris Egerton c6c39c0062
MINOR: Use project path instead of name for spotlessApplyModules in Gradle script (#16177)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-03 15:19:18 -04:00
gongxuanzhang 342e69192f
KAFKA-10787 Add import ordering checkstyle rule and configure an automatic formatter (#16097)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-06-03 04:02:07 +08:00
Greg Harris 9c2b1b8d0b
KAFKA-16809: Run Javadoc in CI (#16025)
Signed-off-by: Greg Harris <greg.harris@aiven.io>
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2024-05-31 10:31:08 -07:00
Murali Basani 701f8e7ad4
KAFKA-16802: Move java versions inside java block to resolve deprecation (#16135)
Reviewers: Greg Harris <greg.harris@aiven.io>
2024-05-30 12:11:35 -07:00
Ken Huang a57c05b73b
KAFKA-16805 Stop using a ClosureBackedAction to configure Spotbugs reports (#16081)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-28 16:42:33 +08:00
KrishVora01 2432a1866e
KAFKA-16373: KIP-1028: Adding code to support Apache Kafka Docker Official Images (#16027)
This PR aims to add JVM based Docker Official Image for Apache Kafka as per the following KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka

This PR adds the following functionalities:
Introduces support for Apache Kafka Docker Official Images via:

GitHub Workflows:

- Workflow to prepare static source files for Docker images
- Workflow to build and test Docker official images
- Scripts to prepare source files and perform Docker image builds and tests

A new directory for Docker official images, named docker/docker_official_images. This is the new directory to house all Docker Official Image assets.

Co-authored-by: Vedarth Sharma <vesharma@confluent.io>

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Vedarth Sharma <vesharma@confluent.io>
2024-05-24 22:21:02 +05:30
TingIāu "Ting" Kì c3018ef55e
KAFKA-16804: Replace archivesBaseName with archivesName (#16016)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Greg Harris <greg.harris@aiven.io>
2024-05-23 10:15:56 -07:00
Greg Harris 7de58f7359
KAFKA-16739: Exclude protected members from aggregated release javadocs (#15940)
Reviewers: Matthias J. Sax <matthias@confluent.io>
2024-05-21 13:54:11 -07:00
Ken Huang 7fea279ff9
KAFKA-16763 Upgrade to scala 2.12.19 and scala 2.13.14 (#15958)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-18 00:36:38 +08:00
David Jacot c1e7aa77de
MINOR: Move `group-coordinator` import control to its own file (#15934)
Follow the pattern used by the other modules.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-14 00:05:32 -07:00
David Jacot 0df340d64d
KAFKA-16470 kafka-dump-log --offsets-decoder should support new records (#15652)
When the consumer group protocol is used in a cluster, it is, at the moment, impossible to see all records stored in the __consumer_offsets topic with kafka-dump-log --offsets-decoder. It does not know how to handle all the new records.

This patch refactors the OffsetsMessageParser used internally by kafka-dump-log to use the RecordSerde used by the new group coordinator. It ensures that the tool is always in sync with the coordinator implementation. The patch also changes the format to using the toString'ed representations of the records instead of having custom logic to dump them. It ensures that all the information is always dumped. The downside of the latest is that inner byte arrays (e.g. assignment in the classic protocol) are no longer deserialized. Personally, I feel like that it is acceptable and it is actually better to stay as close as possible to the actual records in this tool. It also avoids issues like https://issues.apache.org/jira/browse/KAFKA-15603.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-05-07 08:49:31 +08:00
Linu Shibu aeca384641
KAFKA-16356: Remove class-name dispatch in RemoteLogMetadataSerde (#15620)
Reviewers: Greg Harris <greg.harris@aiven.io>, Luke Chen <showuon@gmail.com>, Igor Soarez <soarez@apple.com>, The-Gamer-01 <19974361760@163.com>
2024-05-06 16:49:35 -07:00
Gantigmaa Selenge a9b4b88e54
MINOR: Remove unneccessary version from excluded dependencies of clients (#15804)
Small cleanup: removed version when excluding shaded dependencies from clients library as it's not needed.

Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-04-29 16:58:12 +08:00
Omnia Ibrahim d88c15fc3e
KAFKA-15853 Move KRAFT configs out of KafkaConfig (#15775)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-27 07:02:31 +08:00
Arnout Engelen bed23b7978
MINOR: skip 'zinc' phase from gradle dependency-check plugin (#15054)
This avoids `gradle dependencyCheckAggregate` from reporting on
advisories in build-time dependencies (e.g. CVE-2023-46122) which
typically should not affect us.

I checked that this does not prevent advisories in 'regular'
dependencies from being reported (but there currently are none).

Reviewers: Josep Prat <josep.prat@aiven.io>
2024-04-25 16:10:30 +02:00
Omnia Ibrahim 363f4d2847
KAFKA-15853 Move consumer group and group coordinator configs out of core (#15684)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-17 20:41:22 +08:00
Nikolay be18775af0
MINOR: adds '-parameters' compiler option for :core tests (#15729)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-17 19:06:07 +08:00
Omnia Ibrahim 8c0458861c
KAFKA-15853 Move KafkaConfig Replication properties and docs out of … (#15575)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-16 15:28:35 +08:00
Omnia Ibrahim 61baa7ac6b
KAFKA-15853 Move transactions configs out of core (#15670)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-13 00:29:51 +08:00
PoAn Yang 5c1b819b02
KAFKA-16472 Fix integration tests in Java with parameter name (#15663)
Following test cases don't really run kraft case. The reason is that the test info doesn't contain parameter name, so it always returns false in TestInfoUtils#isKRaft.

1) TopicCommandIntegrationTest
2) DeleteConsumerGroupsTest
3) AuthorizerIntegrationTest
4) DeleteOffsetsConsumerGroupCommandIntegrationTest

We can fix it by adding options.compilerArgs << '-parameters' after

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-04-06 10:43:22 +08:00
Apoorv Mittal ef7f823136
KAFKA-16359: Corrected manifest file for kafka-clients (#15532)
The issue KAFKA-16359 reported inclusion of kafka-clients runtime dependencies in MANIFEST.MF Class-Path.

The root cause is the issue defined here with the usage of shadow plugin.

Looking into the specifics of plugin and documentation, specifies that any dependency marked as shadow will be treated as following by the shadow plugin:

    1. Adds the dependency as runtime dependency in resultant pom.xml - code here
    2. Adds the dependency as Class-Path in MANIFEST.MF as well - code here

Resolution

We do need the runtime dependencies available in the pom (1 above) but not on manifest (2 above). Also there is no clear way to separate the behaviour as both above tasks relies on shadow configuration.

To fix, I have defined another custom configuration named shadowed which is later used to populate the correct pom (the code is similar to what shadow plugin does to populate pom for runtime dependencies).

Though this might seem like a workaround, but I think that's the only way to fix the issue. I have checked other SDKs which suffered with same issue and went with similar route to populate pom.

Reviewers: Luke Chen <showuon@gmail.com>, Reviewers: Mickael Maison <mickael.maison@gmail.com>, Gaurav Narula <gaurav_narula2@apple.com>
2024-04-04 17:56:08 +08:00
Nikolay d8673b26bf
KAFKA-15899 [1/2] Move kafka.security package from core to server module (#15572)
1) This PR moves kafka.security classes from core to server module.
2) AclAuthorizer not moved, because it has heavy dependencies on core classes that not rewrited from scala at the moment.
3) AclAuthorizer will be deleted as part of ZK removal

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-03-30 11:54:22 +08:00
Nikolay 6f38fe5e0a
KAFKA-14588 ZK configuration moved to ZkConfig (#15075)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-03-27 22:37:01 +08:00
Andras Katona eda0dc0eb0
MINOR: Preventing running the :core tests twice when testing with coverage (#15580)
reportScoverage task (which was used previously as dependency of the registered coverage task)
creates a task for each Test task and executes them. There's unitTest, integrationTest and
the test tasks (which is just for executing both unit and integration), so reportScoverage
executes all three with their corresponding scoverage task, hence running all tests twice.
Solution is just to use the reportTestScoverage task as dependency.

Reviewers: Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
2024-03-27 10:51:31 +01:00
Nikolay b6183a4134
KAFKA-14589 ConsumerGroupCommand rewritten in java (#14471)
This PR contains changes to rewrite ConsumerGroupCommand in java and transfer it to tools module

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-03-20 15:34:45 +08:00
Hector Geraldino 178761eb36
KAFKA-14683 Cleanup WorkerSinkTaskTest (#15506)
1) Rename WorkerSinkTaskMockitoTest back to WorkerSinkTaskTest
2) Tidy up the code a bit
3) rewrite "fail" by "assertThrow"

Reviewers: Omnia Ibrahim <o.g.h.ibrahim@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-03-15 03:50:57 +08:00
Andras Katona 2c49a76573
KAFKA-13922: Adjustments for jacoco, coverage reporting (#11982)
Jacoco and scoverage reporting hasn't been working for a while. This commit fixes report generation. After this PR only subproject level reports are generated as Jenkins and Sonar only cares about that.
This PR doesn't change Kafka's Jenkinsfile.

Reviewers: Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
2024-03-14 10:36:46 +01:00
Matthias J. Sax ccf4bd5f46
MINOR: Add 3.7 to Kafka Streams system tests (#15443)
Reviewers: Bruno Cadonna <bruno@confluent.io>
2024-03-06 12:02:58 -08:00
Cheng-Kai, Zhang 8f5bb8cfb3
KAFKA-16252: Fix the documentation and adjust the format (#15473)
Currently, there are few document files generated automatically like the task genConnectMetricsDocs
However, the unwanted log information also added into it.
And the format is not aligned with other which has Mbean located of the third column.

I modified the code logic so the format could follow other section in ops.html
Also close the log since we take everything from the std as a documentation

Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
2024-03-06 17:19:54 +08:00
Nikolay eea369af94
KAFKA-14588 Log cleaner configuration move to CleanerConfig (#15387)
In order to move ConfigCommand to tools we must move all it's dependencies which includes KafkaConfig and other core classes to java. This PR moves log cleaner configuration to CleanerConfig class of storage module.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-03-05 18:11:56 +08:00
Ismael Juma 2c0cab39ae
MINOR: Remove unnecessary easymock/powermock dependencies (#15460)
These projects don't actually use easymock/powermock.

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
2024-03-03 23:31:57 -08:00
Almog Gavra 1c9f360f4a
KAFKA-15215: migrate StreamedJoinTest to Mockito (#15424)
Migrate StreamedJoinTest to Mockito

Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>, 
Divij Vaidya <diviv@amazon.com>
2024-02-26 18:52:25 -08:00
Nikolay 88c5543ccf
KAFKA-14589: [1/3] Tests of ConsoleGroupCommand rewritten in java (#15256)
This PR is part of #14471
Is contains some of ConsoleGroupCommand tests rewritten in java.
Intention of separate PR is to reduce changes and simplify review.

Reviewers: Luke Chen <showuon@gmail.com>
2024-02-13 11:02:36 +08:00
Matthias Berndt adbffcacb4
MINOR: Remove collections-compat dependency for streams module when on Scala 2.13 (#15239)
Reviewers: Divij Vaidya <diviv@amazon.com>, Josep Prat <jlprat@apache.org>

---------

Co-authored-by: Matthias Berndt <matthias.berndt@ttmzero.com>
2024-02-09 11:57:32 +01:00
Apoorv Mittal b35c62ef92
MINOR: Downgrade version of shadow jar plugin (#15308)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Andrew Schofield <aschofield@confluent.io>
2024-02-02 18:32:03 +01:00
Matthias J. Sax 93712eca15
KAFKA-15594: Add version 3.6 to Kafka Streams system tests (#15151)
Reviewers: Walker Carlson <wcarlson@confluent.io>
2024-01-26 14:59:24 -08:00
Omnia Ibrahim 62ce551826
KAFKA-15853: Move KafkaConfig.Defaults to server module (#15158)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Ismael Juma <ismael@juma.me.uk>
, David Jacot <djacot@confluent.io>, Nikolay <NIzhikov@gmail.com>
2024-01-22 15:29:11 +01:00
David Jacot a8203f9c7a
KAFKA-14505; [4/N] Wire transaction verification (#15142)
This patch wires the transaction verification in the new group coordinator. It basically calls the verification path before scheduling the write operation. If the verification fails, the error is returned to the caller.

Note that the patch uses `appendForGroup`. I suppose that we will move away from using it when https://github.com/apache/kafka/pull/15087 is merged.

Reviewers: Justine Olshan <jolshan@confluent.io>
2024-01-11 04:58:57 -08:00
Vedarth Sharma 116762fdce
KAFKA-16016: Add docker wrapper in core and remove docker utility script (#15048)
Migrates functionality provided by utility to Kafka core. This wrapper will be used to generate property files and format storage when invoked from docker container.

Reviewers: Mickael Maison <mickael.maison@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
2024-01-08 18:07:38 +05:30
Apoorv Mittal 8a76c16a02
MINOR: Fix shadow jar publishing for the clients module (#15127)
The PR fixes the publishing of kafka-clients artifact to remote maven. The kafka-clients jar was recently shadowed which would publish the artifacts to the local maven repo successfully but would throw an error when publishing to remote maven. (as part of the release process)

The issue triggers only with publishMavenJavaPublicationToMavenRepository due to signing. Generating signed asc files error out for shadowed release artifacts as the module name (clients) differs from the artifact name (kafka-clients).

The fix is basically to explicitly define artifact of shadowJar to signing and publish plugin. project.shadow.component(mavenJava) previously outputs the name as client-<version>-all.jar though the classifier and archivesBaseName are already defined correctly in :clients and shadowJar construction.
2024-01-08 10:42:49 +01:00
Jason Gustafson 599e22b842
MINOR: Move Raft io thread implementation to Java (#15119)
This patch moves the `RaftIOThread` implementation into Java. I changed the name to `KafkaRaftClientDriver` since the main thing it does is drive the calls to `poll()`. There shouldn't be any changes to the logic.

Reviewers: José Armando García Sancio <jsancio@apache.org>
2024-01-05 09:27:36 -08:00
Fiore Mario Vitale 314de9f23c
KAFKA-15996: Improve JsonConverter performance (#14992)
Improve JsonConverter performance by using afterBurnModule of Jackson library.

Reviewers: Divij Vaidya <diviv@amazon.com>, Mickael Maison <mickael.maison@gmail.com>
2023-12-24 21:47:12 +01:00
Nikolay 45bd19f2ef
KAFKA-14588: Move ConfigType to server-common (#14867)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-12-22 18:35:27 +01:00
Michael Westerby ccb36fe0c7
KAFKA-15358: Added QueuedReplicaToDirAssignments metric (#14942)
Reviewers: Igor Soarez <i@soarez.me>, Ron Dagostino <rdagostino@confluent.io>
2023-12-12 03:17:00 -05:00
Vedarth Sharma 0cd2be8718
KAFKA-15879: Add documentation and examples for docker image (#14938)
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
2023-12-12 10:29:05 +05:30
Colin Patrick McCabe 32fdb8d173
KAFKA-15956: MetadataShell must take the log directory lock when reading (#14899)
MetadataShell should take an advisory lock on the .lock file of the directory it is reading from.
Add an integration test of this functionality in MetadataShellIntegrationTest.java.

Note: in build.gradle, I had to add some dependencies on server-common's test files in order to use
MockFaultHandler, etc.

MetadataBatchLoader.java: fix a case where a log message was incorrect.  The intention was to print
the number equivalent to (offset + index).  Instead it was printing the offset, followed by the
index. So if the offset was 100 and the index was 1, 1001 would be printed rather than 101.

Co-authored-by: Igor Soarez <i@soarez.me>
Reviewers: David Arthur <mumrah@gmail.com>, José Armando García Sancio <jsancio@apache.org>
2023-12-10 19:18:34 -08:00
Matthew de Detrich 964e73178b
KAFKA-14132; Replace EasyMock with Mockito in StandaloneHerderTest (#12728)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-12-07 10:01:17 -05:00
Vedarth Sharma eec1530da0
KAFKA-15445: Add JVM Docker image (#14552)
This PR aims to add Apache Kafka JVM Docker image as per the following KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka

Reviewers:  Ismael Juma <ismael@juma.me.uk>, Ashwin Pankaj <apankaj@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com>, Sanjay Awatramani <sawatramani@confluent.io>, 
Nikita Konev
2023-12-06 15:59:13 +05:30
Nikolay 783698c525
KAFKA-15645: Move ReplicationQuotasTestRig to tools module (#14588)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Justine Olshan <jolshan@confluent.io>, Taras Ledkov <tledkov@apache.org>
2023-12-05 10:03:33 +01:00
Bruno Cadonna 75572f904b
KAFKA-15555: Ensure wakeups are handled correctly in poll() (#14746)
We need to be careful when aborting a long poll with wakeup() since the
consumer might never return records if the poll is interrupted after the
consumer position has been updated but the records have not been returned
to the caller of poll().

This PR avoid wake-ups during this critical period.

Reviewers: Philip Nee <pnee@confluent.io>, Kirk True <ktrue@confluent.io>, Lucas Brutschy <lbrutschy@confluent.io>
2023-11-23 10:53:17 +01:00
Ismael Juma df78204e05
KAFKA-15854: Move Java classes from `kafka.server` to the `server` module (#14796)
We only move Java classes that have minimal or no dependencies on Scala classes in this PR.

Details:
* Configured `server` module in build files.
* Changed `ControllerRequestCompletionHandler` to be an interface since it has no implementations.
* Cleaned up various import control files.
* Minor build clean-ups for `server-common`.
* Disabled `testAssignmentAggregation` when executed with Java 8, this is an existing issue (see #14794).

For broader context on this change, please check:
* KAFKA-15852: Move server code from `core` to `server` module

Reviewers: Divij Vaidya <diviv@amazon.com>
2023-11-19 22:09:19 -08:00
Igor Soarez 0390d5b1a2
KAFKA-15355: Message schema changes (#14290)
Reviewers: Christo Lolov <lolovc@amazon.com>, Colin P. McCabe <cmccabe@apache.org>, Proven Provenzano <pprovenzano@confluent.io>, Ron Dagostino <rdagostino@confluent.io>
2023-11-02 09:46:05 -04:00
Apoorv Mittal b850b46cf9
KAFKA-15668: Adding Opentelmetry shadowed library (KIP-714) (#14618)
Part of KIP-714.

The PR comprises of changes to include opentlemetry library as defined in KIP-714. The libraries are shadowed to prevent conflicts.

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>

Reviewers: Andrew Schofield <aschofield@confluent.io>, Xavier Léauté <xavier@confluent.io>, Walker Carlson <wcarlson@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2023-11-01 21:53:24 -07:00
Apoorv Mittal ad2677bb7b
KAFKA-15614: Define interfaces and classes for client telemetry (#14575)
This PR for KIP-714 - KAFKA-1564 lays out interfaces and classes for capturing client telemetry metrics.

Below image defines interaction of different classes among them interfaces have been included in the PR.

Reviewers: Walker Carlson <wcarlson@apache.org>, Matthias J. Sax <matthias@confluent.io>, Andrew Schofield <andrew_schofield@uk.ibm.com>, Kirk True <ktrue@confluent.io>, Philip Nee <pnee@confluent.io>, Jun Rao <junrao@gmail.com>,
2023-10-26 15:06:38 -05:00
Mickael Maison 8b9f6d17f2
KAFKA-15093: Add 3.5 Streams upgrade system tests (#14602)
Reviewers: Matthias J. Sax <mjsax@apache.org>
2023-10-23 13:26:50 +02:00
shuoer86 27a155c80a
MINOR: Fix typos in build.gradle, tests and trogdor (#14574)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, hudeqi <1217150961@qq.com>
2023-10-23 12:30:57 +02:00
Mickael Maison 9c77c17c4e
KAFKA-15664: Add 3.4 Streams upgrade system tests (#14601)
Reviewers: Luke Chen <showuon@gmail.com>,  Matthias J. Sax <mjsax@apache.org>
2023-10-23 10:33:59 +02:00
Greg Harris ffcb6d4a1a
KAFKA-14767: Fix missing commitId build error after git gc (#13315)
git gc moves commit hashes from individual .git/refs/heads/ to .git/packed-refs which is not read
by the determineCommitId function.

Replace the existing lookup within the .git directory with a GrGit lookup that handles packed and
unpacked refs transparently.

Reviewers: Ismael Juma <ismael@juma.me.uk>
2023-10-22 11:08:01 -07:00
Jeff Kim abee8f711c
KAFKA-14519; [1/N] Implement coordinator runtime metrics (#14417)
Implements the following metrics:

kafka.server:type=group-coordinator-metrics,name=num-partitions,state=loading
kafka.server:type=group-coordinator-metrics,name=num-partitions,state=active
kafka.server:type=group-coordinator-metrics,name=num-partitions,state=failed
kafka.server:type=group-coordinator-metrics,name=event-queue-size
kafka.server:type=group-coordinator-metrics,name=partition-load-time-max
kafka.server:type=group-coordinator-metrics,name=partition-load-time-avg
kafka.server:type=group-coordinator-metrics,name=thread-idle-ratio-min
kafka.server:type=group-coordinator-metrics,name=thread-idle-ratio-avg
The PR makes these metrics generic so that in the future the transaction coordinator runtime can implement the same metrics in a similar fashion.

Also, CoordinatorLoaderImpl#load will now return LoadSummary which encapsulates the start time, end time, number of records/bytes.

Co-authored-by: David Jacot <djacot@confluent.io>

Reviewers:  Ritika Reddy <rreddy@confluent.io>, Calvin Liu <caliu@confluent.io>, David Jacot <djacot@confluent.io>, Justine Olshan <jolshan@confluent.io>
2023-10-17 16:06:23 -07:00
Omnia G.H Ibrahim 9af1e74b5e
KAFKA-14596: Move TopicCommand to tools (#13201)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>
2023-10-17 11:40:15 +02:00
Hector Geraldino 4150595b0a
KAFKA-14684: Replace EasyMock/PowerMock with Mockito in WorkerSinkTaskThreadedTest (#14505)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Christo Lolov <christololov@gmail.com>
2023-10-16 11:24:52 +02:00
Ismael Juma 4cf86c5d2f
KAFKA-15492: Upgrade and enable spotbugs when building with Java 21 (#14533)
Spotbugs was temporarily disabled as part of KAFKA-15485 to support Kafka build with JDK 21. This PR upgrades the spotbugs version to 4.8.0 which adds support for JDK 21 and enables it's usage on build again.

Reviewers: Divij Vaidya <diviv@amazon.com>
2023-10-12 14:09:10 +02:00
Arnout Engelen 1983ebebc7
MINOR: fix dependencycheck warnings (#14476)
Add suppressions and skip benchmarking/testing projects

Reviewers: Josep Prat <josep.prat@aiven.io>
2023-10-11 16:18:19 +02:00
atu-sharm c223a9c376
KAFKA-15521: Refactor build.gradle to align gradle swagger plugin with swagger dependencies (#14473)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-10-09 12:30:46 +02:00
Nikolay e90f82b62a
MINOR: Fix dependency issue in tools module (#14475)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-10-03 18:03:38 +02:00
Omnia G.H Ibrahim 7553d3f562
KAFKA-14593: Move LeaderElectionCommand to tools (#13204)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>
2023-10-03 11:59:56 +02:00
Nikolay 8f8dbad564
KAFKA-14595 ReassignPartitionsIntegrationTest rewritten in java (#14456)
This PR is part of #13247
It contains ReassignPartitionsIntegrationTest rewritten in java.
Goal of PR is reduce changes size in main PR.

Reviewers: Taras Ledkov  <tledkov@apache.org>, Justine Olshan <jolshan@confluent.io>
2023-10-02 13:22:17 -07:00
Arnout Engelen dcd8c7d05f
MINOR: suppress dependencycheck warning for CVE-2023-35116 (#14460)
Like in most projects, this attack cannot be achieved by an external
attacker in Kafka.

Reviewers: Josep Prat <josep.prat@aiven.io>
2023-10-02 18:38:29 +02:00
Mickael Maison 4ba9cfd9d9
MINOR: Fix build failure in genConnectOpenAPIDocs (#14464)
Reviewers: Yash Mayya <yash.mayya@gmail.com>
2023-09-29 13:51:39 +02:00
Ismael Juma 99e6f12dd0
KAFKA-15485: Support Java 21 (3/3) (#14433)
* Update CI to build with Java 21 instead of Java 20
* Disable spotbugs when building with Java 21 as it doesn't support it yet (filed KAFKA-15492 for
   addressing this)
* Disable SslTransportLayerTest.testValidEndpointIdentificationCN with Java 21 (same as Java 20)

Reviewers: Divij Vaidya <diviv@amazon.com>
2023-09-25 05:17:08 -07:00
Ismael Juma d1ad1d7b70
MINOR: Upgrade gradle, plugins and test libraries (#14431)
To prepare Java 21 support, upgrade gradle, its plugins and test libraries.
Release notes for major and minor updates included below.

The highlight is faster Java compilation by not shutting down the daemon at the end of the build.
Gradle's internal performance tests show up to a 30% build time improvement for builds that are
dominated by compiling Java sources.

Mockito turns out to be a complex case where we use one of 3 different versions depending on
the Scala and Java versions used. In addition, the default mocking strategy changed from `subclass`
to `inline` in Mockito 5.0. We now use `inline` across the board (we previously used both `subclass`
and `inline`). See comments in the relevant parts of the code for more details.

* Gradle 8.3 release notes: https://docs.gradle.org/8.3/release-notes.html
* jmh 1.37: virtual thread support and various bug fixes
* JUnit 5.10.0 release notes: https://junit.org/junit5/docs/5.10.0/release-notes/index.html
* Mockito 5.x release notes:
  * https://github.com/mockito/mockito/releases/tag/v5.0.0
  * https://github.com/mockito/mockito/releases/tag/v5.1.0
  * https://github.com/mockito/mockito/releases/tag/v5.2.0
  * https://github.com/mockito/mockito/releases/tag/v5.3.0
  * https://github.com/mockito/mockito/releases/tag/v5.4.0
  * https://github.com/mockito/mockito/releases/tag/v5.5.0
* EasyMock 5.2.0 release notes: https://github.com/easymock/easymock/releases/tag/easymock-5.2.0

Reviewers: Divij Vaidya <diviv@amazon.com>
2023-09-24 06:01:28 -07:00
Tyler Bertrand eea1854479
KAFKA-15476: Resolves cache misses in checkstyle (#14344)
Resolves cache misses in checkstyle tasks due to absolute paths in configProperties.

Sets configDirectory extension property, which is made available by the checkstyle plugin as ${config_loc} in the checkstyle xml files, as shown in the Checkstyle Gradle docs. The absolute paths set in configProperties are then replaced by relative paths from configDirectory. Because the header and suppression config file names are static and only referenced once, these were removed from configProperties and the file names are given directly in checkstyle.xml

Reviewers: Divij Vaidya <diviv@amazon.com>
2023-09-19 10:51:57 +02:00
Yash Mayya 88b554fdbd
KAFKA-15179: Add integration tests for the file sink and source connectors (#14279)
Reviewers: Ashwin Pankaj <apankaj@confluent.io>, Chris Egerton <chrise@aiven.io>
2023-09-07 12:24:13 -04:00
Gantigmaa Selenge 180dcd3969
KAFKA-15294: Publish remote storage configs (#14266)
This change does the following:

1. Make RemoteLogManagerConfigs that are implemented public

2. Add tasks to generate html docs for the configs

3. Include config docs in the main site

Reviewers: Divij Vaidya <diviv@amazon.com>, Luke Chen <showuon@gmail.com>, Christo Lolov <lolovc@amazon.com>, Satish Duggana <satishd@apache.org>
2023-08-28 10:35:11 +02:00
Kamal Chandraprakash 6492164d9c
KAFKA-15167: Tiered Storage Test Harness Framework (#14116)
`TieredStorageTestHarness` is a base class for integration tests exercising the tiered storage functionality. This uses  `LocalTieredStorage` instance as the second-tier storage system and `TopicBasedRemoteLogMetadataManager` as the remote log metadata manager.

Co-authored-by: Alexandre Dupriez <alexandre.dupriez@gmail.com>
Co-authored-by: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>
2023-08-20 17:15:52 +05:30
Greg Harris a9efca0bf6
KAFKA-14759: Move Mock, Schema, and Verifiable connectors to new test-plugins module (#13302)
Reviewers: Hector Geraldino <hgeraldino@gmail.com>, Chris Egerton <chrise@aiven.io>
2023-08-16 10:30:24 -07:00
Greg Harris f5655d31d3
KAFKA-15030: Add connect-plugin-path command-line tool (#14064)
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-11 12:05:51 -07:00
bachmanity1 f137da04fa
KAFKA-14132: Replace Easymock & Powermock with Mockito in KafkaBasedLogTest (#14153)
Reviewers: Yash Mayya <yash.mayya@gmail.com>, Divij Vaidya <diviv@amazon.com>
2023-08-11 10:50:37 +02:00
Federico Valeri 8de3e0436a
KAFKA-15239: Fix system tests using producer performance service (#14092)
Reviewers: Greg Harris <greg.harris@aiven.io>
2023-08-10 14:23:43 -07:00
Yash Mayya f2ebd333e8
KAFKA-13187: Replace EasyMock / PowerMock with Mockito in DistributedHerderTest (#14102)
Reviewers: Chris Egerton <chrise@aiven.io>
2023-08-10 16:47:30 -04:00
Luke Chen 748175ce62
KAFKA-15189: only init remote topic metrics when enabled (#14133)
Only initialize remote topic metrics when system-wise remote storage is enabled to avoid impacting performance for existing brokers. Also add tests.

Reviewers: Divij Vaidya <diviv@amazon.com>, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>
2023-08-05 13:00:16 +08:00
Christo Lolov 7ecf5183dd
KAFKA-14661: Upgrade Zookeeper to 3.8.1 (#13260)
Reviewers: Divij Vaidya <diviv@amazon.com>, Mickael Maison <mickael.maison@gmail.com>
2023-08-01 16:10:39 +02:00
Said Boudjelda 8135b6dfd8
KAFKA-15235: Fix broken coverage reports since migration to Gradle 8.x (#14075)
Reviewers: Divij Vaidya <diviv@amazon.com>
2023-07-26 19:52:02 +02:00
Said Boudjelda e5fb9b61fb
MINOR: upgrade version of gradle plugin (ben-manes.versions) to 0.47.0 (#14098)
Reviewers: Divij Vaidya <diviv@amazon.com>
2023-07-26 19:12:27 +02:00
Greg Harris 125dbb9286
KAFKA-14760: Move ThroughputThrottler from tools to clients, remove tools dependency from connect-runtime (#13313)
Reviewers: Ismael Juma <ismael@juma.me.uk>
2023-07-20 12:58:48 -07:00
Gantigmaa Selenge 5cb2504c49
KAFKA-14669: Use the generated docs for MirrorMaker configs in the doc (#13658)
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-18 11:27:19 -04:00
Abhijeet Kumar fd3b1137d2
KAFKA-14953: Add tiered storage related metrics (#13944)
* KAFKA-14953: Adding RemoteLogManager metrics
In this PR, I have added the following metrics that are related to tiered storage mentioned in[ KIP-405](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage).
|Metric|Description|
|-----------------------------------------|--------------------------------------------------------------|
| RemoteReadRequestsPerSec                    | Number of remote storage read requests per second               |
| RemoteWriteRequestsPerSec                    | Number of remote storage write requests per second              |
| RemoteBytesInPerSec                                | Number of bytes read from remote storage per second           |
| RemoteReadErrorsPerSec                          | Number of remote storage read errors per second                   |
| RemoteBytesOutPerSec                             | Number of bytes copied to remote storage per second            |
| RemoteWriteErrorsPerSec                          | Number of remote storage write errors per second                   |
| RemoteLogReaderTaskQueueSize             | Number of remote storage read tasks pending for execution.  |
| RemoteLogReaderAvgIdlePercent             | Average idle percent of the remote storage reader thread pool|
| RemoteLogManagerTasksAvgIdlePercent | Average idle percent of RemoteLogManager thread pool          |

Added unit tests for all the rate metrics.

Reviewers: Luke Chen <showuon@gmail.com>, Divij Vaidya <diviv@amazon.com>, Kamal Chandraprakash<kamal.chandraprakash@gmail.com>, Jorge Esteban Quilcate Otoya <quilcate.jorge@gmail.com>, Staniel Yao<yaolixinylx@gmail.com>, hudeqi<1217150961@qq.com>, Satish Duggana <satishd@apache.org>
2023-07-18 20:16:19 +05:30
Omnia G H Ibrahim 0c6b1a4e9a
KAFKA-14737: Move kafka.utils.json to server-common (#13585)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>
2023-07-18 11:02:40 +02:00
Clay Johnson 451fff8937
MINOR: Capture build scans on ge.apache.org to benefit from deep build insights (#13676)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Divij Vaidya <diviv@amazon.com>
2023-07-12 10:47:32 +02:00
Satish Duggana 7e2f878713
KAFKA-14522 Rewrite/Move of RemoteIndexCache to storage module. (#13275)
KAFKA-14522 Rewrite and Move of RemoteIndexCache to storage module.
Cleanedup index file suffix usages and other minor cleanups

Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, Divij Vaidya <diviv@amazon.com>, Kamal Chandraprakash<kamal.chandraprakash@gmail.com>, Jorge Esteban Quilcate Otoya <quilcate.jorge@gmail.com>
2023-07-11 23:55:23 +05:30
Hector Geraldino 6368d14a1d
KAFKA-14059 Replace PowerMock with Mockito in WorkerSourceTaskTest (#13383)
Reviewers: Chris Egerton <chrise@aiven.io>
2023-07-10 12:58:54 -04:00
hudeqi 8be601d051
MINOR: Move TROGDOR.md to trogdor module (#13979)
Reviewers: Divij Vaidya <diviv@amazon.com>

---------

Co-authored-by: Deqi Hu <deqi.hu@shopee.com>
2023-07-10 18:11:21 +02:00
Ismael Juma 1f4cbc5d53
MINOR: Add JDK 20 CI build and remove some branch builds (#12948)
It's good for us to add support for Java 20 in preparation for Java 21 - the next LTS.

Given that Scala 2.12 support has been deprecated, a Scala 2.12 variant is not included.

Also remove some branch builds that add load to the CI, but have
low value: JDK 8 & Scala 2.13 (JDK 8 support has been deprecated),
JDK 11 & Scala 2.12 (Scala 2.12 support has been deprecated) and
JDK 17 & Scala 2.12 (Scala 2.12 support has been deprecated).

A newer version of Mockito (4.9.0 -> 4.11.0) is required for Java 20 support, but we
only use it with Scala 2.13+ since it causes compilation errors with Scala 2.12. Similarly,
we upgrade easymock when the Java version is 16 or newer as it's incompatible
with powermock (which doesn't support Java 16 or newer).

Filed KAFKA-15117 for a test that fails with Java 20 (SslTransportLayerTest.testValidEndpointIdentificationCN).

Finally, fixed some lossy conversions that were added after #13582 was submitted.

Reviewers: Ismael Juma <ismael@juma.me.uk>
2023-06-30 01:12:00 -07:00
Divij Vaidya 88e784f7c6
KAFKA-15084: Remove lock contention from RemoteIndexCache (#13850)
Use thread safe Caffeine to cache indexes fetched from RemoteTier locally. This PR removes a lock contention that led to higher fetch latencies as the IO threads spent time unnecessarily waiting on global cache lock while a single thread fetches the index from remote tier. See PR #13850 for details and rejected alternatives.

Reviewers: Luke Chen <showuon@gmail.com>, Satish Duggana <satishd@apache.org>
2023-06-21 18:22:49 +02:00
David Jacot 2aa1555423
MINOR: rat should depend on processMessages task (#13854)
This fix the following issue that we occasionally see in [builds](https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13848/4/pipeline/13/).

```
[2023-06-14T11:41:50.769Z] * What went wrong:
[2023-06-14T11:41:50.769Z] A problem was found with the configuration of task ':rat' (type 'RatTask').
[2023-06-14T11:41:50.769Z]   - Gradle detected a problem with the following location: '/home/jenkins/jenkins-agent/workspace/Kafka_kafka-pr_PR-13848'.
[2023-06-14T11:41:50.769Z]     
[2023-06-14T11:41:50.769Z]     Reason: Task ':rat' uses this output of task ':clients:processTestMessages' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
[2023-06-14T11:41:50.769Z]     
[2023-06-14T11:41:50.769Z]     Possible solutions:
[2023-06-14T11:41:50.769Z]       1. Declare task ':clients:processTestMessages' as an input of ':rat'.
[2023-06-14T11:41:50.769Z]       2. Declare an explicit dependency on ':clients:processTestMessages' from ':rat' using Task#dependsOn.
[2023-06-14T11:41:50.769Z]       3. Declare an explicit dependency on ':clients:processTestMessages' from ':rat' using Task#mustRunAfter.
[2023-06-14T11:41:50.769Z]     
[2023-06-14T11:41:50.769Z]     Please refer to https://docs.gradle.org/8.1.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
```

Validated manually as well:

```
% ./gradlew rat

> Configure project :
Starting build with version 3.6.0-SNAPSHOT (commit id 874081ca) using Gradle 8.1.1, Java 17 and Scala 2.13.10
Build properties: maxParallelForks=10, maxScalacThreads=8, maxTestRetries=0

> Task :storage:processMessages
MessageGenerator: processed 4 Kafka message JSON files(s).

> Task :raft:processMessages
MessageGenerator: processed 1 Kafka message JSON files(s).

> Task :core:processMessages
MessageGenerator: processed 2 Kafka message JSON files(s).

> Task :group-coordinator:processMessages
MessageGenerator: processed 16 Kafka message JSON files(s).

> Task :streams:processMessages
MessageGenerator: processed 1 Kafka message JSON files(s).

> Task :metadata:processMessages
MessageGenerator: processed 20 Kafka message JSON files(s).

> Task :clients:processMessages
MessageGenerator: processed 146 Kafka message JSON files(s).

> Task :clients:processTestMessages
MessageGenerator: processed 4 Kafka message JSON files(s).

BUILD SUCCESSFUL in 8s
```

Reviewers: Divij Vaidya <diviv@amazon.com>
2023-06-16 18:38:43 +02:00
David Jacot 7eea2a3908
MINOR: Move MockTime to server-common (#13823)
This patch rewrite `MockTime` in Java and moves it to `server-common` module. This is a prerequisite to move `MockTimer` later on to `server-common` as well. 

Reviewers: David Arthur <mumrah@gmail.com>
2023-06-09 08:54:25 +02:00
Yash Mayya 9bb2f78d53
KAFKA-15034: Improve performance of the ReplaceField SMT; add JMH benchmark (#13776)
Reviewers: Chris Egerton <chrise@aiven.io>
2023-06-01 15:14:31 -04:00
atu-sharm 87c9fd9834
KAFKA-15015: Explicit on reload4j version (#13745)
Defining a specific version for reload4j

Reviewers: Luke Chen <showuon@gmail.com>
2023-05-24 09:32:59 +08:00
atu-sharm 3598f12242
KAFKA-14839: Exclude protected variable from JavaDocs (#13633)
Reviewers: Manyanda Chitimbo <manyanda.chitimbo@gmail.com>, John Roesler <john@confluent.io>, Matthias J. Sax <matthias@confluent.io>
2023-05-12 18:32:11 -07:00
Federico Valeri c757af5f7c
KAFKA-14752: Kafka examples improvements - demo changes (#13517)
KAFKA-14752: Kafka examples improvements - demo changes

Reviewers: Luke Chen <showuon@gmail.com>
2023-05-12 10:39:12 +08:00
Said Boudjelda a7d0b3f753
MINOR: Upgrade gradle to 8.1.1 (#13625)
Also upgrade gradle plugins:
 - `org.owasp.dependencycheck` gradle plugin to version `8.2.1`
 - `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1`

Gradle release notes:
* https://docs.gradle.org/8.1.1/release-notes.html

Reviewers: Ismael Juma <ismael@juma.me.uk>
2023-04-25 14:31:04 -07:00
Matthew de Detrich 809966a9a0
KAFKA-13299: Accept duplicate listener on port for IPv4/IPv6 (#11478)
Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address.

Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org>
2023-04-19 20:54:07 +02:00
hudeqi b10716e723
KAFKA-14868: Remove all ReplicaManager metrics when it is closed (#13471)
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Divij Vaidya <diviv@amazon.com>
2023-04-19 18:49:08 +02:00
Ron Dagostino e27926f92b
KAFKA-14735: Improve KRaft metadata image change performance at high … (#13280)
topic counts.

Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change.  Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events.  We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods.

Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
2023-04-17 17:52:28 -04:00
Chia-Ping Tsai 637bc92ba1
MINOR: move RecordReader from org.apache.kafka.tools (client module) to org.apache.kafka.tools.api (tools-api module) (#13454)
Reviewers: Jun Rao <junrao@gmail.com>
2023-04-07 00:20:56 +08:00
vamossagar12 c14f56b484
KAFKA-14586: Moving StreamResetter to tools (#13127)
Moves StreamResetter to tools project.

Reviewers: Federico Valeri <fedevaleri@gmail.com>, Christo Lolov <lolovc@amazon.com>, Bruno Cadonna <cadonna@apache.org>
2023-03-28 14:43:22 +02:00
Lucas Brutschy 6fae237638
MINOR: Use JUnit-5 extension to enforce strict stubbing (#13347)
A privious change disabled strict stubbing for the `RocksDBMetricsRecorderTest`. To re-enable the behavior in JUnit-5, we need to pull in a new dependency in the `streams` gradle project.

Reviewers: Guozhang Wang <wangguoz@gmail.com>
2023-03-20 13:49:35 -07:00
Mickael Maison ccfc389a63
KAFKA-14804: Update swagger plugin/dependencies to 2.2.8 (#13388)
Reviewers: David Arthur <mumrah@gmail.com>
2023-03-14 12:06:12 +01:00
David Arthur 4f990dcd40
Revert swagger plugin to 2.2.0 (#13387)
Reviewers: Colin P. McCabe <cmccabe@apache.org>
2023-03-13 13:44:40 -04:00
Chris Egerton 39d41e5aac
KAFKA-14781: Downgrade MM2 log message severity when no ACL authorizer is configured on source broker (#13351)
Reviewers: Mickael Maison <mickael.maison@gmail.com>
2023-03-08 10:25:55 -05:00
David Jacot 788cc11f45
KAFKA-14462; [3/N] Add `onNewMetadataImage` to `GroupCoordinator` interface (#13357)
The new group coordinator needs to access cluster metadata (e.g. topics, partitions, etc.) and it needs a mechanism to be notified when the metadata changes (e.g. to trigger a rebalance). In KRaft clusters, the easiest is to subscribe to metadata changes via the MetadataPublisher.

Reviewers: Justine Olshan <jolshan@confluent.io>
2023-03-08 08:52:01 +01:00