git gc moves commit hashes from individual .git/refs/heads/ to .git/packed-refs which is not read
by the determineCommitId function.
Replace the existing lookup within the .git directory with a GrGit lookup that handles packed and
unpacked refs transparently.
Reviewers: Ismael Juma <ismael@juma.me.uk>
This change does the following:
1. Make RemoteLogManagerConfigs that are implemented public
2. Add tasks to generate html docs for the configs
3. Include config docs in the main site
Reviewers: Divij Vaidya <diviv@amazon.com>, Luke Chen <showuon@gmail.com>, Christo Lolov <lolovc@amazon.com>, Satish Duggana <satishd@apache.org>
`TieredStorageTestHarness` is a base class for integration tests exercising the tiered storage functionality. This uses `LocalTieredStorage` instance as the second-tier storage system and `TopicBasedRemoteLogMetadataManager` as the remote log metadata manager.
Co-authored-by: Alexandre Dupriez <alexandre.dupriez@gmail.com>
Co-authored-by: Kamal Chandraprakash <kamal.chandraprakash@gmail.com>
Only initialize remote topic metrics when system-wise remote storage is enabled to avoid impacting performance for existing brokers. Also add tests.
Reviewers: Divij Vaidya <diviv@amazon.com>, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>
* KAFKA-14953: Adding RemoteLogManager metrics
In this PR, I have added the following metrics that are related to tiered storage mentioned in[ KIP-405](https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage).
|Metric|Description|
|-----------------------------------------|--------------------------------------------------------------|
| RemoteReadRequestsPerSec | Number of remote storage read requests per second |
| RemoteWriteRequestsPerSec | Number of remote storage write requests per second |
| RemoteBytesInPerSec | Number of bytes read from remote storage per second |
| RemoteReadErrorsPerSec | Number of remote storage read errors per second |
| RemoteBytesOutPerSec | Number of bytes copied to remote storage per second |
| RemoteWriteErrorsPerSec | Number of remote storage write errors per second |
| RemoteLogReaderTaskQueueSize | Number of remote storage read tasks pending for execution. |
| RemoteLogReaderAvgIdlePercent | Average idle percent of the remote storage reader thread pool|
| RemoteLogManagerTasksAvgIdlePercent | Average idle percent of RemoteLogManager thread pool |
Added unit tests for all the rate metrics.
Reviewers: Luke Chen <showuon@gmail.com>, Divij Vaidya <diviv@amazon.com>, Kamal Chandraprakash<kamal.chandraprakash@gmail.com>, Jorge Esteban Quilcate Otoya <quilcate.jorge@gmail.com>, Staniel Yao<yaolixinylx@gmail.com>, hudeqi<1217150961@qq.com>, Satish Duggana <satishd@apache.org>
KAFKA-14522 Rewrite and Move of RemoteIndexCache to storage module.
Cleanedup index file suffix usages and other minor cleanups
Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Luke Chen <showuon@gmail.com>, Divij Vaidya <diviv@amazon.com>, Kamal Chandraprakash<kamal.chandraprakash@gmail.com>, Jorge Esteban Quilcate Otoya <quilcate.jorge@gmail.com>
It's good for us to add support for Java 20 in preparation for Java 21 - the next LTS.
Given that Scala 2.12 support has been deprecated, a Scala 2.12 variant is not included.
Also remove some branch builds that add load to the CI, but have
low value: JDK 8 & Scala 2.13 (JDK 8 support has been deprecated),
JDK 11 & Scala 2.12 (Scala 2.12 support has been deprecated) and
JDK 17 & Scala 2.12 (Scala 2.12 support has been deprecated).
A newer version of Mockito (4.9.0 -> 4.11.0) is required for Java 20 support, but we
only use it with Scala 2.13+ since it causes compilation errors with Scala 2.12. Similarly,
we upgrade easymock when the Java version is 16 or newer as it's incompatible
with powermock (which doesn't support Java 16 or newer).
Filed KAFKA-15117 for a test that fails with Java 20 (SslTransportLayerTest.testValidEndpointIdentificationCN).
Finally, fixed some lossy conversions that were added after #13582 was submitted.
Reviewers: Ismael Juma <ismael@juma.me.uk>
Use thread safe Caffeine to cache indexes fetched from RemoteTier locally. This PR removes a lock contention that led to higher fetch latencies as the IO threads spent time unnecessarily waiting on global cache lock while a single thread fetches the index from remote tier. See PR #13850 for details and rejected alternatives.
Reviewers: Luke Chen <showuon@gmail.com>, Satish Duggana <satishd@apache.org>
This fix the following issue that we occasionally see in [builds](https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13848/4/pipeline/13/).
```
[2023-06-14T11:41:50.769Z] * What went wrong:
[2023-06-14T11:41:50.769Z] A problem was found with the configuration of task ':rat' (type 'RatTask').
[2023-06-14T11:41:50.769Z] - Gradle detected a problem with the following location: '/home/jenkins/jenkins-agent/workspace/Kafka_kafka-pr_PR-13848'.
[2023-06-14T11:41:50.769Z]
[2023-06-14T11:41:50.769Z] Reason: Task ':rat' uses this output of task ':clients:processTestMessages' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
[2023-06-14T11:41:50.769Z]
[2023-06-14T11:41:50.769Z] Possible solutions:
[2023-06-14T11:41:50.769Z] 1. Declare task ':clients:processTestMessages' as an input of ':rat'.
[2023-06-14T11:41:50.769Z] 2. Declare an explicit dependency on ':clients:processTestMessages' from ':rat' using Task#dependsOn.
[2023-06-14T11:41:50.769Z] 3. Declare an explicit dependency on ':clients:processTestMessages' from ':rat' using Task#mustRunAfter.
[2023-06-14T11:41:50.769Z]
[2023-06-14T11:41:50.769Z] Please refer to https://docs.gradle.org/8.1.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
```
Validated manually as well:
```
% ./gradlew rat
> Configure project :
Starting build with version 3.6.0-SNAPSHOT (commit id 874081ca) using Gradle 8.1.1, Java 17 and Scala 2.13.10
Build properties: maxParallelForks=10, maxScalacThreads=8, maxTestRetries=0
> Task :storage:processMessages
MessageGenerator: processed 4 Kafka message JSON files(s).
> Task :raft:processMessages
MessageGenerator: processed 1 Kafka message JSON files(s).
> Task :core:processMessages
MessageGenerator: processed 2 Kafka message JSON files(s).
> Task :group-coordinator:processMessages
MessageGenerator: processed 16 Kafka message JSON files(s).
> Task :streams:processMessages
MessageGenerator: processed 1 Kafka message JSON files(s).
> Task :metadata:processMessages
MessageGenerator: processed 20 Kafka message JSON files(s).
> Task :clients:processMessages
MessageGenerator: processed 146 Kafka message JSON files(s).
> Task :clients:processTestMessages
MessageGenerator: processed 4 Kafka message JSON files(s).
BUILD SUCCESSFUL in 8s
```
Reviewers: Divij Vaidya <diviv@amazon.com>
This patch rewrite `MockTime` in Java and moves it to `server-common` module. This is a prerequisite to move `MockTimer` later on to `server-common` as well.
Reviewers: David Arthur <mumrah@gmail.com>
Also upgrade gradle plugins:
- `org.owasp.dependencycheck` gradle plugin to version `8.2.1`
- `com.github.johnrengelman.shadow gradle` plugin to version `8.1.1`
Gradle release notes:
* https://docs.gradle.org/8.1.1/release-notes.html
Reviewers: Ismael Juma <ismael@juma.me.uk>
Loosens the validation so that Kafka can accept duplicate listeners on the same port but if and only if the listeners are valid IP addresses with one address being an IPv4 address and the other being an IPv6 address.
Reviewers: Josep Prat <jlprat@apache.org>, Luke Chen <showuon@apache.org>
topic counts.
Introduces the use of persistent data structures in the KRaft metadata image to avoid copying the entire TopicsImage upon every change. Performance that was O(<number of topics in the cluster>) is now O(<number of topics changing>), which has dramatic time and GC improvements for the most common topic-related metadata events. We abstract away the chosen underlying persistent collection library via ImmutableMap<> and ImmutableSet<> interfaces and static factory methods.
Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>, Purshotam Chauhan <pchauhan@confluent.io>
A privious change disabled strict stubbing for the `RocksDBMetricsRecorderTest`. To re-enable the behavior in JUnit-5, we need to pull in a new dependency in the `streams` gradle project.
Reviewers: Guozhang Wang <wangguoz@gmail.com>
The new group coordinator needs to access cluster metadata (e.g. topics, partitions, etc.) and it needs a mechanism to be notified when the metadata changes (e.g. to trigger a rebalance). In KRaft clusters, the easiest is to subscribe to metadata changes via the MetadataPublisher.
Reviewers: Justine Olshan <jolshan@confluent.io>
This fixes the following `./gradlew install` issue:
```text
* What went wrong:
A problem was found with the configuration of task ':storage:srcJar' (type 'Jar').
- Gradle detected a problem with the following location: '/Users/ijuma/src/kafka/storage/src/generated/java'.
Reason: Task ':storage:srcJar' uses this output of task ':storage:processMessages' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
Possible solutions:
1. Declare task ':storage:processMessages' as an input of ':storage:srcJar'.
2. Declare an explicit dependency on ':storage:processMessages' from ':storage:srcJar' using Task#dependsOn.
3. Declare an explicit dependency on ':storage:processMessages' from ':storage:srcJar' using Task#mustRunAfter.
Please refer to https://docs.gradle.org/8.0.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.
```
Reviewers: David Jacot <david.jacot@gmail.com>
Also re-enable it in CI. We do this by adjusting the `Jenkinsfile`
to use a more general task (`./gradlew check -x test`).
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Dejan Stojadinović <dejan2609@users.noreply.github.com>
Details:
* gradle upgrade: 7.6 -> 8.0.1
* spotbugs plugin upgrade: 5.0.9 -> 5.0.13
* tweaked the mechanics for `-release`/`-source`/`-target` to workaround idiosyncrasies in Gradle 8.0.1 and newer Scala 2.13 versions.
* streams-scala `test` task no longer triggers the `spotless` task since a newer version is required for Gradle 8 support, but the newer version requires Java 11.
Note: relates to #5479
Gradle upgrade highlights:
* "Scala Incremental Compilation for Multi-Module projects broken in 7.x": https://github.com/gradle/gradle/issues/20101
* "Incremental compilation of java modules is broken with Gradle 7.6": https://github.com/gradle/gradle/issues/23067
Full release notes: https://docs.gradle.org/8.0/release-notes.html
Reviewers: Ismael Juma <ismael@juma.me.uk>
Reviewers: Daniel Urban <durban@cloudera.com>, Greg Harris <greg.harris@aiven.io>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>, Mickael Maison <mickael.maison@gmail.com>
Greg Harris
Mickael Maison
Gantigmaa Selenge
Kamal Chandraprakash
bachmanity1
Federico Valeri
Yash Mayya
Luke Chen
Christo Lolov
Said Boudjelda
Abhijeet Kumar
Omnia G H Ibrahim
Clay Johnson
Satish Duggana
Hector Geraldino
hudeqi
Ismael Juma
Divij Vaidya
David Jacot
atu-sharm
Matthew de Detrich
Ron Dagostino
Chia-Ping Tsai
vamossagar12
Lucas Brutschy
David Arthur
Chris Egerton
Dejan Stojadinović