mirror of https://github.com/apache/kafka.git
KAFKA-15443: Upgrade RocksDB to 9.7.3 (#18275)
This PR upgrades RocksDB from 7.9.2 to 9.7.3 and addresses the following compatibility issues introduced by the RocksDB upgrade: - Removal of AccessHint: The AccessHint class was completely removed in RocksDB 9.7.3. This required removing all import statements, variable declarations, method parameters, method return types, and static method calls related to AccessHint in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java Unused methods are removed in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java - Removal of NO_FILE_CLOSES: The NO_FILE_CLOSES metric was also removed in RocksDB 9.7.3. The calculation for numberOfOpenFiles in RocksDBMetricsRecorder.java has been adjusted to now track the total number of file opens since the last reset. The previous calculation, which subtracted NO_FILE_CLOSES from NO_FILE_OPENS, is no longer possible. The reason RocksDB removed NO_FILE_CLOSES seems to be that it did not properly work: https://github.com/search?q=repo%3Afacebook%2Frocksdb+NO_FILE_CLOSES&type=issues - Removal of methods related to compressed block cache configuration in BlockBasedTableConfig - Change of the signature of org.rocksdb.Options.setLogger() Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Matthias J. Sax <matthias@confluent.io>, Bruno Cadonna <cadonna@apache.org>
This commit is contained in:
Swikar Patel
GitHub
parent
6fa5537ceb
commit
8b72204bfd
|
|
@ -257,7 +257,7 @@ metrics-core-4.1.12.1
|
|||
metrics-core-2.2.0
|
||||
opentelemetry-proto-1.0.0-alpha
|
||||
plexus-utils-3.5.1
|
||||
rocksdbjni-7.9.2
|
||||
rocksdbjni-9.7.3
|
||||
scala-library-2.13.15
|
||||
scala-logging_2.13-3.9.5
|
||||
scala-reflect-2.13.15
|
||||
|
|
|
|||
|
|
@ -3475,7 +3475,7 @@ customized state stores; for built-in state stores, currently we have:
|
|||
</tr>
|
||||
<tr>
|
||||
<td>number-open-files</td>
|
||||
<td>The number of current open files.</td>
|
||||
<td>This metric will return constant -1 because the RocksDB's counter NO_FILE_CLOSES has been removed in RocksDB 9.7.3</td>
|
||||
<td>kafka.streams:type=stream-state-metrics,thread-id=([-.\w]+),task-id=([-.\w]+),[store-scope]-id=([-.\w]+)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
|
|
|
|||
|
|
@ -173,6 +173,16 @@
|
|||
See <a href="https://cwiki.apache.org/confluence/x/TZCMEw">KIP-1112</a> for more details.
|
||||
</p>
|
||||
|
||||
<p>
|
||||
Upgraded RocksDB dependency to version 9.7.3 (from 7.9.2). This upgrade incorporates various improvements and optimizations within RocksDB. However, it also introduces some API changes.
|
||||
The <code>org.rocksdb.AccessHint</code> class, along with its associated methods, has been removed.
|
||||
Several methods related to compressed block cache configuration in the <code>BlockBasedTableConfig</code> class have been removed, including <code>blockCacheCompressedNumShardBits</code>, <code>blockCacheCompressedSize</code>, and their corresponding setters. These functionalities are now consolidated under the <code>cache</code> option, and developers should configure their compressed block cache using the <code>setCache</code> method instead.
|
||||
The <code>NO_FILE_CLOSES</code> field has been removed from the <code>org.rocksdb.TickerTypeenum</code> as a result the <code>number-open-files</code> metrics does not work as expected. Metric <code>number-open-files</code> returns constant -1 from now on until it will officially be removed.
|
||||
The <code>org.rocksdb.Options.setLogger()</code> method now accepts a <code>LoggerInterface</code> as a parameter instead of the previous <code>Logger</code>.
|
||||
Some data types used in RocksDB's Java API have been modified. These changes, along with the removed class, field, and new methods, are primarily relevant to users implementing custom RocksDB configurations.
|
||||
These changes are expected to be largely transparent to most Kafka Streams users. However, those employing advanced RocksDB customizations within their Streams applications, particularly through the <code>rocksdb.config.setter</code>, are advised to consult the detailed RocksDB 9.7.3 changelog to ensure a smooth transition and adapt their configurations as needed. Specifically, users leveraging the removed <code>AccessHint</code> class, the removed methods from the <code>BlockBasedTableConfig</code> class, the <code>NO_FILE_CLOSES</code> field from <code>TickerType</code>, or relying on the previous signature of <code>setLogger()</code> will need to update their implementations.
|
||||
</p>
|
||||
|
||||
<h3><a id="streams_api_changes_390" href="#streams_api_changes_390">Streams API changes in 3.9.0</a></h3>
|
||||
|
||||
<p>
|
||||
|
|
|
|||
|
|
@ -116,7 +116,7 @@ versions += [
|
|||
protobuf: "3.25.5", // a dependency of opentelemetryProto
|
||||
pcollections: "4.0.1",
|
||||
re2j: "1.7",
|
||||
rocksDB: "7.9.2",
|
||||
rocksDB: "9.7.3",
|
||||
// When updating the scalafmt version please also update the version field in checkstyle/.scalafmt.conf. scalafmt now
|
||||
// has the version field as mandatory in its configuration, see
|
||||
// https://github.com/scalameta/scalafmt/releases/tag/v3.1.0.
|
||||
|
|
|
|||
|
|
@ -22,7 +22,6 @@ import org.rocksdb.AbstractComparator;
|
|||
import org.rocksdb.AbstractEventListener;
|
||||
import org.rocksdb.AbstractSlice;
|
||||
import org.rocksdb.AbstractWalFilter;
|
||||
import org.rocksdb.AccessHint;
|
||||
import org.rocksdb.BuiltinComparator;
|
||||
import org.rocksdb.Cache;
|
||||
import org.rocksdb.ColumnFamilyOptions;
|
||||
|
|
@ -37,6 +36,7 @@ import org.rocksdb.DBOptions;
|
|||
import org.rocksdb.DbPath;
|
||||
import org.rocksdb.Env;
|
||||
import org.rocksdb.InfoLogLevel;
|
||||
import org.rocksdb.LoggerInterface;
|
||||
import org.rocksdb.MemTableConfig;
|
||||
import org.rocksdb.MergeOperator;
|
||||
import org.rocksdb.Options;
|
||||
|
|
@ -332,14 +332,6 @@ public class RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
|
|||
return dbOptions.statistics();
|
||||
}
|
||||
|
||||
@Deprecated
|
||||
public int baseBackgroundCompactions() {
|
||||
final String message = "This method has been removed from the underlying RocksDB. " +
|
||||
"It is currently a no-op method which returns a default value of -1.";
|
||||
log.warn(message);
|
||||
return -1;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Options setMaxSubcompactions(final int maxSubcompactions) {
|
||||
dbOptions.setMaxSubcompactions(maxSubcompactions);
|
||||
|
|
@ -571,34 +563,6 @@ public class RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
|
|||
return dbOptions.dbWriteBufferSize();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Options setAccessHintOnCompactionStart(final AccessHint accessHint) {
|
||||
dbOptions.setAccessHintOnCompactionStart(accessHint);
|
||||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public AccessHint accessHintOnCompactionStart() {
|
||||
return dbOptions.accessHintOnCompactionStart();
|
||||
}
|
||||
|
||||
@Deprecated
|
||||
public Options setNewTableReaderForCompactionInputs(final boolean newTableReaderForCompactionInputs) {
|
||||
final String message = "This method has been removed from the underlying RocksDB. " +
|
||||
"It was not affecting compaction even in earlier versions. " +
|
||||
"It is currently a no-op method.";
|
||||
log.warn(message);
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated
|
||||
public boolean newTableReaderForCompactionInputs() {
|
||||
final String message = "This method has been removed from the underlying RocksDB. " +
|
||||
"It is now a method which always returns false.";
|
||||
log.warn(message);
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Options setCompactionReadaheadSize(final long compactionReadaheadSize) {
|
||||
dbOptions.setCompactionReadaheadSize(compactionReadaheadSize);
|
||||
|
|
@ -843,7 +807,7 @@ public class RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
|
|||
}
|
||||
|
||||
@Override
|
||||
public Options setLogger(final org.rocksdb.Logger logger) {
|
||||
public Options setLogger(final LoggerInterface logger) {
|
||||
dbOptions.setLogger(logger);
|
||||
return this;
|
||||
}
|
||||
|
|
@ -914,6 +878,16 @@ public class RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
|
|||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Options setMemtableMaxRangeDeletions(final int n) {
|
||||
columnFamilyOptions.setMemtableMaxRangeDeletions(n);
|
||||
return this;
|
||||
}
|
||||
|
||||
@Override
|
||||
public int memtableMaxRangeDeletions() {
|
||||
return columnFamilyOptions.memtableMaxRangeDeletions();
|
||||
}
|
||||
|
||||
@Override
|
||||
public Options setBottommostCompressionType(final CompressionType bottommostCompressionType) {
|
||||
|
|
@ -1464,26 +1438,6 @@ public class RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter extends
|
|||
return dbOptions.allowIngestBehind();
|
||||
}
|
||||
|
||||
@Deprecated
|
||||
public Options setPreserveDeletes(final boolean preserveDeletes) {
|
||||
final String message = "This method has been removed from the underlying RocksDB. " +
|
||||
"It was marked for deprecation in earlier versions. " +
|
||||
"The behaviour can be replicated by using user-defined timestamps. " +
|
||||
"It is currently a no-op method.";
|
||||
log.warn(message);
|
||||
// no-op
|
||||
return this;
|
||||
}
|
||||
|
||||
@Deprecated
|
||||
public boolean preserveDeletes() {
|
||||
final String message = "This method has been removed from the underlying RocksDB. " +
|
||||
"It was marked for deprecation in earlier versions. " +
|
||||
"It is currently a no-op method with a default value of false.";
|
||||
log.warn(message);
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Options setTwoWriteQueues(final boolean twoWriteQueues) {
|
||||
dbOptions.setTwoWriteQueues(twoWriteQueues);
|
||||
|
|
|
|||
|
|
@ -462,8 +462,7 @@ public class RocksDBMetricsRecorder {
|
|||
writeStallDuration += valueProviders.statistics.getAndResetTickerCount(TickerType.STALL_MICROS);
|
||||
bytesWrittenDuringCompaction += valueProviders.statistics.getAndResetTickerCount(TickerType.COMPACT_WRITE_BYTES);
|
||||
bytesReadDuringCompaction += valueProviders.statistics.getAndResetTickerCount(TickerType.COMPACT_READ_BYTES);
|
||||
numberOfOpenFiles += valueProviders.statistics.getAndResetTickerCount(TickerType.NO_FILE_OPENS)
|
||||
- valueProviders.statistics.getAndResetTickerCount(TickerType.NO_FILE_CLOSES);
|
||||
numberOfOpenFiles = -1;
|
||||
numberOfFileErrors += valueProviders.statistics.getAndResetTickerCount(TickerType.NO_FILE_ERRORS);
|
||||
final HistogramData memtableFlushTimeData = valueProviders.statistics.getHistogramData(HistogramType.FLUSH_TIME);
|
||||
memtableFlushTimeSum += memtableFlushTimeData.getSum();
|
||||
|
|
|
|||
|
|
@ -28,7 +28,6 @@ import org.rocksdb.AbstractCompactionFilter;
|
|||
import org.rocksdb.AbstractCompactionFilter.Context;
|
||||
import org.rocksdb.AbstractCompactionFilterFactory;
|
||||
import org.rocksdb.AbstractWalFilter;
|
||||
import org.rocksdb.AccessHint;
|
||||
import org.rocksdb.BuiltinComparator;
|
||||
import org.rocksdb.ColumnFamilyOptions;
|
||||
import org.rocksdb.CompactionPriority;
|
||||
|
|
@ -112,6 +111,8 @@ public class RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest {
|
|||
add("setMaxBackgroundCompactions");
|
||||
add("maxBackgroundFlushes");
|
||||
add("setMaxBackgroundFlushes");
|
||||
add("tablePropertiesCollectorFactory");
|
||||
add("setTablePropertiesCollectorFactory");
|
||||
addAll(walRelatedMethods);
|
||||
}
|
||||
};
|
||||
|
|
@ -176,9 +177,6 @@ public class RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest {
|
|||
case "java.util.Collection":
|
||||
parameters[i] = new ArrayList<>();
|
||||
break;
|
||||
case "org.rocksdb.AccessHint":
|
||||
parameters[i] = AccessHint.NONE;
|
||||
break;
|
||||
case "org.rocksdb.Cache":
|
||||
parameters[i] = new LRUCache(1L);
|
||||
break;
|
||||
|
|
|
|||
|
|
@ -474,10 +474,8 @@ public class RocksDBMetricsRecorderTest {
|
|||
final double expectedCompactionTimeMaxSensor = 24.0;
|
||||
|
||||
when(statisticsToAdd1.getAndResetTickerCount(TickerType.NO_FILE_OPENS)).thenReturn(5L);
|
||||
when(statisticsToAdd1.getAndResetTickerCount(TickerType.NO_FILE_CLOSES)).thenReturn(3L);
|
||||
when(statisticsToAdd2.getAndResetTickerCount(TickerType.NO_FILE_OPENS)).thenReturn(7L);
|
||||
when(statisticsToAdd2.getAndResetTickerCount(TickerType.NO_FILE_CLOSES)).thenReturn(4L);
|
||||
final double expectedNumberOfOpenFilesSensor = (5 + 7) - (3 + 4);
|
||||
final double expectedNumberOfOpenFilesSensor = -1;
|
||||
|
||||
when(statisticsToAdd1.getAndResetTickerCount(TickerType.NO_FILE_ERRORS)).thenReturn(34L);
|
||||
when(statisticsToAdd2.getAndResetTickerCount(TickerType.NO_FILE_ERRORS)).thenReturn(11L);
|
||||
|
|
@ -485,8 +483,8 @@ public class RocksDBMetricsRecorderTest {
|
|||
|
||||
recorder.record(now);
|
||||
|
||||
verify(statisticsToAdd1, times(17)).getAndResetTickerCount(isA(TickerType.class));
|
||||
verify(statisticsToAdd2, times(17)).getAndResetTickerCount(isA(TickerType.class));
|
||||
verify(statisticsToAdd1, times(15)).getAndResetTickerCount(isA(TickerType.class));
|
||||
verify(statisticsToAdd2, times(15)).getAndResetTickerCount(isA(TickerType.class));
|
||||
verify(statisticsToAdd1, times(2)).getHistogramData(isA(HistogramType.class));
|
||||
verify(statisticsToAdd2, times(2)).getHistogramData(isA(HistogramType.class));
|
||||
verify(bytesWrittenToDatabaseSensor).record(expectedBytesWrittenToDatabaseSensor, now);
|
||||
|
|
|
|||
Loading…
Reference in New Issue