498f63e60b
Merge pull request #17029 from pr00se/wal-checkpoint-dropped-samples
...
TSDB: use timestamps rather than WAL segment numbers to track how long deleted series should be retained in checkpoints
2025-08-20 11:15:10 +01:00
a86d9a3858
Merge pull request #16925 from prometheus/codesome/stale-series-tracking
...
tsdb: Track stale series in the Head block based on stale sample
2025-08-19 15:35:19 -07:00
3904b3cd5f
Restore stale series count from chunk snapshots
...
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2025-08-19 15:07:37 -07:00
b29ce3e489
Restore stale series count on WAL replay
...
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2025-08-19 15:07:37 -07:00
0c3d3d7466
Test the stale series tracking in Head
...
Signed-off-by: Ganesh Vernekar <ganesh.vernekar@reddit.com>
2025-08-19 15:07:37 -07:00
82a4b12507
Add t.parallel() for ./tsdb
...
Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>
2025-08-12 14:12:42 +02:00
676f7665fa
Use testutil.RequireEqual to handle dedupelabels in test
...
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-08-08 14:52:03 -04:00
ead6dc32b9
Fix test
...
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-08-08 14:34:56 -04:00
5cb0192626
Address linter errors
...
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-08-08 14:25:14 -04:00
0fea41ed53
Refactor keep function to work for both agent and non-agent implementations
...
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-08-08 14:12:47 -04:00
6875022873
Update head.walExpiries with record timestamps during WAL replay
...
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-08-08 14:12:47 -04:00
218558f543
Store mint rather than the last WAL segment in head.walExpiries during head GC
...
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-08-08 14:12:41 -04:00
cef219c31c
chore: enable unused-receiver rule from revive
...
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2025-08-04 09:43:33 +00:00
1d79f0f47e
chore(tsdb): add a few more testcases for unlock of unlocked mtx 16332 ( #16848 )
...
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2025-07-09 16:24:46 +02:00
89f011ba13
fix: unlock of unlocked mutex ( #16332 )
...
* fix: unlock on unlocked mutex
Signed-off-by: Usama Alhanaqtah <a.usama@yandex.ru>
* test coverage
Signed-off-by: Usama Alhanaqtah <a.usama@yandex.ru>
---------
Signed-off-by: Usama Alhanaqtah <a.usama@yandex.ru>
Co-authored-by: alhanaqtah.usama <alhanaqtah.usama@DEV-254.local>
2025-07-09 15:37:55 +02:00
b07b552139
[PERF] TSDB: Pass down label value limit into implementation ( #16158 )
...
* allow limiting label values calls
Signed-off-by: Andre Branchizio <andrejbranch@gmail.com>
2025-05-06 18:54:48 +01:00
e7e3ab2824
Fix linting issues found by golangci-lint v2.0.2 ( #16368 )
...
buf.build / lint and publish (push) Waiting to run
Details
CI / Go tests (push) Waiting to run
Details
CI / More Go tests (push) Waiting to run
Details
CI / Go tests with previous Go version (push) Waiting to run
Details
CI / UI tests (push) Waiting to run
Details
CI / Go tests on Windows (push) Waiting to run
Details
CI / Mixins tests (push) Waiting to run
Details
CI / Build Prometheus for common architectures (0) (push) Waiting to run
Details
CI / Build Prometheus for common architectures (1) (push) Waiting to run
Details
CI / Build Prometheus for common architectures (2) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (0) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (1) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (10) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (11) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (2) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (3) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (4) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (5) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (6) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (7) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (8) (push) Waiting to run
Details
CI / Build Prometheus for all architectures (9) (push) Waiting to run
Details
CI / Report status of build Prometheus for all architectures (push) Blocked by required conditions
Details
CI / Check generated parser (push) Waiting to run
Details
CI / golangci-lint (push) Waiting to run
Details
CI / fuzzing (push) Waiting to run
Details
CI / codeql (push) Waiting to run
Details
CI / Publish main branch artifacts (push) Blocked by required conditions
Details
CI / Publish release artefacts (push) Blocked by required conditions
Details
CI / Publish UI on npm Registry (push) Blocked by required conditions
Details
Scorecards supply-chain security / Scorecards analysis (push) Waiting to run
Details
* Fix linting issues found by golangci-lint v2.0.2
---------
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2025-05-03 19:05:13 +02:00
ca416c580c
Merge branch 'main' into slicelabels
...
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2025-05-02 10:31:57 +01:00
8487ed8145
Merge pull request #16440 from bboreham/faster-benchmark-loadwls
...
[TESTS] TSDB: Faster WAL benchmarks
2025-04-22 15:59:03 +01:00
1d4b1d76a5
[TESTS] More efficient label creation in BenchmarkLoadWLs
...
Use the Builder abstraction instead of going via a map.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2025-04-16 18:02:47 +01:00
848df13d3a
[TESTS] Faster WAL Benchmarks by reusing buffer
...
Less garbage collection.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2025-04-16 17:58:09 +01:00
bb76966992
Use stringlabels by default
...
This removes the stringlabels build tag, makes that implementation the default one, and moves the old labels implementation under the slicelabels build tag.
Fixes #16064 .
Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>
2025-04-15 17:52:24 +01:00
e4fe8d8684
Create memSeries with pendingCommit=true
...
This fixes TestHead_RaceBetweenSeriesCreationAndGC.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2025-03-27 11:11:57 +01:00
df33f1aace
Add TestHead_RaceBetweenSeriesCreationAndGC
...
This test consistently fails missing ~10 series.
If it doesn't fail on your machine, just increase totalSeries, that's
how race conditions work.
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
2025-03-27 10:56:24 +01:00
5fa1146e21
chore: enable gci linter ( #16245 )
...
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2025-03-22 15:46:13 +00:00
bc595263c1
Merge pull request #16231 from pr00se/multiref-improvements
...
TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay
2025-03-19 16:15:50 -04:00
f6903bcc22
Let HistogramAppender.appendable return CounterResetHeader instead of… ( #16195 )
...
Let HistogramAppender.appendable return CounterResetHeader instead of boolean
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
---------
Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Signed-off-by: Björn Rabenstein <github@rabenste.in>
Co-authored-by: Björn Rabenstein <github@rabenste.in>
2025-03-18 17:40:27 +01:00
e4e1b515bc
TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay
...
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-03-18 12:22:33 -04:00
37c2ebb5fd
Make out-of-order native histograms flag a no-op and always enable ( #16207 )
...
* Remove experimental out-of-order native histogram flag
This feature has been available in Prometheus since September 2024,
and has no known issues. Therefore proposing to remove the flag
entirely and always have it on. Note that there are still two
settings that need to be configured (out-of-order time window > 0
and native histograms enabled) for this feature to work.
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
* Update CHANGELOG
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
* Keep feature flag with warning
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
* Update CHANGELOG
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
* Update tsdb/head_append.go
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
* Update CHANGELOG.md
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
* Update tsdb/head_append.go
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
* Additional cleanup of comments and test names
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
---------
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
Signed-off-by: Fiona Liao <fiona.y.liao@gmail.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
2025-03-18 10:59:02 +00:00
86eeaf1886
Skip writing series records uniformly across the benchmark, so we skip some OOO series as well
...
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-03-17 15:17:53 -04:00
2147538d1e
Add missing series refs to benchmark
...
Signed-off-by: Patryk Prus <p@trykpr.us>
2025-03-17 15:17:53 -04:00
7a7bc65237
Add util/compression package to consolidate snappy/zstd use in Prometheus. ( #16156 )
...
# Conflicts:
# tsdb/db_test.go
Apply suggestions from code review
tmp
Addressed comments.
Update util/compression/buffers.go
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
2025-03-10 10:36:26 +00:00
61aa82865d
TSDB: keep duplicate series records in checkpoints while their samples may still be present ( #16060 )
...
Renames the head's deleted map to walExpiries, and creates entries for any
duplicate series records encountered during WAL replay, with the expiry set
to the highest current WAL segment number. Any subsequent WAL
checkpoints will see the duplicate series entry in the walExpiries map, and
keep the series record until the last WAL segment that could contain its
samples is deleted.
Other considerations:
WBL: series records aren't written to the WBL, so there are no duplicates to deal with
agent mode: has its own WAL replay logic that handles duplicate series records differently, and is outside the scope of this PR
2025-03-05 13:45:08 -05:00
c7d4b53ec1
chore: enable unused-parameter from revive
...
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
2025-02-19 19:50:28 +01:00
d644324407
feat(tsdb/(head|agent)): reuse pools across segments to avoid generating garbage during WL replay
...
This is part of the "reduce WAL replay overhead/garbage" effort to help with https://github.com/prometheus/prometheus/issues/6934 .
Signed-off-by: machine424 <ayoubmrini424@gmail.com>
2025-02-10 22:40:24 +01:00
6ba25ba93f
tsdb tests: avoid 'defer' till end of function
...
'defer' only runs at the end of the function, so explicitly close the
querier after we finish with it. Also check it didn't error.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2025-01-27 19:59:43 +00:00
1e420ef373
Merge branch 'main' into cedwards/nhcb-wal-wbl
...
# Conflicts:
# tsdb/tsdbutil/histogram.go
2025-01-02 12:50:19 +01:00
cfa32f3d28
TSDB: Move merge of head postings into index
...
This enables it to take advantage of a more compact data structure
since all postings are known to be `*ListPostings`.
Remove the `Get` member which was not used for anything else, and fix up
tests.
Signed-off-by: Bryan Boreham <bjboreham@gmail.com>
2024-12-20 19:22:30 +00:00
39f5a07236
fix TestOOOHeadChunkReader_Chunk on 32-bit
...
Signed-off-by: Joel Beckmeyer <joel@beckmeyer.us>
2024-12-16 10:45:07 -05:00
a046417bc0
Use new record type only for NHCB
2024-12-06 13:46:20 -08:00
6684344026
Rename old histogram record type, use old names for new records
2024-12-05 09:21:47 -08:00
37df50adb9
Attempt for record type
2024-12-05 09:21:47 -08:00
c599d37668
Always return unknown hint for first sample in non-gauge histogram chunk ( #15343 )
...
Always return unknown hint for first sample in non-gauge histogram chunk
---------
Signed-off-by: Fiona Liao <fiona.liao@grafana.com>
Co-authored-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-11-12 15:14:06 +01:00
e6a682f046
Reproduce populateWithDelChunkSeriesIterator corrupting chunk meta
...
When handling recoded histogram chunks the min time of the chunk is
updated by mistake. It should only update when the chunk is completely
new.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-18 10:34:22 +02:00
631fadc4ca
Unit test for data race in head.Appender.AppendHistogram
...
Two Appenders race when creating a series with a native histogram
as the memSeries will be common and the lastHistogram field is written
without lock.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
2024-10-10 14:10:07 +02:00
ab64966e9d
fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()" ( #15094 )
...
* fix: use "ErrorContains" or "EqualError" instead of "Contains(t, err.Error()" and "Equal(t, err.Error()"
---------
Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2024-10-06 16:35:29 +00:00
95a53ef982
Join tests for appending float and histogram CTs
...
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-09-26 11:29:31 -03:00
6bd9b1a7cc
Histogram CT Zero ingestion
...
Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com>
2024-09-26 11:29:22 -03:00
14e3c05ce8
tsdb: Add support for ingestion of out-of-order native histogram samples ( #14546 )
...
Add support for ingesting OOO native histograms
* Add flag for enabling and disabling OOO native histogram ingestion
* Update OOO querying tests to include native histogram samples
* Add OOO head tests
* Add test for OOO native histogram counter reset headers
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored by: Carrie Edwards <edwrdscarrie@gmail.com>
Co-authored by: Jeanette Tan <jeanette.tan@grafana.com>
Co-authored by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Co-authored by: Fiona Liao <fiona.liao@grafana.com>
2024-09-17 11:19:06 +02:00
919dc0cbc6
storage: Update LabelQuerier interface to return sorted label values ( #14849 )
...
* Change LabelQuerier.LabelValues() to return sorted values
---------
Signed-off-by: 🌲 Harry 🌊 John 🏔 <johrry@amazon.com>
2024-09-17 08:55:02 +02:00
Bryan Boreham
Ganesh Vernekar
Ganesh Vernekar
pipiland2612
Patryk Prus
Matthieu MOREL
George Krajcsovits
Banana Duck
Andre Branchizio
Arve Knudsen
Lukasz Mierzwa
Oleg Zaytsev
Ziqi Zhao
Fiona Liao
Bartlomiej Plotka
machine424
György Krajcsovits
Joel Beckmeyer
Carrie Edwards
Arthur Silva Sens
Harry John