263 Commits

Author	SHA1	Message	Date
NamanParlecha	594f9d63a5	refactor(textparse): Introduce Variadic options in textParse.New (#17155) ... * refactor(textparse): introduce ParserOptions struct for cleaner parser initialization Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(fuzz): update fuzzParseMetricWithContentType to use ParserOptions Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): simplify ParserOptions usage in tests and implementations Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parse): using variadic options Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): add fallbackType & SymbolTable to variadic options Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): private fields Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(scrape): compose parser options Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): add comments Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): update to use ParserOptions struct for configuration Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(scrape): remove unused parserOptions field from scrapeLoop Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> * refactor(parser): update ParserOptions field names and add comments for clarity Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com> --------- Signed-off-by: Naman-B-Parlecha <namanparlecha@gmail.com>	2025-09-11 10:49:42 +01:00
George Krajcsovits	31e4d84edd	refactor(textparse): allow for parsers with direct NHCB support (#17153) ... Hide adding NHCB parser on top another parser in New() function so we can easily add direct NHCB capable parsers. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2025-09-06 11:45:44 +02:00
Owen Williams	6ee965c255	common: Update to prom/common v0.66.0, fix TextParser creation (#17139) ... TextParser as of prom/common v0.66.0 requires an explicit validation scheme. Signed-off-by: Owen Williams <owen.williams@grafana.com>	2025-09-03 11:20:04 -04:00
George Krajcsovits	d09db02854	fix(nhcb): flaky test TestConvertClassicHistogramsToNHCB (#17112) ... * fix(nhcb): flaky test TestConvertClassicHistogramsToNHCB The test was e2e, including actually scraping an HTTP endpoint and running the scrape loop. This led to some timing issues. I've simplified it to call the scrape loop append directly. I think that this isn't nice as that is a private interface, but should gets rid of the flakiness and there's already a bunch of test doing this. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2025-09-02 13:37:14 +02:00
Bryan Boreham	70bf09cb2b	Merge pull request #16429 from prymitive/scrapeCacheStaleNaN ... Append staleness markers only for known series	2025-09-02 10:41:07 +01:00
beorn7	747c5ee2b1	Apply analyzer "modernize" to the whole codebase ... See https://pkg.go.dev/golang.org/x/tools/gopls/internal/analysis/modernize for details. This ran into a few issues (arguably bugs in the modernize tool), which I will fix in the next commit, so that we have transparency what was done automatically. Beyond those hiccups, I believe all the changes applied are legitimate. Even where there might be no tangible direct gain, I would argue it's still better to use the "modern" way to avoid micro discussions in tiny style PRs later. Signed-off-by: beorn7 <beorn@grafana.com>	2025-08-27 14:48:41 +02:00
Arve Knudsen	0a40df33fb	Make metric/label name validation scheme explicit (#16928) ... * Parameterize metric/label name validation scheme Parameterized metric/label name validation scheme --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com> Co-authored-by: Julius Hinze <julius.hinze@grafana.com>	2025-08-18 08:09:00 +00:00
pipiland2612	1607b5c8cc	parallelize ./scrape test ... Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>	2025-08-12 14:12:48 +02:00
Matthieu MOREL	cef219c31c	chore: enable unused-receiver rule from revive ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2025-08-04 09:43:33 +00:00
vanshika	0fc5e75504	scraping: Create a span and send the traceparent header during scrape requests (#16425) ... * Traceparent header Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> (cherry picked from commit 44a620dd73) Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> * changes Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> (cherry picked from commit 6e98a77b2d) Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> * adding test Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> (cherry picked from commit 97f288ad87) Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> * more changes Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> (cherry picked from commit d5dd861544) Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> * extract http client creation to newScrapeClient Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> (cherry picked from commit 3cd8092b155df069d02d9409b6327fe60c788bec) Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> * rebase Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> * rebase Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> * reverting Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> * ctx Signed-off-by: Vanshikav123 <vanshikav928@gmail.com> --------- Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>	2025-07-15 09:37:24 +02:00
George Krajcsovits	dae80e66a5	test(scrape): make nhcb scrape test more tolerant (#16862) ... Fixes #16689 well, maybe not 100%, but should improve it. Increase the scrape timeout to be more tolerant of slow test and also use eventually when checking for targets. Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2025-07-11 12:09:53 +02:00
Arve Knudsen	5a5424cbc1	Consolidate around prometheus/common/model.ValidationScheme (#16806) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-07-03 15:37:46 +02:00
Lukasz Mierzwa	c75768739a	Sort series by labels in requireEqual() ... Tests that look at samples with StaleNaN values will fail because these samples are generated from map iteration and so the order can be unstable. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	2025-07-01 14:22:01 +01:00
Lukasz Mierzwa	e2193f634f	Add a test for StaleNaNs after hitting sample_limit ... I was confused why there are no StaleNaN markers appended when a scrape hits sample_limit, but reading the code I see that's expected, so add a test for it. Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	2025-07-01 14:22:01 +01:00
Lukasz Mierzwa	1f7a23cced	Add tests for staleness markers appended to TSDB when sample_limit is set ... Signed-off-by: Lukasz Mierzwa <l.mierzwa@gmail.com>	2025-07-01 14:22:01 +01:00
Carrie Edwards	7289d41940	Add tests for relabeling of type and unit labels (#16743) ... Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>	2025-06-23 16:42:38 +01:00
Bartlomiej Plotka	8e6b008608	feature: type-and-unit-labels (PROM-39 implementation) (#16228) ... * feature: type-and-unit-labels (extended MetricIdentity) Experimental implementation of https://github.com/prometheus/proposals/pull/39 Previous (unmerged) experiments: * https://github.com/prometheus/prometheus/compare/main...dashpole:prometheus:type_and_unit_labels * https://github.com/prometheus/prometheus/pull/16025 Signed-off-by: bwplotka <bwplotka@gmail.com> feature: type-and-unit-labels (extended MetricIdentity) Experimental implementation of https://github.com/prometheus/proposals/pull/39 Previous (unmerged) experiments: * https://github.com/prometheus/prometheus/compare/main...dashpole:prometheus:type_and_unit_labels * https://github.com/prometheus/prometheus/pull/16025 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fix compilation errors Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Lint Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Revert change made to protobuf 'Accept' header Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Fix compilation errors for 'dedupelabels' tag Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> * Rectored into schema.Metadata Signed-off-by: bwplotka <bwplotka@gmail.com> * texparse: Added tests for PromParse Signed-off-by: bwplotka <bwplotka@gmail.com> * add OM tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * add proto tests Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * add schema label tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> * fix tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * add promql tests. Signed-off-by: bwplotka <bwplotka@gmail.com> * lint Signed-off-by: bwplotka <bwplotka@gmail.com> * Addressed comments. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Arthur Silva Sens <arthursens2005@gmail.com> Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>	2025-05-17 09:37:25 +00:00
George Krajcsovits	2726a18ae3	Merge pull request #16452 from chardch/global-always-scrape-classic-hist ... Add global config option for always_scrape_classic_histograms	2025-05-06 09:29:30 +02:00
chardch	a1c157aaef	Add global config option for always_scrape_classic_histograms ... Addresses https://github.com/prometheus/prometheus/issues/16371 This will help with migrating to native histograms with `convert_classic_histograms_to_nhcb` since users may still need to keep the classic histograms during a migration Signed-off-by: chardch <otwordsne@gmail.com>	2025-05-05 10:14:04 -07:00
Arve Knudsen	e7e3ab2824	Fix linting issues found by golangci-lint v2.0.2 (#16368) ... * Fix linting issues found by golangci-lint v2.0.2 --------- Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2025-05-03 19:05:13 +02:00
chardch	2f59d38054	Add global config option for convert_classic_histograms_to_nhcb ... Addresses the global part of https://github.com/prometheus/prometheus/issues/13532 Signed-off-by: chardch <otwordsne@gmail.com>	2025-03-27 10:55:27 -07:00
Owen Williams	6566c5a2b3	scrape: Add config option for escaping scheme request. (#16066) ... The new metric_name_escaping_scheme config option works in parallel with metric_name_validation_scheme and controls which escaping scheme is requested when scraping. When not specified, the scheme will request underscores if the validation scheme is set to legacy, and will request allow-utf-8 when the validation scheme is set to utf8. This setting allows users to allow utf8 names if they like, but explicitly request an escaping scheme rather than UTF-8. Fixes https://github.com/prometheus/prometheus/issues/16034 Built on https://github.com/prometheus/prometheus/pull/16080 Signed-off-by: Owen Williams <owen.williams@grafana.com>	2025-03-26 18:27:28 -04:00
György Krajcsovits	fec2f2603a	fix(scrape): skip native histogram series if ingestion is disabled ... No need to validate, track, add sample, add exemplar if series isn't going to be ingested. Basically this is the same as if this series was dropped by relabelling. Fixes #16217 Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>	2025-03-17 14:48:09 +01:00
Owen Williams	94b43c5d4c	utf8: Remove support for legacy global validation setting ... Global and Data Source configurations can specify legacy mode, but Prometheus now requires that the overall validation mode be set to UTF-8 Signed-off-by: Owen Williams <owen.williams@grafana.com>	2025-03-13 10:47:24 -04:00
Bryan Boreham	e32d89af7f	[BUGFIX] Scraping: bump cache iteration after error (#16174) ... We use the cache iteration number to detect when the same metric has occurred twice in a scrape. We need to bump this number at the end of every scrape, not just successful scrapes. The `iter` number is also used: * After a successful scrape, to delete older metrics and metadata. * To detect when metadata changed in this scrape. None of those additional cases is broken by incrementing the number on error. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2025-03-08 18:45:37 +00:00
Björn Rabenstein	7bbbb5cb97	Merge pull request #16006 from mmorel-35/revive/unused-parameter ... chore: enable unused-parameter from revive	2025-02-21 13:56:04 +01:00
Matt Hughes	5868e36d91	scraper: fix UTF-8 scraping header always sent with PrometheusText1.0.0 ... The `Accept` header should not include `escape=allow-utf-8` unless explicitly requested. Conveniently, there was already a test covering this header's value, it just required updating so it also asserts that this value in the header is not set in the cases we don't expect it to be set. I also converted those tests into table tests to help make failures clearer. Issue: https://github.com/prometheus/prometheus/issues/15857 Signed-off-by: Matt Hughes <mhughes@uw.co.uk>	2025-02-20 11:33:16 +00:00
Matthieu MOREL	c7d4b53ec1	chore: enable unused-parameter from revive ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2025-02-19 19:50:28 +01:00
Bartlomiej Plotka	733a5e9eb4	textparse: Optimized protobuf parser with custom streaming unmarshal. (#15731) ... * textparse: Optimized protobuf parser with custom streaming decoder. Signed-off-by: bwplotka <bwplotka@gmail.com> Update model/textparse/protobufparse.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Addressing comments. Signed-off-by: bwplotka <bwplotka@gmail.com> decoder: reuse histograms and summaries. Signed-off-by: bwplotka <bwplotka@gmail.com> optimize help returning (5% of mem utilization). Signed-off-by: bwplotka <bwplotka@gmail.com> Apply suggestions from code review Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Update prompb/io/prometheus/client/decoder.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Fix build. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update model/textparse/protobufparse.go Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>	2025-02-13 10:38:35 +00:00
Bartlomiej Plotka	00b69efabb	model/textparse: Change parser interface Metric(...) string to Labels(...) (#16012) ... * model/textparse: Change parser interface Metric(...) string to Labels(...) Simplified the interface given no one is using the return argument. Renamed for clarity too. Found and discussed https://github.com/prometheus/prometheus/pull/15731#discussion_r1950916842 Signed-off-by: bwplotka <bwplotka@gmail.com> * Fixed comments; optimized not needed copy for om and text. Signed-off-by: bwplotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-02-12 15:47:56 +00:00
Bartlomiej Plotka	8cd9069cf1	textparse: Refactor benchmark testdata for all types. (#15998) ... Also: * split benchmark functions to make sure no one compares across parsers. * testdata file have meaningful names reflecting the type representation * promtestdata.txt now has all types, taken directly from long running Prometheus (https://demo.do.prometheus.io/) Needed for https://github.com/prometheus/prometheus/pull/15731 Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-02-10 12:26:18 +00:00
Bartlomiej Plotka	7427753922	scrape: Add realistic data case for scrape loop append bench. (#15966) ... * scrape: Add realistic data case for scrape loop append bench. Signed-off-by: bwplotka <bwplotka@gmail.com> * Update scrape/scrape_test.go Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> --------- Signed-off-by: bwplotka <bwplotka@gmail.com> Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com> Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>	2025-02-03 14:46:39 +00:00
Jan Fajerski	54cf0d6879	Merge pull request #15472 from tjhop/ref/jsonfilelogger-slog-handler ... ref: JSONFileLogger slog handler, add scrape.FailureLogger interface	2025-01-27 20:17:46 +01:00
bwplotka	3119567d5b	scrape: Add metadata for automatic metrics. ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-01-16 19:49:12 +00:00
bwplotka	9385f31147	scrape: Fix metadata in WAL not working for histograms and summaries. ... The was a bug (due to confusion?) on the local metadata cache that is cached by metric family not the series metric name. The fix is to NOT use that local cache at all (it's still needed for current metadata API implementation, added TODO on how we can get rid of it). I went ahead and also rename Metric field in metadata structs to MetricFamily to make clear it's not always __name__. Signed-off-by: bwplotka <bwplotka@gmail.com>	2025-01-15 20:12:38 +00:00
bwplotka	281306765e	scrape: Unified scrape loop benchmark. ... Signed-off-by: bwplotka <bwplotka@gmail.com>	2024-12-29 15:19:06 +00:00
Bryan Boreham	b4ef38cfc8	Scraping: Add benchmark for protobuf format ... Extract helper function textToProto(). Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-12-29 15:13:38 +00:00
Bryan Boreham	8f4557b0b1	Scraping benchmark: more realistic test ... Don't repeat type and help text. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-12-29 15:11:39 +00:00
Bryan Boreham	7b03796d0f	Scraping: stop storing discovered labels (#15261) ... Instead of storing discovered labels on every target, recompute them if required. The `Target` struct now needs to hold some more data required to recompute them, such as ScrapeConfig. This moves the load from every Prometheus all of the time, to just when someone views Service Discovery in the UI. The way `PopulateLabels` is used changes; you are no longer expected to call it with a part-populated `labels.Builder`. The signature of `Target.Labels` changes to take a `labels.Builder` instead of a `ScratchBuilder`, for consistency with `DiscoveredLabels`. This will save a lot of work when many targets are filtered out in relabeling. Combine with `keep_dropped_targets` to avoid ever computing most labels for dropped targets. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-12-21 13:33:08 +00:00
Bryan Boreham	c69aeabf1d	Merge pull request #15563 from GiedriusS/fix_setting_field ... scrape: fix nil panic after scrape loop reload	2024-12-10 15:33:56 +00:00
Giedrius Statkevičius	5d76510bd6	scrape: update test name ... Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	2024-12-09 17:36:36 +02:00
Giedrius Statkevičius	5479fb3cfd	scrape: update test after review ... Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	2024-12-09 08:18:06 +02:00
Bryan Boreham	48287b15d4	Merge pull request #13914 from bboreham/scrape-reload-metric ... [REFACTOR] Scraping: rename variables for clarity	2024-12-02 17:09:25 +00:00
Giedrius Statkevičius	90e832c861	scrape: fix nil panic after scrape loop reload ... Don't forget to set `metrics` field as otherwise scraping will lead to a nil panic in case the body size limit is reached. Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>	2024-12-02 16:08:37 +02:00
TJ Hoplock	e0104a6b7e	ref: JSONFileLogger slog handler, add scrape.FailureLogger interface ... Improves upon #15434, better resolves #15433. This commit introduces a few changes to ensure safer handling of the JSONFileLogger: - the JSONFileLogger struct now implements the slog.Handler interface, so it can directly be used to create slog Loggers. This pattern more closely aligns with upstream slog usage (which is generally based around handlers), as well as making it clear that devs are creating a whole new logger when interacting with it (vs silently modifying internal configs like it did previously). - updates the `promql.QueryLogger` interface to be a union of the methods of both the `io.Closer` interface and the `slog.Handler` interface. This allows for plugging in/using slog-compatible loggers other than the JSONFileLogger, if desired (ie, for downstream projects). - introduces new `scrape.FailureLogger` interface; just like `promql.QueryLogger`, it is a union of `io.Closer` and `slog.Handler` interfaces. Similar logic applies to reasoning. - updates tests where needed; have the `FakeQueryLogger` from promql's engine_test implement the `slog.Handler`, improve JSONFileLogger test suite, etc. Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>	2024-11-28 23:14:31 -05:00
Arve Knudsen	89bbb885e5	Upgrade to golangci-lint v1.62.0 (#15424) ... Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>	2024-11-20 17:22:20 +01:00
Bryan Boreham	26886d6d95	Merge branch 'release-3.0' into merge-release-into-main	2024-11-18 14:43:56 +00:00
Bryan Boreham	3f6a133f26	[Test] Scraping: check reload metrics ... Need to extend `newTestScrapeMetrics`` to get at the Registry. `gatherLabels` function could go upstream to `client_golang`. Signed-off-by: Bryan Boreham <bjboreham@gmail.com>	2024-11-07 17:29:04 +00:00
alexgreenbank	6b09f094e1	scrape: stop erroring on empty scrapes ... Signed-off-by: alexgreenbank <alex.greenbank@grafana.com>	2024-11-07 11:30:03 +00:00
Matthieu MOREL	af1a19fc78	enable errorf rule from perfsprint linter ... Signed-off-by: Matthieu MOREL <matthieu.morel35@gmail.com>	2024-11-06 16:50:36 +01:00