- The tool left an empty line behind that we don't need anymore, see
https://github.com/prometheus/prometheus/pull/17092. (Arguably not a
bug in the tool but just our stricter style about empty lines.)
- In tsdb/index/postings_test.go , our (admittedly somewhat
convoluted) code structure tricked the tool so it spit out something
that wouldn't even compile.
- storage/remote/queue_manager_test.go is just a minor formatting
nit.
Signed-off-by: beorn7 <beorn@grafana.com>
See
https://pkg.go.dev/golang.org/x/tools/gopls/internal/analysis/modernize
for details.
This ran into a few issues (arguably bugs in the modernize tool),
which I will fix in the next commit, so that we have transparency what
was done automatically.
Beyond those hiccups, I believe all the changes applied are
legitimate. Even where there might be no tangible direct gain, I would
argue it's still better to use the "modern" way to avoid micro
discussions in tiny style PRs later.
Signed-off-by: beorn7 <beorn@grafana.com>
* Optimise concurrent rule evaluation for rules querying ALERTS and ALERTS_FOR_STATE
Signed-off-by: Marco Pracucci <marco@pracucci.com>
* Further optimised the case of ALERTS and ALERTS_FOR_STATE without alertname label matcher
Signed-off-by: Marco Pracucci <marco@pracucci.com>
---------
Signed-off-by: Marco Pracucci <marco@pracucci.com>
When calculating dependencies between rules, we sometimes run into `{__name__...}` matchers
These can be used the same way as the actual rule names
This will enable even more rules to run concurrently
The new logic is also not slower:
```
julienduchesne@triceratops prometheus % benchstat test-old.txt test.txt
goos: darwin
goarch: arm64
pkg: github.com/prometheus/prometheus/rules
cpu: Apple M3 Pro
│ test-old.txt │ test.txt │
│ sec/op │ sec/op vs base │
DependencyMap-11 1.206µ ± 7% 1.024µ ± 7% -15.10% (p=0.000 n=10)
│ test-old.txt │ test.txt │
│ B/op │ B/op vs base │
DependencyMap-11 1.720Ki ± 0% 1.438Ki ± 0% -16.35% (p=0.000 n=10)
│ test-old.txt │ test.txt │
│ allocs/op │ allocs/op vs base │
DependencyMap-11 39.00 ± 0% 34.00 ± 0% -12.82% (p=0.000 n=10)
```
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* rulefmt: add tests with YAML aliases for Alert/Record/Expr
Altough somewhat discouraged in favour of using proper configuration
management tools to generate full YAML, it can still be useful in some
situations to use YAML anchors/aliases in rules.
The current implementation is however confusing: aliases will work
everywhere except on the alert/record name and expr
This first commit adds (failing) tests to illustrate the issue, the next
one fixes it. The YAML test file is intentionally filled with anchors
and aliases. Although this is probably not representative of a real-world
use case (which would have less of them), it errs on the safer side.
Signed-off-by: François HORTA <fhorta@scaleway.com>
* rulefmt: support YAML aliases for Alert/Record/Expr
This fixes the use of YAML aliases in alert/recording rule names and
expressions. A side effect of this change is that the RuleNode YAML type is
no longer propagated deeper in the codebase, instead the generic Rule type
can now be used.
Signed-off-by: François HORTA <fhorta@scaleway.com>
* rulefmt: Add test for YAML merge combined with aliases
Currently this does work, but adding a test for the related
functionally here makes sense.
Signed-off-by: David Leadbeater <dgl@dgl.cx>
* rulefmt: Rebase to latest changes
Signed-off-by: David Leadbeater <dgl@dgl.cx>
---------
Signed-off-by: François HORTA <fhorta@scaleway.com>
Signed-off-by: David Leadbeater <dgl@dgl.cx>
Co-authored-by: David Leadbeater <dgl@dgl.cx>
* ruler: stop all rule groups asynchronously on shutdown
During shutdown of the rules manager some rule groups have already stopped and are missing evaluations while we're waiting for other groups to finish their evaluation.
When there are many groups (in the thousands), the whole shutdown process can take up to 10 minutes, during which we get miss evaluations.
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
* Use wrappers in stop(); rename awaitStopped()
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
* Add comment
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
---------
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Add --ignore-unknown-fields that ignores unknown fields in rule group
files. There are lots of tools in the ecosystem that "like" to extend
the rule group file structure but they are currently unreadable by
promtool if there's anything extra. The purpose of this flag is so that
we could use the "vanilla" promtool instead of rolling our own.
Some examples of tools/code:
https://github.com/grafana/mimir/blob/main/pkg/mimirtool/rules/rwrulefmt/rulefmt.go8898eb3cc5/pkg/rules/rules.go (L18-L25)
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
* Rules: Store dependencies instead of boolean
To improve https://github.com/prometheus/prometheus/pull/15681 further, we'll need to store the dependencies and dependents of each
Right now, if a rule has both (at least 1) dependents and dependencies, it is not possible to determine the order to run the rules and they must all run sequentially
This PR only changes the dependents and dependencies attributes of rules, it does not implement a new topological sort algorithm
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* Store a slice of Rule instead
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* Add `BenchmarkRuleDependencyController_AnalyseRules` for future reference
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
---------
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* `RuleConcurrencyController`: Add `SplitGroupIntoBatches` method
The concurrency implementation can now return a slice of concurrent rule batches
This allows for additional concurrency as opposed to the current interface which is limited by the order in which the rules have been loaded
Also, I removed the `concurrencyController` attribute from the group. That information is duplicated in the opts.RuleConcurrencyController` attribute, leading to some confusing behavior, especially in tests.
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* Address PR comments
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* Apply suggestions from code review
Co-authored-by: gotjosh <josue.abreu@gmail.com>
Signed-off-by: Julien Duchesne <julienduchesne@live.com>
---------
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
Signed-off-by: Julien Duchesne <julienduchesne@live.com>
Co-authored-by: gotjosh <josue.abreu@gmail.com>
Refactoring in prevision of https://github.com/prometheus/prometheus/pull/15681
By moving the `eval` function outside of the for loop, we can modify the rule execution order more easily (without huge git changes)
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* feat(ruler): Add new `RuleEvaluationTimeSum` field to groups
Coupled with a metric: `rule_group_last_rule_duration_sum_seconds`
This will give us more observability into how fast a group runs with or without concurrency
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* Update rules/group.go
Co-authored-by: gotjosh <josue.abreu@gmail.com>
Signed-off-by: Julien Duchesne <julienduchesne@live.com>
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* Apply suggestions from code review
Co-authored-by: gotjosh <josue.abreu@gmail.com>
Signed-off-by: Julien Duchesne <julienduchesne@live.com>
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
* Remove `in seconds`. A duration is a duration
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
---------
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
Signed-off-by: Julien Duchesne <julienduchesne@live.com>
Co-authored-by: gotjosh <josue.abreu@gmail.com>
The dependency map being empty meant that all the rules were being set as having no dependencies or dependents. Which is the opposite of what we want
Added two new tests that verify the behavior, they failed before the fix, running all the rules concurrently
Signed-off-by: Julien Duchesne <julien.duchesne@grafana.com>
Fix some edge cases when OOO is enabled
Signed-off-by: Vanshikav123 <vanshikav928@gmail.com>
Signed-off-by: Vanshika <102902652+Vanshikav123@users.noreply.github.com>
Signed-off-by: Jesus Vazquez <jesusvzpg@gmail.com>
Co-authored-by: Jesus Vazquez <jesusvzpg@gmail.com>
For: #14355
This commit updates Prometheus to adopt stdlib's log/slog package in
favor of go-kit/log. As part of converting to use slog, several other
related changes are required to get prometheus working, including:
- removed unused logging util func `RateLimit()`
- forward ported the util/logging/Deduper logging by implementing a small custom slog.Handler that does the deduping before chaining log calls to the underlying real slog.Logger
- move some of the json file logging functionality to use prom/common package functionality
- refactored some of the new json file logging for scraping
- changes to promql.QueryLogger interface to swap out logging methods for relevant slog sugar wrappers
- updated lots of tests that used/replicated custom logging functionality, attempting to keep the logical goal of the tests consistent after the transition
- added a healthy amount of `if logger == nil { $makeLogger }` type conditional checks amongst various functions where none were provided -- old code that used the go-kit/log.Logger interface had several places where there were nil references when trying to use functions like `With()` to add keyvals on the new *slog.Logger type
Signed-off-by: TJ Hoplock <t.hoplock@gmail.com>
The error messages we see in the embedded ruler in Mimir right now are contain `alert="unsupported value type"` and `data="unsupported value type"`
```
ts=2024-10-04T09:59:56.900015691Z caller=alerting.go:384 level=warn component=ruler insight=true user=XXX alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_service_mesh_down_alert: template: __alert_service_mesh_down_alert:1:139: executing \"__alert_service_mesh_down_alert\" at <.registry_name>: can't evaluate field registry_name in type struct { Labels map[string]string; ExternalLabels map[string]string; ExternalURL string; Value interface {} }" data="unsupported value type"
```
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
* Rules: Refactor concurrency controller interface
Even though the main purpose of this refactor is to modify the interface of the concurrency controller to accept a Context. I did two drive-by modifications that I think are sensible:
1. I have moved the check for dependencies on rules to the controller itself - this aligns with how the controller should behave as it is a deciding factor on wether we should run concurrently or not.
2. I cleaned up some unused methods from the days of the old interface before #13527 changed it.
Signed-off-by: gotjosh <josue.abreu@gmail.com>
---------
Signed-off-by: gotjosh <josue.abreu@gmail.com>
Arve Knudsen
Bartlomiej Plotka
beorn7
pipiland2612
Marco Pracucci
Julius Hinze
Matthieu MOREL
Bryan Boreham
Anand Rajagopal
tongjicoder
Subhramit Basu
hardlydearly
George Krajcsovits
Julien Duchesne
frazou
Dimitar Dimitrov
Giedrius Statkevičius
Julien Duchesne
Vanshika
TJ Hoplock
Nathan Baulch
riskrole
Max Amin
Seena Fallah
gotjosh