Commit Graph

24 Commits

Author SHA1 Message Date
Peter Štibraný 3c6768460c
Remove unused index_max_batch_size option to bleve backend. (#112171)
* Remove unused index_max_batch_size option to bleve backend.

* Remove batchSize argument.
2025-10-08 17:04:38 +02:00
Peter Štibraný a44af81082
Unified storage search: Introduce min index update interval (#111978)
* Don't update index more often than specified index_min_update_interval.

* Add artificial sleep at the end of write operations.

* Improve test: check for number of update calls, make diff check less flaky.

* Make test less flaky by allowing for higher diff variance.

* Make test less flaky by allowing for expected update calls variance.
2025-10-06 10:02:03 +02:00
Peter Štibraný 707c486a46
Rebuild search indexes asynchronously (#111829)
* Add "debouncer" queue, which can combine incoming elements.

* Rebuild indexes asynchronously.

* Remove duplicate method.

* Fix bleve tests.

* Extracted combineRebuildRequests and added test for it.

* Add TestShouldRebuildIndex

* Added TestFindIndexesForRebuild

* Added TestFindIndexesForRebuild

* Introduce index_rebuild_workers option.

* Add metric for rebuild queue length.

* Add TestRebuildIndexes.

* Fix import.

* Linter, review feedback.
2025-10-01 11:52:09 +02:00
Peter Štibraný 48250a1605
Introduced options to ignore old indexes based on build time and build version. (#111168) 2025-09-17 10:59:59 +02:00
Peter Štibraný 6fa6a5708a
Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Will Assis ac641e07fe
unified-storage: increase sprinkles default limit from 100 to 10000 (#110135)
increase sprinkles default limit from 100 to 10000
2025-08-26 10:59:15 +02:00
Will Assis 34f1be796d
unified-storage: expose ring replication factor config (#106345)
* config ring replication factor

* change default

* rename

* fix test

* fix
2025-07-22 09:32:22 +02:00
Mustafa Sencer Özcan 974a2c47f9
feat(unified-storage): add qos support for the resource server (#105939)
Actionlint / Lint GitHub Actions files (push) Waiting to run Details
Backend Code Checks / Validate Backend Configs (push) Waiting to run Details
Backend Unit Tests / Detect whether code changed (push) Waiting to run Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (1/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (2/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (3/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (4/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (5/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (6/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (7/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (8/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (1/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (2/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (3/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (4/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (5/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (6/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (7/8) (push) Blocked by required conditions Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (8/8) (push) Blocked by required conditions Details
Backend Unit Tests / All backend unit tests complete (push) Blocked by required conditions Details
CodeQL checks / Analyze (actions) (push) Waiting to run Details
CodeQL checks / Analyze (go) (push) Waiting to run Details
CodeQL checks / Analyze (javascript) (push) Waiting to run Details
Lint Frontend / Detect whether code changed (push) Waiting to run Details
Lint Frontend / Lint (push) Blocked by required conditions Details
Lint Frontend / Typecheck (push) Blocked by required conditions Details
Lint Frontend / Betterer (push) Blocked by required conditions Details
golangci-lint / lint-go (push) Waiting to run Details
Verify i18n / verify-i18n (push) Waiting to run Details
Documentation / Build & Verify Docs (push) Waiting to run Details
End-to-end tests / Detect whether code changed (push) Waiting to run Details
End-to-end tests / Build & Package Grafana (push) Blocked by required conditions Details
End-to-end tests / Build E2E test runner (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (--flags="--env dashboardScene=false", e2e/old-arch/dashboards-suite, dashboards-suite (old arch)) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (--flags="--env dashboardScene=false", e2e/old-arch/panels-suite, panels-suite (old arch)) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (--flags="--env dashboardScene=false", e2e/old-arch/smoke-tests-suite, smoke-tests-suite (old arch)) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (--flags="--env dashboardScene=false", e2e/old-arch/various-suite, various-suite (old arch)) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (e2e/dashboards-suite, dashboards-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (e2e/panels-suite, panels-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (e2e/smoke-tests-suite, smoke-tests-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (e2e/various-suite, various-suite) (push) Blocked by required conditions Details
End-to-end tests / A11y test (push) Blocked by required conditions Details
End-to-end tests / All E2E tests complete (push) Blocked by required conditions Details
Frontend tests / Detect whether code changed (push) Waiting to run Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (1) (push) Blocked by required conditions Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (2) (push) Blocked by required conditions Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (3) (push) Blocked by required conditions Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (4) (push) Blocked by required conditions Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (5) (push) Blocked by required conditions Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (6) (push) Blocked by required conditions Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (7) (push) Blocked by required conditions Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (8) (push) Blocked by required conditions Details
Frontend tests / All frontend unit tests complete (push) Blocked by required conditions Details
Integration Tests / Sqlite (${{ matrix.shard }}) (1/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (2/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (3/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (4/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (5/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (6/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (7/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (8/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (1/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (2/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (3/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (4/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (5/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (6/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (7/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (8/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (1/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (2/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (3/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (4/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (5/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (6/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (7/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (8/8) (push) Waiting to run Details
Integration Tests / All backend integration tests complete (push) Blocked by required conditions Details
publish-technical-documentation-next / sync (push) Waiting to run Details
Reject GitHub secrets / reject-gh-secrets (push) Waiting to run Details
Build Release Packages / setup (push) Waiting to run Details
Build Release Packages / Dispatch grafana-enterprise build (push) Blocked by required conditions Details
Build Release Packages / ${{ needs.setup.outputs.version }} / ${{ matrix.name }} (targz:grafana:darwin/amd64, darwin-amd64) (push) Blocked by required conditions Details
Build Release Packages / ${{ needs.setup.outputs.version }} / ${{ matrix.name }} (targz:grafana:darwin/arm64, darwin-arm64) (push) Blocked by required conditions Details
Build Release Packages / ${{ needs.setup.outputs.version }} / ${{ matrix.name }} (targz:grafana:linux/amd64,deb:grafana:linux/amd64,rpm:grafana:linux/amd64,docker:grafana:linux/amd64,docker:grafana:linux/amd64:ubuntu,npm:grafana,storybook, linux-amd64) (push) Blocked by required conditions Details
Build Release Packages / ${{ needs.setup.outputs.version }} / ${{ matrix.name }} (targz:grafana:linux/arm/v6,deb:grafana:linux/arm/v6, linux-armv6) (push) Blocked by required conditions Details
Build Release Packages / ${{ needs.setup.outputs.version }} / ${{ matrix.name }} (targz:grafana:linux/arm/v7,deb:grafana:linux/arm/v7,docker:grafana:linux/arm/v7,docker:grafana:linux/arm/v7:ubuntu, linux-armv7) (push) Blocked by required conditions Details
Build Release Packages / ${{ needs.setup.outputs.version }} / ${{ matrix.name }} (targz:grafana:linux/arm64,deb:grafana:linux/arm64,rpm:grafana:linux/arm64,docker:grafana:linux/arm64,docker:grafana:linux/arm64:ubuntu, linux-arm64) (push) Blocked by required conditions Details
Build Release Packages / ${{ needs.setup.outputs.version }} / ${{ matrix.name }} (targz:grafana:linux/s390x,deb:grafana:linux/s390x,rpm:grafana:linux/s390x,docker:grafana:linux/s390x,docker:grafana:linux/s390x:ubuntu, linux-s390x) (push) Blocked by required conditions Details
Build Release Packages / ${{ needs.setup.outputs.version }} / ${{ matrix.name }} (targz:grafana:windows/amd64,zip:grafana:windows/amd64,msi:grafana:windows/amd64, windows-amd64) (push) Blocked by required conditions Details
Build Release Packages / ${{ needs.setup.outputs.version }} / ${{ matrix.name }} (targz:grafana:windows/arm64,zip:grafana:windows/arm64, windows-arm64) (push) Blocked by required conditions Details
Build Release Packages / Upload artifacts (push) Blocked by required conditions Details
Run dashboard schema v2 e2e / dashboard-schema-v2-e2e (push) Waiting to run Details
Shellcheck / Shellcheck scripts (push) Waiting to run Details
Verify Storybook / Verify Storybook (push) Waiting to run Details
Swagger generated code / Verify committed API specs match (push) Waiting to run Details
Dispatch sync to mirror / dispatch-job (push) Waiting to run Details
trigger-dashboard-search-e2e / trigger-search-e2e (push) Waiting to run Details
publish-kinds-next / main (push) Has been cancelled Details
2025-07-01 09:22:55 +00:00
Will Assis 7aad041f70
unified-storage: Add integration tests for distributor (#105771)
* Add integration tests for distributor
2025-06-30 09:31:58 -04:00
Arati R. 0982cfd9a0
Unified Storage/Search: Add max count config for indexing (#107255)
* Add max count config for indexing
* Build empty index when max count is exceeded
* Address linting
* Refactor buildIndexes
* Add test for max count threshold
* Update test doc comments
* Refactor TestBuildIndexes_MaxCountThreshold to not use mock framework
* Rename mocks used in TestBuildIndexes_MaxCountThreshold

* Refactor mockResourceIndex

* Test setting of indexing threshold configs

* Tweak comments, log

* Fix logging in buildEmptyIndex

* Export and reuse TestDocumentBuilderSupplier

* Reuse MockResourceIndex
2025-06-27 14:00:39 +02:00
Mustafa Sencer Özcan b48337a7c8
feat: introduce ttl cache for bleve indices (#106842) 2025-06-17 16:28:51 +02:00
Jean-Philippe Quéméner 5f21f320f7
fix(unified-storage): use continue token containing both formats for dualwriter (#106525) 2025-06-13 15:59:46 +02:00
Stephanie Hingtgen 5135d5c87d
Unified storage: Reconstruct index in the background every 24h (#106422) 2025-06-12 14:34:48 -05:00
Will Assis ecea6a9366
unified-storage: expose memberlist cluster label config (#104643)
* tweak ringKey/ringName for consistency and allow us to configure memberlist cluster label config

* allow setting memberlist cluster label verification disabled field

---------

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
2025-04-30 09:43:09 -04:00
Will Assis 4adebd6058
unified-storage: setup ring to shard requests (#103783)
* Updates the instrumentation_server service to use mux instead of the builtin router, and have it store the router in the module server: this is so we can register the /ring endpoint to check the status of the ring
* Create a new Ring service that depends on the instrumentation server and declares it as a dependency for the storage server
* Create standalone MemberlistKV service for Ring service to use
* Update the storage server Search and GetStats handler to distribute requests if applicable
2025-04-25 13:08:44 -04:00
Mustafa Sencer Özcan ebf3a8d205
fix: make data syncer run configurable on mode (2->3) change (#104401) 2025-04-25 13:43:35 +02:00
owensmallwood e6e5d23259
Unified Storage: Adds settings for ca cert path and skipping cert verification (#98412)
Adds settings to US for ca cert path and skipping cert verification
2025-01-06 14:00:38 -06:00
owensmallwood 4837585cab
Unified Storage Search: Add sprinkles (#97820)
* Wire up sprinkles to oss and enterprise. Fetching sprinkles not implemented yet.

* Adds wireset for initializing document builders. Had to init it when creating the service to avoid cyclical imports.

* updates to int64 for stats

* adds config for sprinklesApiServer and gets sprinkles from there when its present

* add comment for later

* adds feature toggle for sprinkles. returns empty results when flag not enabled.

* adds unified storage config setting for sprinkles apiserver page limit

* fixes bug where dashboard uid was not getting set

* when creating dashboard summary, use metadata.name as the dashboard uid

* cleans up wire. use existing oss and enterprise sets to generate doc builders

* remove old wireset

* fix linter - adds missing arg for doc builders

* update dashboard stats in tests

* updates test-data dashboards

* log a warning instead of returning an error if we can't get sprinkles for a namespace

* dont read uid from dashboard json
2024-12-18 11:00:54 -06:00
Jean-Philippe Quéméner 1509fadd5b
refactor(unified-storage): make datasyncer configurable (#97832) 2024-12-12 21:41:01 +01:00
owensmallwood 70c9c3889f
Unified Storage Indexer: Add back metrics (#97310)
* Adds back indexer metrics. Uses config values instead of hardcoded ones.

* cast to int64

* remove unused func

* Index metrics impl doesn't depend on Bleve. Adds a TotalDocs func to SearchBackend interface.

* adds config setting for index_min_count

* rename arg

* rename metric label to namespace instead of slug

* adds default "do nothing" case to satisfy linter

* moves bleve index metrics to search package

* make bleve backend private, dont need to pass in prom reg

* imports

* adds bleve metrics to resource package to avoid circular deps
2024-12-04 15:02:40 -06:00
owensmallwood 995128d1db
Search PoC: Improves initial indexing speed. Makes params configurable. (#95439)
* Improves initial indexing speed. Makes params configurable.

* fix linter errors

* removes kind param

* updates index test

* remove println from test

* removes error check in test

* adds log for high index latency ands updates max goroutine var with workers config var

* fix test timing out - set worker limit

* set the batch size

---------

Co-authored-by: Scott Lepper <scott.lepper@gmail.com>
2024-10-29 12:24:31 -06:00
owensmallwood 9763199b53
Search Poc: Add metrics (#95111) 2024-10-22 12:25:08 -06:00
maicon 377079d8b3
UniStore: Add config to enable periodic DualWriter DataSyncer (#93555)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2024-09-24 22:03:15 +03:00
Leonor Oliveira 2e451b2ed7
Use dw dynamic config (#91882)
* Remove kubernetesPlaylists feature_toggle

* Remove unified_storage_mode

* Remove double import

* Read from config instead from feature_toggle

* cover scenario for when unified storage is not defined

* Be temporarily retro compatible with previous feature toggle

* Properly read unified_storage section

* [WIP] Read new format of config

* Fix test

* Fix other tests

* Generate feature flags file

* Use <group>.<resource> schema

* Use <group>.resource format on the FE as well

* Hide UniStore config from Frontend

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* unwanted changes

* Use feature toggles in the FE. Enforce FTs are present before enabling dual writing
Co-authored-by: Ryan McKinley <ryantxu@users.noreply.github.com>

* use kubernetes playlists feature toggle on the FE

* Remove unwanted code

* Remove configs from the FE

* Remove commented code

* Add more explicit example

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
Co-authored-by: Maicon Costa <maiconscosta@gmail.com>
2024-08-30 11:59:42 +02:00