Commit Graph

36 Commits

Author SHA1 Message Date
Peter Štibraný d801b87db9
LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Ryan McKinley c530cacb1c
Storage: Propagate RV to server on update+delete (#111866) 2025-10-01 18:00:02 +03:00
Peter Štibraný da43e2ae07
Don't use transaction in ListModifiedSince. (#110392)
* Don't use transaction in ListModifiedSince.

To guarantee that we don't include events with RV > LatestRV, we include the check in SQL query instead.

* Fix integration test by converting SQL comments into template comments.
2025-09-01 11:39:02 +02:00
owensmallwood cace999671
Unified Storage: Add ListSinceModified to StorageBackend (#109697)
* WIP added ListSinceModified to StorageBackend interface

* fix compile time check

* Fix method name

* Fix naming

* fix the rest of the ListSinceModified names

* Uses resource key without name field

* get latest rv from resource_version. Update test.

* adds moar tests

* adds method stub for ListModifiedSince to other StorageBackend implementations

* adds dummy impl to noop storage backend for ListModifiedSince

* skip tests for badger kv backend for now

* fixes tests and adds badgerkv impl for ListModifiedSince

* add badger kv impl

* adds test for new query

* adds test data for new query

* adds ListModifiedSince stub to mockStorageBackend

* uncomment tests

* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.

* remove comments

* remove folder from query (dont need it, yet?)

* regen test queries

* updates test

* updates function comment

* use resourcepb.ResourceKey instead of ModifiedResourceKey

* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.

* skip TestListModifiedSince for kv backend

* use WatchEvent_Type for action type

* remove redundant fields from order by clause and regen test data for query

* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
Stephanie Hingtgen 7864c1660f
K8s: Get trash fixes (#106411) 2025-06-11 13:54:38 -05:00
Jean-Philippe Quéméner 002f46736a
refactor(unified-storage): move generated protos to own pkg (#105356)
Actionlint / Lint GitHub Actions files (push) Waiting to run Details
Backend Code Checks / Validate Backend Configs (push) Waiting to run Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (1/8) (push) Waiting to run Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (2/8) (push) Waiting to run Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (3/8) (push) Waiting to run Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (4/8) (push) Waiting to run Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (5/8) (push) Waiting to run Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (6/8) (push) Waiting to run Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (7/8) (push) Waiting to run Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (8/8) (push) Waiting to run Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (1/8) (push) Waiting to run Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (2/8) (push) Waiting to run Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (3/8) (push) Waiting to run Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (4/8) (push) Waiting to run Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (5/8) (push) Waiting to run Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (6/8) (push) Waiting to run Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (7/8) (push) Waiting to run Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (8/8) (push) Waiting to run Details
CodeQL checks / Analyze (actions) (push) Waiting to run Details
CodeQL checks / Analyze (go) (push) Waiting to run Details
CodeQL checks / Analyze (javascript) (push) Waiting to run Details
CodeQL checks / Analyze (python) (push) Waiting to run Details
Lint Frontend / Verify i18n (push) Waiting to run Details
Lint Frontend / Lint (push) Waiting to run Details
Lint Frontend / Typecheck (push) Waiting to run Details
Lint Frontend / Betterer (push) Waiting to run Details
golangci-lint / lint-go (push) Waiting to run Details
End-to-end tests / Build & Package Grafana (push) Waiting to run Details
End-to-end tests / ${{ matrix.suite }} (dashboards-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (panels-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (smoke-tests-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (various-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (old arch) (old-arch/dashboards-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (old arch) (old-arch/panels-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (old arch) (old-arch/smoke-tests-suite) (push) Blocked by required conditions Details
End-to-end tests / ${{ matrix.suite }} (old arch) (old-arch/various-suite) (push) Blocked by required conditions Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (1) (push) Waiting to run Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (2) (push) Waiting to run Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (3) (push) Waiting to run Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (4) (push) Waiting to run Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (5) (push) Waiting to run Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (6) (push) Waiting to run Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (7) (push) Waiting to run Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (1/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (2/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (3/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (4/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (5/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (6/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (7/8) (push) Waiting to run Details
Integration Tests / Sqlite (${{ matrix.shard }}) (8/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (1/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (2/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (3/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (4/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (5/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (6/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (7/8) (push) Waiting to run Details
Integration Tests / MySQL (${{ matrix.shard }}) (8/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (1/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (2/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (3/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (4/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (5/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (6/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (7/8) (push) Waiting to run Details
Integration Tests / Postgres (${{ matrix.shard }}) (8/8) (push) Waiting to run Details
Reject GitHub secrets / reject-gh-secrets (push) Waiting to run Details
Run dashboard schema v2 e2e / dashboard-schema-v2-e2e (push) Waiting to run Details
Dispatch sync to mirror / dispatch-job (push) Waiting to run Details
trigger-dashboard-search-e2e / trigger-search-e2e (push) Waiting to run Details
Trivy Scan / trivy-scan (push) Waiting to run Details
2025-05-15 21:36:52 +02:00
Matheus Macabu fc9f32a9f6
SQLTemplates: Add helper to ensure all templates have a test-case (#103964)
* SQLTemplates: Add helper to ensure all templates have a test-case associated

* UnifiedStorage: Add missing sql template test case

* LegacyDashboards: Add sql templates fs to test cases for exhaustiveness check

* RBACStore: Add sql templates fs to test cases for exhaustiveness check

* LegacyIAM: Add missing sql template test cases
2025-04-22 11:21:51 +02:00
Ryan McKinley 9c49c601f1
Unistore: Add generation column (#102222) 2025-04-01 20:38:23 +03:00
Mustafa Sencer Özcan fe1a72e41b
fix(unified-storage): unlist the deleted resource's versions (#102588) 2025-03-27 18:34:37 +01:00
Stephanie Hingtgen c33a53a47a
K8s: Remove restore functionality; can be done with list (#102560) 2025-03-20 16:38:32 -05:00
Jean-Philippe Quéméner 3e3ab8e064
fix(unified-storage): reduce lock contention when pruning (#102425) 2025-03-19 12:15:04 +01:00
Jean-Philippe Quéméner 1700a8aa9f
feat(unified-storage): prune history table based on limits (#101970) 2025-03-17 11:36:38 +01:00
Georges Chaudy 3589baac68
Unistore: Batch write events (#101381)
* Batch write events

* Improve instrumentation

* Measure batch phases

* Detect lock contention

* remove the execBatch goroutine

* removing tracing prefix

* detect context cancel

* batch channel map
2025-03-13 10:24:12 +02:00
Ryan McKinley a5355fd66c
Storage: Add command line tool to migrate legacy dashboards (and folders) to unified storage (#99199) 2025-02-11 19:57:46 +02:00
Ryan McKinley 9d30911107
SQL/Storage: Use xorm code migration for DeletionMarker (#99353) 2025-01-22 14:07:08 +03:00
Ryan McKinley d91d729749
Storage/SQL: Support more filter by namespace/group/resource (#99249) 2025-01-21 12:15:08 +03:00
Ryan McKinley 356b32008b
Storage: Show history+trash using the list command (#99009)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-01-17 15:54:25 +03:00
Ryan McKinley 429da7fd68
SQL/Blob: Add support for blob storage to SQL backend (#98192) 2025-01-08 22:08:10 +02:00
Stephanie Hingtgen 8f6e9f8ed0
Restores in app platform (#97582) 2024-12-13 16:55:43 -06:00
Ryan McKinley ea17b79c09
Search: Return counts for values within an folder/repository (#97534) 2024-12-10 20:37:37 +02:00
Ryan McKinley 8165258a2d
Storage: Query stats within a namespace (#97403) 2024-12-05 13:58:13 +03:00
Ryan McKinley 9d89d8757f
Search: Build index from resource stats (#97320) 2024-12-03 20:20:27 +03:00
Georges Chaudy c3571752b6
Unitstore: Fix the Folder migration (#96078)
* UniStore: Add folder to the SQL backend

This reverts commit aaa4f4a23e.

* fix folder migration
2024-11-12 12:52:04 +01:00
Leonor Oliveira aaa4f4a23e
Revert "UniStore: Add folder to the SQL backend" (#95966)
Revert "UniStore: Add folder to the SQL backend (#95671)"

This reverts commit f075662696.
2024-11-06 17:04:57 +02:00
Georges Chaudy f075662696
UniStore: Add folder to the SQL backend (#95671)
* Add folder to the SQL backend

* remove unused key

* fix tests

* Return folders from watch
2024-11-05 10:37:23 +01:00
Georges Chaudy d999b415df
UniStore: Use epoch with microsecond resolution as RV (#92638)
* Use epoch with microsecond resolution as RV

* fix backend tests

* Add solution for when the clock goes back

* Add solution for when the clock goes back

* generate mocks

* go lint

* remove comment

* Use Greatest instead of max in msyql and postgres

* update tests

* Update pkg/storage/unified/sql/sqltemplate/dialect_sqlite.go

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>

* cast to bigint

* add additional round trip

* increment the RV using 2 sql round trips instead of 3

* cleanup comments

* cast unix timestamp to integer

* fix postgres query

* remove old increment test data

* remove greatest

* cast unix_timestamp to signed

* Use statement_timestamp instead of clock_timestamp

---------

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-10-11 12:11:33 +03:00
Georges Chaudy 03f55e5288
Unistore : Ensure Watch works in HA mode (#94120)
* Revert "Revert "Unistore : Ensure Watch works in HA mode." (#94097)"

This reverts commit 7c3fc2f261.

* make previous_resource_version nullable

* handle nil case
2024-10-07 10:01:53 +02:00
Georges Chaudy 7c3fc2f261
Revert "Unistore : Ensure Watch works in HA mode." (#94097)
Revert "Unistore : Ensure Watch works in HA mode. (#93428)"

This reverts commit 0a26c9e9ae.
2024-10-01 18:45:47 +00:00
Georges Chaudy 0a26c9e9ae
Unistore : Ensure Watch works in HA mode. (#93428)
* Replace Watch with WatchNext

* remove watchset

* fix previous page and closing the channel

* Remove the broadcaster cache to prevent dupplicated events

* add watch bookmark

* add watch bookmark

* cleanup comments

* disable the tests for bookmarks for now

* Ensure we send previosu events

* lint

* re-introduce the cache

* load from cache

* disabling legacy test

* disabling legacy test

* Update pkg/storage/unified/resource/server.go

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>

* Could not read previous events

* add proper migration

* Add previous_resource_version to both history and resource

* First event should have an RV of 2 and not 1

* Test both storage backends

* fix the inital RV for the sql backend

* ensure graceful stop of the stream decoder

* gocyclo

---------

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-09-30 13:14:07 +02:00
Ryan McKinley 5a30e12a10
SQLTemplate: Make Ident only work for identifiers (not any string) (#92387) 2024-08-27 13:22:40 +03:00
Ryan McKinley f432a1713b
SQLTemplate: Swap the IFace with the concrete struct (#92008) 2024-08-16 15:12:37 +03:00
Ryan McKinley d9cabe5e14
SQL: Add sql template test helper (#91953) 2024-08-16 14:36:56 +03:00
Ryan McKinley f804b0baa3
Storage: Return an iterator from backend List (#91185) 2024-07-31 12:05:59 +03:00
Georges Chaudy e4ed000d05
ResourceStore: Fix sql List (#91043)
* fix resource list

* ensure we get the resource version without FOR SHARE
2024-07-26 13:56:24 +02:00
Diego Augusto Molina 399d77a0fd
Resource server improvements and fixes (#90715)
* cleanup dependencies and improve list method
* Improve Resource Server API, remove unnecessary dependencies
* Reduce the API footprint of ResourceDBInterface and its implementation
* Improve LifecycleHooks to use context
* Improve testing
* reduce API size and improve code
* sqltemplate: add DialectForDriver func and improve naming
* improve lifecycle API
* many small fixes after adding more tests
2024-07-22 20:08:30 +03:00
Georges Chaudy 08c611c68b
ResourceServer: Resource store sql backend (#90170) 2024-07-18 08:03:18 -07:00