d801b87db9
LastImportTime for resource. ( #112153 )
...
* LastImportTime for resource.
* Make StorageBackendImpl implement GetResourceLastImportTimes
* More missing implementations of GetResourceLastImportTimes
* Fix import.
* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.
* Implement GetResourceLastImportTimes by mockStorageBackend
* Bump test tolerance.
* Fix postgres query and timezone.
* Fix postgres query and timezone.
* Make linter happy.
2025-10-09 11:27:11 +02:00
2d801eed3c
Remove unused arguments from ResourceIndex and SearchBackend interfaces. ( #112043 )
...
* Remove unused arguments from ResourceIndex and SearchBackend interfaces.
2025-10-06 11:47:52 +02:00
c530cacb1c
Storage: Propagate RV to server on update+delete ( #111866 )
2025-10-01 18:00:02 +03:00
2f2289f226
Chore: Update authlib (foder as top level argument) ( #111800 )
2025-10-01 14:40:28 +00:00
870131a4aa
unified-storage: restrict search to title of documents ( #111842 )
...
* restrict search to title of documents
2025-10-01 08:06:26 -04:00
cfbf64c3fd
Validation: Move validation into apimachinery package ( #111736 )
2025-09-30 12:59:33 +00:00
72d212c5f9
Authlib: Update authz client to use zookies ( #111291 )
...
* Authlib: Update authz client to use zookies
* fix zookie return
* fix linter
2025-09-18 16:24:22 +02:00
294fd943c0
Chore: Update authlib ( #110880 )
...
* Chore: Update authlib
* exclude incompatible version of github.com/grafana/gomemcache
* Update go-jose to v4
* fix jose imports
* remove jose v3 from go.mod
* fix tests
* fix serialize
* fix failing live tests
* add v1 of ES256 testkeys. Port tests to use ES256 instead of HS256
* accept more signature algs for okta and azuread
* azure social graph token sig
* accept more signature algs for oauth refresh and jwt auth
* update workspace
* add a static signer for inproc
* rebase and fix ext_jwt
* fix jwt tests
* apply alex patch on gomemcache
* update linting
* fix ext_jwt panic
* update workspaces
---------
Co-authored-by: Jo Garnier <git@jguer.space>
2025-09-15 12:45:15 +02:00
6fa6a5708a
Cleanup of old search functionality ( #110861 )
...
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.
* Remove reference to deprecated feature toggle.
* Remove feature toggle completely.
* Remove code related to indexing on watch events.
* Fix compilation error.
* Remove unused field.
2025-09-11 08:23:03 +00:00
d715bda8af
Unified Storage: Adds pruner to kv backend ( #110549 )
...
* WIP adding pruner to kv store impl
* pruner only keeps 20 most recent versions
* ignore grafana-kv-data folder
* extracts some stuff to pruner.go file. Adds tests. Adds kvBackendOptions.
* update logging, comments, exports kvbackendoptions fields
* update nooppruner ref
* fixes field casing in test
* fix test
* linter fixes
* remove comment
* make KvStorageBackend private
* Adds pruner key validation and tests. Fixes broken tests.
* update error message when validating pruner key
2025-09-05 10:02:11 -06:00
238f121e10
LastModifiedSince: return latestRV even when it hasn't changed. ( #110391 )
...
Return latestRV.
2025-09-01 08:45:40 +00:00
3a3ba483b1
unified-storage: Skip query when ListModifiedSince cannot return anything. ( #110338 )
...
* Skip query when ListModifiedSince cannot return anything.
* Only save listRV if it's different than sinceRV. This saves a disk access if not needed.
* Add test for ListModifiedSince with same RV.
2025-08-29 14:49:20 +02:00
699bffc9e4
Unified Storage: Dont need to test casing with ordering ( #110245 )
...
Dont need to test casing with ordering.
2025-08-27 14:06:43 -06:00
7921c7da23
unified-storage: reuse index if RV is set even if it is outdated ( #110184 )
...
* unified-storage: reuse index if RV is set even if it is outdated and searchAfterWrite FF is enabled
2025-08-27 11:10:54 -04:00
b4ed217656
Unified Storage: Removes check for name ordering and adds test ( #110183 )
...
removes check for name ordering. Adds test to assert expected name and rv ordering.
2025-08-26 12:39:30 -06:00
78ac555c4d
CI: Run gofmt on Linting action ( #110159 )
...
* CI: Run gofmt on Linting action
* Chore: Run gofmt on all packages
2025-08-26 16:14:03 +02:00
4c996a8a7f
search: Guarantee search-after-write consistency ( #109972 )
2025-08-25 10:13:07 +02:00
cace999671
Unified Storage: Add ListSinceModified to StorageBackend ( #109697 )
...
* WIP added ListSinceModified to StorageBackend interface
* fix compile time check
* Fix method name
* Fix naming
* fix the rest of the ListSinceModified names
* Uses resource key without name field
* get latest rv from resource_version. Update test.
* adds moar tests
* adds method stub for ListModifiedSince to other StorageBackend implementations
* adds dummy impl to noop storage backend for ListModifiedSince
* skip tests for badger kv backend for now
* fixes tests and adds badgerkv impl for ListModifiedSince
* add badger kv impl
* adds test for new query
* adds test data for new query
* adds ListModifiedSince stub to mockStorageBackend
* uncomment tests
* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.
* remove comments
* remove folder from query (dont need it, yet?)
* regen test queries
* updates test
* updates function comment
* use resourcepb.ResourceKey instead of ModifiedResourceKey
* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.
* skip TestListModifiedSince for kv backend
* use WatchEvent_Type for action type
* remove redundant fields from order by clause and regen test data for query
* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
7b4fcd486a
Revert "fix: unified resource server list queries order column" ( #109529 )
2025-08-12 17:29:50 +02:00
c28b2215e0
Improve indexing observability ( #108901 )
...
* Add details to tracing spans when creating index.
* Log reason for building index.
* Log reason for building index.
* Remove initialization of labels to avoid unnecessary metrics.
* Track succcessful, failed and skiped index builds. Track index build time for individual index, not all indexes.
* Revert removal of labels initialization.
2025-07-30 16:34:15 +02:00
4a779c4ccb
fix: unified resource server list queries order column ( #108094 )
2025-07-16 08:09:31 +02:00
7002ab90ae
unistore: save returns a writecloser ( #107955 )
...
* unistore: save returns a writecloser
* go-lint
* address comments
2025-07-11 18:25:48 +02:00
ea0ddb3fc9
unistore: refactor get to return a reader ( #107951 )
2025-07-11 11:10:19 +02:00
b6dd08da2f
unistore: fix delete and db closed in kv store ( #107918 )
...
CodeQL checks / Analyze (actions) (push) Has been cancelled
Details
CodeQL checks / Analyze (go) (push) Has been cancelled
Details
CodeQL checks / Analyze (javascript) (push) Has been cancelled
Details
* fix delete and db closed
* fix tests
2025-07-10 11:34:36 +02:00
9d2eadcfd2
PoC: replace using testing.Short in storage/unified package with integration test skip method ( #107887 )
...
* PoC: check if testing.Short is called from integration tests only.
* Rename helper function.
* Fix logic.
* Remove skipping of integration tests from non-integration tests.
* Remove skipping of integration tests from non-integration tests.
* Fix import.
2025-07-09 15:32:10 +00:00
7787ff6b2e
unistore: add more kvstore tests ( #107692 )
...
add more kvstore tests
2025-07-07 15:00:40 +02:00
8d8b824f73
unistore: skipping badger test failing atm ( #107572 )
...
skipping badger test failing atm
2025-07-03 12:31:07 +00:00
d55541735a
unistore: use different ns for each test ( #107509 )
...
use different ns for each test
2025-07-02 13:50:30 +02:00
696657bdd1
unistore: add kv based storage backend ( #107305 )
...
* Add datastore
* too many slashes
* lint
* add metadata store
* simplify meta
* Add eventstore
* golint
* lint
* Add datastore
* too many slashes
* lint
* pr comments
* extract ParseKey
* readcloser
* remove get prefix
* use dedicated keys
* parsekey
* sameresource
* unrelated
* name
* renmae tests
* add key validation
* fix tests
* refactor a bit
* lint
* allow empty ns
* get keys instead of list
* rename the functions
* refactor yield candidate
* update test
* unistore: add LastResourceVersion to datastore
* lint
* use map string
* missing err check
* fix
* Add storage backend
* remove hasmore
* fix tests
* small refactor
* pre-alloc
* extract the folder
* lint
* refactor
* handle context canceled in ListHistory to pass the tests
* fix the resource test
* unistore: provide generic tests for the kv interface (#107443 )
unistore: move the kv tests to the testing package
* Update pkg/storage/unified/resource/storage_backend_test.go
Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
* address comments
* comments
* comments
* comments
* normalise the names and add helper method
* events comments
* rename function
---------
Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
2025-07-02 10:57:37 +00:00
0982cfd9a0
Unified Storage/Search: Add max count config for indexing ( #107255 )
...
* Add max count config for indexing
* Build empty index when max count is exceeded
* Address linting
* Refactor buildIndexes
* Add test for max count threshold
* Update test doc comments
* Refactor TestBuildIndexes_MaxCountThreshold to not use mock framework
* Rename mocks used in TestBuildIndexes_MaxCountThreshold
* Refactor mockResourceIndex
* Test setting of indexing threshold configs
* Tweak comments, log
* Fix logging in buildEmptyIndex
* Export and reuse TestDocumentBuilderSupplier
* Reuse MockResourceIndex
2025-06-27 14:00:39 +02:00
5150ebfba8
unistore: index search references ( #106954 )
2025-06-24 14:54:07 -05:00
9062d88ea0
unistore: filter trash requests ( #106767 )
...
* deleted by user
* use the correct checker
* add tests
* refactor
2025-06-24 10:17:34 +02:00
7864c1660f
K8s: Get trash fixes ( #106411 )
2025-06-11 13:54:38 -05:00
eb8f3bc071
fix(unified-storage): wait for result to be available ( #105922 )
2025-05-23 11:03:01 +02:00
a13fe75de8
refactor(unified-storage): set the GUID in the resource server ( #105683 )
2025-05-21 09:49:49 +02:00
002f46736a
refactor(unified-storage): move generated protos to own pkg ( #105356 )
Actionlint / Lint GitHub Actions files (push) Waiting to run
Details
Backend Code Checks / Validate Backend Configs (push) Waiting to run
Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (1/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (2/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (3/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (4/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (5/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (6/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (7/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana (${{ matrix.shard }}) (8/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (1/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (2/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (3/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (4/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (5/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (6/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (7/8) (push) Waiting to run
Details
Backend Unit Tests / Grafana Enterprise (${{ matrix.shard }}) (8/8) (push) Waiting to run
Details
CodeQL checks / Analyze (actions) (push) Waiting to run
Details
CodeQL checks / Analyze (go) (push) Waiting to run
Details
CodeQL checks / Analyze (javascript) (push) Waiting to run
Details
CodeQL checks / Analyze (python) (push) Waiting to run
Details
Lint Frontend / Verify i18n (push) Waiting to run
Details
Lint Frontend / Lint (push) Waiting to run
Details
Lint Frontend / Typecheck (push) Waiting to run
Details
Lint Frontend / Betterer (push) Waiting to run
Details
golangci-lint / lint-go (push) Waiting to run
Details
End-to-end tests / Build & Package Grafana (push) Waiting to run
Details
End-to-end tests / ${{ matrix.suite }} (dashboards-suite) (push) Blocked by required conditions
Details
End-to-end tests / ${{ matrix.suite }} (panels-suite) (push) Blocked by required conditions
Details
End-to-end tests / ${{ matrix.suite }} (smoke-tests-suite) (push) Blocked by required conditions
Details
End-to-end tests / ${{ matrix.suite }} (various-suite) (push) Blocked by required conditions
Details
End-to-end tests / ${{ matrix.suite }} (old arch) (old-arch/dashboards-suite) (push) Blocked by required conditions
Details
End-to-end tests / ${{ matrix.suite }} (old arch) (old-arch/panels-suite) (push) Blocked by required conditions
Details
End-to-end tests / ${{ matrix.suite }} (old arch) (old-arch/smoke-tests-suite) (push) Blocked by required conditions
Details
End-to-end tests / ${{ matrix.suite }} (old arch) (old-arch/various-suite) (push) Blocked by required conditions
Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (1) (push) Waiting to run
Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (2) (push) Waiting to run
Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (3) (push) Waiting to run
Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (4) (push) Waiting to run
Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (5) (push) Waiting to run
Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (6) (push) Waiting to run
Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (7) (push) Waiting to run
Details
Frontend tests / Unit tests (${{ matrix.chunk }} / 8) (8) (push) Waiting to run
Details
Integration Tests / Sqlite (${{ matrix.shard }}) (1/8) (push) Waiting to run
Details
Integration Tests / Sqlite (${{ matrix.shard }}) (2/8) (push) Waiting to run
Details
Integration Tests / Sqlite (${{ matrix.shard }}) (3/8) (push) Waiting to run
Details
Integration Tests / Sqlite (${{ matrix.shard }}) (4/8) (push) Waiting to run
Details
Integration Tests / Sqlite (${{ matrix.shard }}) (5/8) (push) Waiting to run
Details
Integration Tests / Sqlite (${{ matrix.shard }}) (6/8) (push) Waiting to run
Details
Integration Tests / Sqlite (${{ matrix.shard }}) (7/8) (push) Waiting to run
Details
Integration Tests / Sqlite (${{ matrix.shard }}) (8/8) (push) Waiting to run
Details
Integration Tests / MySQL (${{ matrix.shard }}) (1/8) (push) Waiting to run
Details
Integration Tests / MySQL (${{ matrix.shard }}) (2/8) (push) Waiting to run
Details
Integration Tests / MySQL (${{ matrix.shard }}) (3/8) (push) Waiting to run
Details
Integration Tests / MySQL (${{ matrix.shard }}) (4/8) (push) Waiting to run
Details
Integration Tests / MySQL (${{ matrix.shard }}) (5/8) (push) Waiting to run
Details
Integration Tests / MySQL (${{ matrix.shard }}) (6/8) (push) Waiting to run
Details
Integration Tests / MySQL (${{ matrix.shard }}) (7/8) (push) Waiting to run
Details
Integration Tests / MySQL (${{ matrix.shard }}) (8/8) (push) Waiting to run
Details
Integration Tests / Postgres (${{ matrix.shard }}) (1/8) (push) Waiting to run
Details
Integration Tests / Postgres (${{ matrix.shard }}) (2/8) (push) Waiting to run
Details
Integration Tests / Postgres (${{ matrix.shard }}) (3/8) (push) Waiting to run
Details
Integration Tests / Postgres (${{ matrix.shard }}) (4/8) (push) Waiting to run
Details
Integration Tests / Postgres (${{ matrix.shard }}) (5/8) (push) Waiting to run
Details
Integration Tests / Postgres (${{ matrix.shard }}) (6/8) (push) Waiting to run
Details
Integration Tests / Postgres (${{ matrix.shard }}) (7/8) (push) Waiting to run
Details
Integration Tests / Postgres (${{ matrix.shard }}) (8/8) (push) Waiting to run
Details
Reject GitHub secrets / reject-gh-secrets (push) Waiting to run
Details
Run dashboard schema v2 e2e / dashboard-schema-v2-e2e (push) Waiting to run
Details
Dispatch sync to mirror / dispatch-job (push) Waiting to run
Details
trigger-dashboard-search-e2e / trigger-search-e2e (push) Waiting to run
Details
Trivy Scan / trivy-scan (push) Waiting to run
Details
2025-05-15 21:36:52 +02:00
e162c69c34
search: fix document missing at startup ( #105198 )
...
* fix document missing at startup
* go-lint
* fix tests
* fix tests
* fix integration tests now that we are storing real values
2025-05-12 07:56:25 +02:00
15b3de5893
search: add index batching ( #104163 )
...
* add basic search backend integration tests
* add search backend benchmark
* add benchmark indexServer
* fix
* lint
* add more tests
* lint
* do not use the poller
* batch write
* refactor and add tests
* improvements
* improvements
* cleanup
* only observe index success
* add monitorIndexEvents method
* nit use switch instead of if
* make newIndexQueueProcessor private
* simplify runProcessor
* go lint
2025-05-09 15:36:21 +02:00
b3b92e2471
Storage: Change timeout for flaky test ( #104065 )
...
Change timeout for flaky test
Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
2025-04-15 19:28:27 +01:00
e69052a417
unistore: add check when update the folder of a resource ( #102699 )
...
* Add check for move folder
* make the server test generic
* address comment
2025-04-14 16:57:40 +02:00
be696dd70c
unified storage: check for iterator errors after each call to iter.Next. ( #102804 )
...
* unified storage: check for iterator errors after each call to iter.Next.
* Extracted test to separate method, add 500 events before listing.
2025-04-11 16:25:40 +02:00
f0a6327edc
Unified Storage: Don't read before create ( #102906 )
...
* Unified Storage: Don't read before create
* test: use the existing test infra
* fix: support pq
We use pgx, but it seems to be wrapped in a pq driver shim, causing the errors to be remapped to pq's type. Weird
situation.
* feat: support CDK backend
* revert: there is a postgres_tests block
* fix(CDK): only check existence on ADDED updates
* fix(CDK): use ReadResource to deal with deleted files
2025-03-31 15:06:31 +02:00
fe1a72e41b
fix(unified-storage): unlist the deleted resource's versions ( #102588 )
2025-03-27 18:34:37 +01:00
6d570db312
Storage: Add ascending order support for NotOlderThan queries and introduce ResourceVersionMatch_Unset as default ( #102505 )
...
* Add support for ASC ordering and introduce ResourceVersionMatch_Unset as default
Add SortAscending to continue token and add integration test for pagination.
* Change protobuf order
* Make backwards compatible
* Update pkg/storage/unified/sql/backend.go
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
---------
Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
2025-03-21 18:35:32 +02:00
0bafd4e99d
unistore: update spanner ddl to include unistore tables ( #102198 )
...
* provision unistore tables
* update the tests to use generated namespace
* update ddl
* update ddl
* clean streams
* missing space
* use engine.db.query
2025-03-18 14:37:11 +01:00
3589baac68
Unistore: Batch write events ( #101381 )
...
* Batch write events
* Improve instrumentation
* Measure batch phases
* Detect lock contention
* remove the execBatch goroutine
* removing tracing prefix
* detect context cancel
* batch channel map
2025-03-13 10:24:12 +02:00
8c935c8f4a
[unistore] Add benchmark for write throughput ( #101345 )
...
* Add generic benchmark
* address comments
2025-02-26 17:17:35 +02:00
8f37822050
Unified/Blob: Allow direct access to UIDs ( #101339 )
2025-02-26 16:18:59 +02:00
53e91fd5e8
unistore: close event stream on context cancelation ( #101293 )
...
* add tests for broacaster
* fix sql notifier not closing the stream
* fix sql notifier not closing the stream
* close sub
* fix broadcaster test
* fix broadcaster test
* suggestion
2025-02-25 08:28:31 -08:00
e13bd52da6
unistore: move continue to the resource package ( #101206 )
...
* minor improvements to tests
* move continue token
* update sql backend
2025-02-24 18:02:30 +02:00
Peter Štibraný
Ryan McKinley
Will Assis
Alexander Zobnin
owensmallwood
Matheus Macabu
Mustafa Sencer Özcan
Georges Chaudy
Arati R.
Stephanie Hingtgen
Jean-Philippe Quéméner
Marco de Abreu
Mariell Hoversholm