Commit Graph

35 Commits

Author SHA1 Message Date
Matias Chomicki 2573cbec08
Loki: Added support to split queries by stream shard (#94245)
* Add shard query splitting implementation

* Shard query splitting: reuse function from query splitting

* Shard query splitting: remove max line limit

* Shard query splitting: update test

* Shard query splitting: fix types and non-sharded queries

* Merge responses: fix log merging

* Merge responses: remove legacy code

* Query splitting: add support to retry failed requests

* Query splitting: unit test request retrying

* Query splitting: add unsubscriptions

* Shard query splitting: fix retrying

* Shard query splitting: switch to dynamic grouping

* Shard query splitting: update group size thresholds and fix -1 query

* Shard query splitting: update initial group size + don't retry parse errors

* Shard query splitting: update unit test

* chore: update mock value

* Shard query splitting: add support for multiple targets

* chore: update description

* Shard query splitting: use group targets

* chore: filter hidden queries

* Shard query splitting: issue initial log query without sharding

* Splitting: fix retrying in both methods

* Merge responses: keep execution time

* Shard query splitting: remove no-shard attempt

* Shard query splitting: adjust groups based on rate of change

* chore: clean up experiments

* Shard query splittng: remove log query restrictions

* Shard query splitting: remove fallback to time splitting

* Loki: add new query direction

* Missing generated file

* LokiOptionField: integrate new query direction

* Shard query splitting: delegate non-scan queries to time splitting

* Query splitting: do not retry queries with parse errors

* Loki datasource: add placeholder for feature flag

* Shard query splitting: add function with support criteria

* Shard query splitting: refactor query modification and shard logs volume

* Shard query splitting: update unit tests

* chore: Update scan direction tooltip

* chore: formatting

* LogsVolumePanel: fix missing state in logs volume panel data

* Merge responses: better handle missing nanoseconds

* LokiQueryOptionFields: display query direction for log queries

* loki: process scan direction as backward

* Loki datasource: restrict sharding to Explore

* Retrying: invert criteria and move to response utils

* Formatting

* Use log volume refId constant

* Fix import order

* Create feature flag

* Use feature toggle

* LogsVolumePanel: prevent flashing no data while streaming
2024-10-23 14:21:03 +03:00
Matias Chomicki bca8bd3c8b
Logs: Added query splitting support for forward searching logs (#93450)
Query splitting: add support for forward searching logs
2024-09-23 11:22:06 +02:00
Ivana Huckova a83e01918a
Loki: Remove dependecy on core and move to `@grafana/o11y-ds-frontend` (#81284)
* Loki: Remove dependecy on core and move to o11y-ds-frontend

* Fix ctr -> ctrl

* Remove test helpers
2024-01-29 11:19:34 +01:00
Matias Chomicki aa670280fc
Logs: add infinite scrolling to Explore (#76348)
* Explore: propose action, thunk, and decorators for load more

* LogsContainer: add loadMore method

* Query: remove unused var

* Loading more: use navigation to simulate scrolling

* Explore: figure out data combination

* Fix imports

* Explore: deduplicate results when using query splitting

* LogsNavigation: add scroll behavior

* Remove old code

* Scroll: adjust delta value

* Load more: remove refIds from signature

We can resolve them inside Explore state

* Load more: rename to loadMoreLogs

* Infinite scrolling: use scrollElement to listen to scrolling events

* Explore logs: add fixed height to scrollable logs container

* Logs: make logs container the scrolling element

* Logs: remove dynamic logs container size

It works very well with 1 query, but breaks with more than 1 query or when Logs is not the last rendered panel

* Logs navigation: revert changes

* Infinite scroll: create component

* Infinite scroll: refactor and clean up effect

* Infinite scroll: support oldest first scrolling direction

* Infinite scroll: support loading oldest logs in ascending and descending order

* Infinite scroll: use scroll to top from logs navigation

* Logs: make logs container smaller

* Logs: make container smaller

* State: integrate explore's loading states

* Infinite scroll: add loading to effect dependency array

* Infinite scroll: display message when scroll limit is reached

* Infinite scroll: add support to scroll in both directions

* Infinite scroll: capture wheel events for top scroll

* scrollableLogsContainer: deprecate in favor of logsInfiniteScrolling

* Infinite scroll: implement timerange limits

* Infinite scroll: pass timezone

* Fix unused variables and imports

* Infinite scroll: implement timerange limits for absolute time

* Infinite scroll: fix timerange limits for absolute and relative times

* Infinite scroll: reset out-of-bounds message

* Logs: make container taller

* Line limit: use "displayed" instead of "returned" for infinite scrolling

* Infinite scrolling: disable behavior when there is no scroll

* Remove console log

* Infinite scroll: hide limit reached message when using relative time

* Logs: migrate styles to object notation

* Prettier formatting

* LogsModel: fix import order

* Update betterer.results

* Logs: remove exploreScrollableLogsContainer test

* Infinite scroll: display loader

* Infinite scroll: improve wheel handling

* Explore: unify correlations code

* Explore: move new function to helpers

* Remove comment

* Fix imports

* Formatting

* Query: add missing awaits in unit test

* Logs model: add unit test

* Combine frames: move code to feature/logs

* Explore: move getCorrelations call back to query

It was causing a weird test failure

* Fix imports

* Infinite scroll: parametrize scrolling threshold

* Logs: fix overflow css

* Infinite scroll: add basic unit test

* Infinite scroll: add unit test for absolute time ranges

* Formatting

* Explore query: add custom interaction for scrolling

* Query: move correlations before update time

* Fix import in test

* Update comment

* Remove comment

* Remove comment

* Infinite scroll: report interactions from component

* Fix import order

* Rename action

* Infinite scroll: update limit reached message

* Explore logs: remove type assertion

* Update betterer
2024-01-12 13:22:03 +02:00
Matias Chomicki c58ed41d50
Loki Logs volume: Added a query splitting loading indicator to the Logs Volume graph (#79681)
* Logs volume: distinguish annotations from time series frames

* Update unit tests

* Update unit tests

* Formatting
2023-12-19 14:33:18 +02:00
Galen Kistler d6eca75bf7
Loki: strip out invalid options for logs/metrics queries (#79409)
* fix bug in split query where metric queries with maxLines of 0 would not execute
2023-12-13 07:08:30 -06:00
Matias Chomicki 24082d61c2
Query splitting: add key to merged response (#78522)
* Query splitting: add key to response

* Query splitting: use queryGroupId as key

* Add unit test
2023-11-30 14:27:44 +02:00
Torkel Ödegaard fc0933d5f4
Schema: Clean up dashboard variables schema (#76121)
* Cleanup variable schema model

* minor fix

* remove type ignores that are no longer type errors
2023-10-09 09:10:52 +02:00
Sven Grossmann 07eb4b1b90
Loki: Remove `distinct` operation (#73938)
* remove distinct

* trigger ci

* update yarn.lock

* fix import
2023-08-28 19:26:59 +02:00
Matias Chomicki 6f3dfb2d8b
Loki Query Splitting: Ignore empty queries like hidden queries (#72376)
Loki query splitting: ignore empty queries like hidden queries
2023-07-26 15:18:55 +01:00
Ivana Huckova 82c125d450
Loki: Implement step editor (#69648)
* Loki: Implement step editor

* Update to keep value

* Remove console.log

* Remove white space

* Update public/app/plugins/datasource/loki/querybuilder/components/LokiQueryBuilderOptions.tsx

Co-authored-by: Matias Chomicki <matyax@gmail.com>

* Import trim

* Update using of step in split queries

* Add tests

* Add tests

* Remove step interpolation

---------

Co-authored-by: Matias Chomicki <matyax@gmail.com>
2023-06-16 19:08:29 +03:00
Sven Grossmann 15d55c371c
Loki: Add feature tracking to reduce repetition (#69900)
add tracking to predefined queries
2023-06-12 13:03:47 +02:00
Ivana Huckova 38fc22538a
Loki: Don't split queries if they use $__range variables (#69852)
* Loki: Don't split queries if they use  variables

* Update public/app/plugins/datasource/loki/querySplitting.ts

Co-authored-by: Matias Chomicki <matyax@gmail.com>

---------

Co-authored-by: Matias Chomicki <matyax@gmail.com>
2023-06-12 11:22:04 +02:00
Matias Chomicki e31ea1da9d
Query Splitting: Display progress of sub-requests using an annotation frame (#69574)
* Loki Query Splitting: Add and update loading frame to merged response

* Update unit test

* Loki Query Splitting: use data frame constructor function
2023-06-07 13:52:06 +00:00
Matias Chomicki ee0ee70aa1
Loki: Exclude queries using DISTINCT from query splitting (#69377)
* Query utils: add function to identify distinct queries

* Chore: return false when the expected node is found

* Query splitting: exclude distinct queries from splitting

* Group queries excluded from splitting
2023-06-02 12:50:23 +02:00
Matias Chomicki 2bfd415c07
Loki Query Splitting: Enable tracking for split queries (#68645)
* Loki datasource: move tracking out of runQuery

* Types: move LokiGroupedRequest to types file

* Tracking: add function to track grouped queries

* Query splitting: add tracking to split queries runner

* Remove unnecessary types

* Add unit test

* Tracking: include test case with hidden query

* Update public/app/plugins/datasource/loki/tracking.ts

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* Update public/app/plugins/datasource/loki/tracking.ts

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* Update public/app/plugins/datasource/loki/tracking.ts

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* Update public/app/plugins/datasource/loki/tracking.ts

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* Tracking: add is_split dimension

---------

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>
2023-05-25 18:39:16 +03:00
Domas 5dc529a55d
Query Splitting: Fix for handling queries with no requestId (#66161)
Loki query splitting fix for when there's no request id
2023-04-07 13:31:18 +03:00
Matias Chomicki 44beef2e41
Loki Query Splitting: Fix bug for mixed split durations (#65925)
* Query splitting: fix next request group pointer calculation

* Update unit test
2023-04-04 15:51:41 +02:00
Matias Chomicki 663ed7ba82
Loki Query Splitting: Rename from "chunk" to "splitting" (#65630)
* Update chunking mentions in the code and ui

* Rename files and exported functions

* Rename configuration attribute

* Rename grouped querying function name

* Update more function names

* Update unit test

* Update unit tests

* More renames

* Rename time splitting functions
2023-04-03 12:30:08 +00:00
Matias Chomicki 6093e45178
Loki Query Chunking: Refactor naming conventions and reorganize code (#65056)
* Rename splitting files to chunking

* Rename time chunking functions

* Move response functions to response utils

* Remove some blank spaces

* Add an extra test case for frame refIds and names
2023-03-21 15:54:18 +00:00
Sven Grossmann 7261c6f7cd
DataSourceWithBackend: Add `queryGroupId` to find correlated/related queries (#64587)
* add `correlationId` to queries

* trace correlation id in backend

* add correlation id to loki's span

* add correlation id to query chunks

* fix test

* fix DataSourceWithBackend test

* change to `queryGroupId`

* remove empty line

* fix test in `DataSourceWithBackend`
2023-03-17 15:48:13 +01:00
Sven Grossmann 40014f1454
Loki: Add option to define chunk duration per query (#64834)
* add query option to configure chunk ranges

* remove `isValidDuration` check

* Update public/app/plugins/datasource/loki/querybuilder/components/LokiQueryBuilderOptions.tsx

Co-authored-by: Matias Chomicki <matyax@gmail.com>

* change to `chunkDuration`
added tests

* no need to call `toString`

---------

Co-authored-by: Matias Chomicki <matyax@gmail.com>
2023-03-16 16:30:12 +01:00
Gábor Farkas 5d8ec2756e
loki: query chunking: better error handling (#64811)
* loki: query chunking: better error handling

* better comment text

Co-authored-by: Matias Chomicki <matyax@gmail.com>

* better comment text

Co-authored-by: Matias Chomicki <matyax@gmail.com>

* changed merge approach

* simplified code

* fixed test

* removed forgotten code

---------

Co-authored-by: Matias Chomicki <matyax@gmail.com>
2023-03-16 08:54:32 +01:00
Ashley Harrison 53186c14a4
Chore: Improve some types (#64675)
* some type fixes

* few more

* more type fixes

* fix the majority of (window as any) calls

* don't make new variable for event

* few more

* MOAR
2023-03-14 09:51:44 +00:00
Matias Chomicki 5ffb28989e
Range splitting: remove hardcoded chunk limit (#64625)
* Range splitting: allow splits larger than 30 days while keeping the fail safe

* Remove limit

* Fix test

* Remove unnecessary try/catch

* Remove unused export
2023-03-13 05:33:01 -04:00
Matias Chomicki ede3e9e5c4
Range Splitting: Process instant queries as an independent query group (#64049)
* Query splitting: enable instant queries

* Range splitting: send instant queries as another request group

* Range splitting: increase grouped splitted requests stability

We were defaulting to the `0` index as the first group for the next request batch, but there was no guarantee that the group `0` had a `.partition` entry for `requestN-1`. Now we find the first defined and use that index as the next starting group.

* Range splitting: update unit test
2023-03-07 07:44:13 -05:00
Matias Chomicki accef84ca5
Range splitting: Call subscriber.next only when there are new results to report (#64171) 2023-03-07 13:05:40 +01:00
Matias Chomicki 40ac0fa14b
Query Splitting: Add support for multiple queries (#63663)
* Range splitting: group metric and logs queries

* Range splitting: intercalate logs and metric queries when > 1 queries

* Range splitting: fix possibly undefined error and remove console log

* Range splitting: update test imports

* Range splitting: add unit tests for multiple queries

* Query splitting: use lodash partition function

* Chore: rename variable

* Chore: attempt to improve readability
2023-03-02 10:32:45 +01:00
Matias Chomicki a0bea04a02
Range splitting: Read errors from the received response and report them (#63368) 2023-02-16 12:39:26 -05:00
Matias Chomicki 0240f4eb45
Loki Range Splitting: Create a new instance for the initial frame (#63361)
* Range splitting: create a new instance of the initial frame

* Chore: rename variable
2023-02-16 12:55:31 +01:00
Gábor Farkas 0ee9d11a91
loki: query splitting: better canceling (#63315)
loki: query splitting: better cancelling
2023-02-13 17:52:30 +01:00
Matias Chomicki 84fdb7f908
Loki Range Splitting: Calculate dynamic maxLines per target based on the current response state (#63248)
* Range splitting: update maxLines for logs queries

* Range splitting: add unit tests for dynamic limit requesting
2023-02-13 07:59:20 +01:00
Matias Chomicki 8cd3d1bfda
Loki Range splitting: Ignore hidden queries (#63250)
Range splitting: ignore hidden queries
2023-02-10 18:19:17 +01:00
Gábor Farkas 27d70819cc
loki: query splitting: split logs queries (#63091)
* loki: calculate logs-chunk-boundaries

* renamed function
2023-02-10 12:01:33 +01:00
Matias Chomicki 94241f6676
Loki Query Splitting: Split queries into sub-queries with smaller time interval (#62767)
* Range splitting: range splitting function

* Range splitting: experiment with 1 hour splits

* Range splitting: reorganize code

* Range splitting: improve code readability and meaning

* Range splitting: add partition limit to prevent infinite loops

* Range splitting: add error handling

* Range splitting: disable for logs queries

* Range splitting: support any arbitrary time splitting + respect original from/to in the partition

* Chore: remove console logs

* Chore: delete unused import

* Range splitting: actually send requests in sequence

* Range splitting: do not split when > 1 query

* Range splitting: combine frames

* Chore: rename function

* split in reverse

* polished reversing

* keep reference to the right frame in the result

* Range splitting: change request state to Streaming

* Range splitting: fix moving only 1 unit of time instead of the provided one

* Chore: change default parameter to timeShift = 1

* Range splitting: do not split for range queqries

* Range splitting: add initial support for log queries

* Range splitting: do not use MutableDataFrame

It has bad performance and it's not required

* Chore: remove unused export

* Query Splitting: move to module

* loki: split: fix off-by-one error (#62966)

loki: split: fix off-by-one loop

* Range splitting: disable for logs volume queries

* Range splitting: combine any number of fields, not just hardcoded 2

* Range splitting: optimize frame-combining function

* Range splitting: further optimize

* Range splitting: combine frame length

* Range splitting: combine stats

* Range splitting: combine stats without assuming the same order

* Query splitting: catch and raise errors

* Range splitting: create feature flag

* Range splitting: implement feature flag

* Range splitting: add unit test for datasource query

* Range splitting: add basic test for runPartitionedQuery

* Range splitting: add unit test for resultLimitReached

* Range splitting: test frame merging

* Chore: fix unit test

---------

Co-authored-by: Sven Grossmann <svennergr@gmail.com>
Co-authored-by: Gábor Farkas <gabor.farkas@gmail.com>
2023-02-09 12:27:02 -05:00