Commit Graph

90 Commits

Author SHA1 Message Date
Sam 0cf6b94fa2
tsdb/elasticsearch: remove TimeInterval field from tsdb/elasticsearch client (#78264)
remove TimeInterval field from tsdb/elasticsearch client #78187

Co-authored-by: Sam <samucap@users.noreply.github.com>
2023-11-16 14:57:21 +01:00
NikolayTsvetkov ce462e8cd7
Elasticsearch: Added support for calendar_interval in ES date histogram queries (#75459)
* Introduce support for calendar intervals in ES date histogram queries

* Add missing undef type check for ES calendar interval query support
2023-10-09 12:37:38 +02:00
Marcus Efraimsson e4c1a7a141
Tracing: Standardize on otel tracing (#75528) 2023-10-03 14:54:20 +02:00
Marcus Efraimsson 0128d0403f
Tracing: Use tracing.InitializeTracerForTest (#75479) 2023-09-27 09:51:57 +02:00
Ivana Huckova 4f0b31d21b
Elasticsearch: Add tracing to data source (#74750)
* Elasticsearch: Add tracing do data source

* Fix tests

* Address feedback

* Update pkg/tsdb/elasticsearch/response_parser.go

Co-authored-by: Sven Grossmann <sven.grossmann@grafana.com>

* Update pkg/tsdb/elasticsearch/response_parser.go

Co-authored-by: Sven Grossmann <sven.grossmann@grafana.com>

* Track error across both spans

* Add span for decoding of response

* Fix test

* Update setting of errors + fix test

---------

Co-authored-by: Sven Grossmann <sven.grossmann@grafana.com>
2023-09-18 10:49:12 +02:00
Ivana Huckova 49bd93aa2b
Elasticsearch: Adjust logging to follow convention (#74554)
* Elasticsearch: Adjust naming in logging according to convention

* Log response parsing per response

* Update

* Fix logging of errors when no response

* Add path to error loggigng

* Update pkg/tsdb/elasticsearch/response_parser.go
2023-09-07 18:15:24 +02:00
Ivana Huckova 51391a762b
Elasticsearch: Improve backend instrumentation of `QueryData` calls (#74172)
* Elasticsearch: Improve backend instrumentation of QueryData calls

* Add fromAlert

* Fix tests, move logger and context to struct

* Add instrumentation for processing response

* Move log for sending request closer to sending request

* Update

* Fix logging, improved messages, fix printing of queries

* Update log text

* Fix tests

* Fix lint

* Update logging to follow our guidelines

* Remove key-value pairs from logs that are going to be passed from instrumentation

* Update pkg/tsdb/elasticsearch/elasticsearch.go
2023-09-07 13:54:16 +02:00
Ryan McKinley 025b2f3011
Chore: use any rather than interface{} (#74066) 2023-08-30 18:46:47 +03:00
Ivana Huckova a80fd02f95
Elasticsearch: Unify default value for geo hash grid precision across the code to 3 (#73922)
* Unify default value

* Use variable to keep default precisions in sync

* Use default precision variable

* Update precision description

* Update defaultPrecisionString and move

* Be more specific in naming of variabkle

* Revert "Merge remote-tracking branch 'origin' into ivana/es-precision-default-value"

This reverts commit 599f236a77, reversing
changes made to 6742be0c6d.

* Revert wrong merge

* Revert wrong merge with turned off lefthook
2023-08-28 16:38:44 +02:00
Gábor Farkas e17540bdcd
Elasticsearch: Fix using multiple indexes with comma separated string (#71284)
* Revert "Elasticsearch: Use array of strings as index in backend queries (#67276)"

This reverts commit d0ced39847.

* updated tests
2023-07-11 09:47:16 +02:00
Gábor Farkas fcef387151
Elasticsearch: Handle no-index case in backend mode (#68534)
* elastic: backend migration: fix no-index case

* improved code
2023-05-17 15:24:46 +02:00
Gábor Farkas 6189f48be7
elastic: more tests (#68533)
* elastic: more tests

* fix lint error
2023-05-16 17:09:09 +02:00
Ivana Huckova 95cf598423
Elasticsearch: Fix processing of logs with not-recognized time format (#67767)
* Elasticsearch: Fix parsing of invalid time

* Fix lint

* Add more test data to data.js

* Add tests

* Fix lint

* Update pkg/tsdb/elasticsearch/client/search_request.go
2023-05-04 19:33:00 +02:00
Ivana Huckova d0ced39847
Elasticsearch: Use array of strings as index in backend queries (#67276)
Elasticsearch: Use array of strings as indice in backend queries
2023-04-26 16:07:15 +02:00
Ivana Huckova abc11d1dcf
Elasticsearch: Run log context queries through backend (#65805)
* Elasticsearch: Run context queries trough backend

* Fix typing

* Update

* Add possibility to run context query torugh backend and frontend

* Correctly sort
2023-04-05 18:32:02 +02:00
Gábor Farkas 0cff917f2a
Elasticsearch: Removed reference to obsolete esVersion value (#65415)
* elastic: removed reference to obsolete esVersion value

* removed unused code

* cleaned up tests
2023-03-28 17:04:56 +03:00
Ivana Huckova e8bb894a02
Elasticsearch: Remove GetMinInterval method that is not used anymore (#65203) 2023-03-23 09:36:37 +01:00
Ivana Huckova bd0fa79542
Elasticsearch: Add log highlight processing on backend (#63924)
* Elasticsearch: Add highlight for logs

* Fix running of queries trough backend only when shouldRunTroughBackend

* Fix incorrect json in test string

* Address feedback

* Keep order of words same
2023-03-09 13:34:53 +01:00
Ivana Huckova d258c8ef8a
Elasticsearch: Implement processing of logs query results in backend (#63647)
* Elasticsearch: Add processing of logs query to backend

* Add and fix tests

* Add snapshot tests

* Fix test in ES client

* Small updates, remove redundant logic

* Refactor setPreferredVisType to improve readability
2023-03-01 05:50:56 -05:00
Serge Zaitsev 0bf4093005
Chore: Fix goimports grouping (#62428)
* fix goimports order

* fix goimports order
2023-01-30 08:50:27 +00:00
Gábor Farkas 88119ad6c3
Elasticsearch: Support nested aggregation (#62301)
* Add nested query support

* Add nested support for alerts

* update nested aggregation

* cleanup types

* Add nested integration test

* Move aggdef to nested

* fixed merge conflict

* fixed lint warning

* mark nested-mode experimental

---------

Co-authored-by: Ethan Gallant <ethan@ziax.com>
Co-authored-by: Ethan J. Gallant <ethan.gallant@acquia.com>
2023-01-27 16:18:36 +01:00
Ivana Huckova 772e8cbf60
Elasticsearch: Use interval provided by data request in backend (#60480)
* Elasticsearch: Remove interval caulation and use interval provises by grafana

* Remove redundant code

* Adjust snapshot tests

* Update test

* Fix lint
2023-01-05 12:26:27 +01:00
Ivana Huckova a1ef3d77f2
Elasticsearch: Use constants for default values and types (#60673) 2022-12-22 15:08:15 +01:00
Ivana Huckova d3ef86bd90
Elasticsearch: Add query building for log queries (#60182)
* Elasticsearch: Fix ordering in raw_document and add logic for raw_data

* Add comments

* Fix raw data request to use correct timefield

* Fix linting

* Add raw data as metric type

* Fix linting

* Elasticsearch: Add defaults for log query

* Add higlight

* Fix lint

* Add snapshot test

* Implement correct query for logs

* Update

* Adjust naming and comments

* Fix lint

* Remove ifs
2022-12-14 13:56:09 +01:00
Ivana Huckova 0973501233
Elasticsearch: Add query building for raw data and raw document queries to backend (#59741)
* Elasticsearch: Fix ordering in raw_document and add logic for raw_data

* Add comments

* Fix raw data request to use correct timefield

* Fix linting

* Add raw data as metric type

* Fix linting

* Hopefully fix lint
2022-12-13 13:11:44 +01:00
Gábor Farkas b8b2de1ac4
elastic: remove unused debug-code (#59712)
* elastic: remove unused debug-code

* removed unused structs
2022-12-05 04:21:15 -05:00
Yuriy Tseretyan d9c40ca41e
Chore: Update sqleng, elasticsearch, tempo and opentsdb plugins to support contextual logs. (#57777)
* make sql engine use pick log context for logs
* update tempo to get log context
* update opentsdb to use log context
* update es client to use log context
2022-11-02 10:03:50 -04:00
Gábor Farkas 2efd7fa481
Elasticsearch: Reuse http client in the backend (#55172)
* elastic: backend: reuse http client

* fixed lint error
2022-09-26 14:27:46 +02:00
Gábor Farkas 6dcc8534f7
elastic: simplify unit test (#55168) 2022-09-14 08:59:35 -04:00
Gábor Farkas 35253a909d
Elasticsearch: remove already-disabled code (#52932)
* elastic: removed unused code from go-part

* elastic: removed unused code from js-part
2022-08-22 16:25:20 +02:00
Jo 062d255124
Handle ioutil deprecations (#53526)
* replace ioutil.ReadFile -> os.ReadFile

* replace ioutil.ReadAll -> io.ReadAll

* replace ioutil.TempFile -> os.CreateTemp

* replace ioutil.NopCloser -> io.NopCloser

* replace ioutil.WriteFile -> os.WriteFile

* replace ioutil.TempDir -> os.MkdirTemp

* replace ioutil.Discard -> io.Discard
2022-08-10 15:37:51 +02:00
Gábor Farkas a2eb4e85e5
elasticsearch: always use fixed_interval (#50297) 2022-06-13 10:28:29 +02:00
Kristin Laemmert 8f6877e12a
chore: remove all remaining uses of golang.org/x/net/context (#47564)
* chore: remove all remaining uses of  golang.org/x/net/context

This PR finishes the work started in #47532, replacing all calls to that package with the stdlib context and using http.NewRequestWithContext to include the context where necessary.

Bonus: small formatting fixes to goimports in these files.

closes #44178

* tweak: use context.Background in favor of TODO for tests
2022-04-11 14:20:10 -04:00
Piotr Jamróz 9fb8339f87
Elastic: Allow using long/int as date field for alerts (#44027)
* Use integers for time range filter

Previously it was passed as a string which is automatically converted by Elastic to a number only if the field type is "date". For other types (e.g. "long") such conversion doesn't work. In theory "date" could be passed as a formatted string but we don't use it this way and always pass it as a number so it is safe to always pass numbers, not strings.

* Fix time_series_query_test

* Retrigger build
2022-01-17 15:45:09 +01:00
Giordano Ricci 1b99d88337
Elasticsearch: Add support for Elasticsearch 8.0 (Beta) (#41729)
* use fixed_interval in date_histogram

* Add 8.0 to available versions in datasource settings

* Remove moving_avg from available metric aggregations

* Add ES8 devenv

* Update public/app/plugins/datasource/elasticsearch/components/QueryEditor/MetricAggregationsEditor/utils.ts

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>

* Add FE tests

* Add BE test

* fix FE test

* fix BE test

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
2021-11-17 20:30:53 +01:00
Giordano Ricci 7b7c193551
Elasticsearch: Add time zone setting to Date Histogram aggregation (#40882)
* TimeZonePicker: Allow specifying internal timezones to display

* Elasticsearch: Add time zone setting to Date Histogram aggregation

* rename time_zone to timeZone

* Add tests

* fix typo

* Update packages/grafana-data/src/datetime/timezones.ts

Co-authored-by: Chris Cowan <chris@chriscowan.us>

Co-authored-by: Chris Cowan <chris@chriscowan.us>
2021-11-02 16:18:39 +00:00
Will Browne 76d88a6cec
change log level to debug (#41102) 2021-11-01 09:00:00 +00:00
Serge Zaitsev a45861298f
Chore: Refactor GoConvey in tsdb/es/client (#40846) 2021-10-26 13:15:47 +02:00
ying-jeanne c989f0fbbe
SQL: Migrate to use SDK contracts (#36635)
* convert SQLs to use sdk contracts

* make draft

* postgres

* intermedia

* get datasourceinfo filled at the beginning of the service

* move the interval into package because of cyclict  import and fix all postgres tests

* fix mysql test

* fix mssql

* fix the test for pr https://github.com/grafana/grafana/issues/35839

* fix some issue about intervalv2 package

* update sql test

* wire migration for SQLs

* add sqls to the background process

* make it register instead of register and start

* revert formatting

* fix tests

* fix linter

* remove integration test

* Postgres test fix

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2021-09-07 09:35:37 +02:00
Dimitris Sotirakis 0df1b33d71
Plugins: Migrate Elasticsearch to backend plugin SDK (#36132)
* Migrate Elasticsearch to backend plugin SDK

* Fix linting

* Move away from Convey!

* Rebase commit

* Small logger fix

* Fixes according to reviewer's comments

* Fixes according to reviewer's comments

* Fixes according to reviewer's comments

* More cleanup

* Move things around - small refactoring

* Fix typo

* Update calculator - add tests

* Fixes according to reviewer's comments
2021-07-15 16:45:59 +02:00
Giordano Ricci f6b83a4f47
Elasticsearch: Add frozen indices search support (#36018)
* Revert "Revert "Elasticsearch: add frozen indices search support (#27472)" (#27726)"

This reverts commit 4c7131425b.

* Make label width a bit more consistent

* Add documentation for X-Pack & Frozen Indices support in Elasticsearch

* Change UI & docs casing

* create default empty dataframe

* Remove backticks and simplify regex

* small doc improvement
2021-07-15 15:52:02 +02:00
Chris Cowan f580c9149c
Elasticsearch: Add Top Metrics Aggregation and X-Pack support (#33041)
* Elasticsearch: Add Top Metrics Aggregation

* Adding support for non-timeseries visualizations

* removing console.logs

* restoring loadOptions type

* Honor xpack setting

* Adding test for elastic_response

* adding test for query builder

* Adding support of alerting

* Fixing separator spelling

* Fixing linting issues

* attempting to reduce cyclomatic complexity

* Adding elastic77 Docker block

* Update public/app/plugins/datasource/elasticsearch/components/QueryEditor/MetricAggregationsEditor/MetricEditor.test.tsx

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* refactoring MetricsEditor tests

* Fixing typo

* Change getFields type & move TopMetrics to a separate component

* Fix SegmentAsync styles in TopMetrics Settings

* Fix field types for TopMetrics

* WIP

* Refactoring client side to support multiple top metrics

* Adding tests and finishing go implimentation

* removing fmt lib from debugging

* fixing tests

* reducing the cyclomatic complexity

* Update public/app/plugins/datasource/elasticsearch/elastic_response.ts

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* Update public/app/plugins/datasource/elasticsearch/hooks/useFields.ts

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>

* Checking for possible nil value

* Fixing types

* fix fake-data-gen param

* fix useFields hook

* Removing aggregateBy and size

* Fixing go tests

* Fixing TS tests

* fixing tests

* Fixes

* Remove date from top_metrics fields

* Restore previous formatting

* Update pkg/tsdb/elasticsearch/client/models.go

Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>

* Update pkg/tsdb/elasticsearch/client/models.go

Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>

* Fix code review comments on processTopMetricValue

* Remove underscore from variable names

* Remove intermediate array definition

* Refactor test to use testify

Co-authored-by: Giordano Ricci <grdnricci@gmail.com>
Co-authored-by: Elfo404 <me@giordanoricci.com>
Co-authored-by: Dimitris Sotirakis <dimitrios.sotirakis@grafana.com>
2021-06-04 11:07:59 +01:00
wangjian 530072fce7
Elasticsearch: fix NewClient not passing httpClientProvider to client impl (#34539)
* Update client.go

fix missing "httpClientProvider" in NewClient

* Update client.go
2021-05-24 14:45:29 +01:00
Marcus Efraimsson 348e76fc8e
Datasource: Shared HTTP client provider for core backend data sources and any data source using the data source proxy (#33439)
Uses new httpclient package from grafana-plugin-sdk-go introduced 
via grafana/grafana-plugin-sdk-go#328. 
Replaces the GetHTTPClient, GetTransport, GetTLSConfig methods defined 
on DataSource model.
Longer-term the goal is to migrate core HTTP backend data sources to use the 
SDK contracts and using httpclient.Provider for creating HTTP clients and such.

Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
2021-05-19 23:53:41 +02:00
Giordano Ricci e98a8bd11b
Elasticsearch: use semver strings to identify ES version (#33646)
* Elasticsearch: use proper semver strings to identify ES version

* Update BE & tests

* refactor BE tests

* refactor isValidOption check

* update test

* Update pkg/tsdb/elasticsearch/client/client.go

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>

* Update pkg/tsdb/elasticsearch/client/search_request_test.go

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>

* Remove leftover FIXME comment

* Add new test cases for new version format

* Docs: add documentation about version dropdown

* Update docs/sources/datasources/elasticsearch.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update docs/sources/datasources/elasticsearch.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update docs/sources/datasources/elasticsearch.md

Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>

* Update provisioning documentation

Co-authored-by: Piotr Jamróz <pm.jamroz@gmail.com>
Co-authored-by: achatterjee-grafana <70489351+achatterjee-grafana@users.noreply.github.com>
2021-05-11 09:44:00 +01:00
Giordano Ricci cdb4785496
Elasticsearch: Allow omitting field when metric supports inline script (#32839)
* Elasticsearch: Allow omitting field when metric supports inline script

* Add tests for MetricEditor to show a None option

* Add tests for useFields hook

* Alerting: allow elasticsearch metrics without field
2021-04-14 15:18:06 +01:00
Giordano Ricci f7b408f99f
Elasticsearch: use application/x-ndjson content type for multisearch requests (#32282) 2021-03-29 23:41:45 +01:00
Arve Knudsen b79e61656a
Introduce TSDB service (#31520)
* Introduce TSDB service

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

Co-authored-by: Erik Sundell <erik.sundell87@gmail.com>
Co-authored-by: Will Browne <will.browne@grafana.com>
Co-authored-by: Torkel Ödegaard <torkel@grafana.org>
Co-authored-by: Will Browne <wbrowne@users.noreply.github.com>
Co-authored-by: Zoltán Bedi <zoltan.bedi@gmail.com>
2021-03-08 07:02:49 +01:00
Emil Hessman b2b3a603e8
Chore: Rewrite elasticsearch client test to standard library (#30093) 2021-01-07 09:35:56 +01:00
Arve Knudsen c2cad26ca9
Chore: Disable default golangci-lint filter (#29751)
* Disable default golangci-lint filter

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Chore: Fix linter warnings

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
2020-12-15 09:32:06 +01:00