Commit Graph

128 Commits

Author SHA1 Message Date
Martijn van Groningen f9fe7fb315
Disable ilm history in data streams rest qa module. (#61291)
Closes #61273
2020-08-19 09:59:58 +02:00
Martijn van Groningen 8b2c088614
Add tests that simulate new indexing strategy upgrade procedure. (#61082)
Closes #58251
2020-08-18 14:32:28 +02:00
Alan Woodward 3a81b11073
Make MetadataFieldMapper extend ParametrizedFieldMapper (#59847)
This commit cuts over all metadata field mappers to parametrized format.
2020-08-10 17:21:42 +01:00
Martijn van Groningen fb1fc7215f
Added missing data streams tests (#60878)
This tests exist in 7.x branch, but not in master branch.
2020-08-10 13:18:29 +02:00
Martijn van Groningen f3b305aa44
Improve error message for non append-only writes that target data stream (#60809)
Closes #60581
2020-08-10 10:45:50 +02:00
Alan Woodward d6fc439fef
Move mapper validation to the mappers themselves (#60072)
Currently, validation of mappers (checking that cross-references are correct, limits on
field name lengths and object depths, multiple definitions, etc) is performed by the
MapperService. This means that any mapper-specific validation, for example that done
on the CompletionFieldMapper, needs to be called specifically from core server code,
and so we can't add validation to mappers that live in plugins.

This commit reworks the validation framework so that mapper-specific validation is
done on the Mapper itself. Mapper gets a new `validate(MappingLookup)`
method (already present on `MetadataFieldMapper` and now pulled up to the parent
interface), which is called from a new `DocumentMapper.validate()` method. All
the validation code currently living on `MapperService` moves either to individual
mapper implementations (FieldAliasMapper, CompletionFieldMapper) or into
`MappingLookup`, an altered `DocumentFieldMappers` which now knows about 
object fields and can check for duplicate definitions, or into DocumentMapper 
which handles soft limit checks.
2020-08-04 12:19:47 +01:00
Jake Landis 86952d78f4
Cleanup xpack build.gradle (#60554)
This commit does three things:
* Removes all Copyright/license headers for the build.gradle files under x-pack. (implicit Apache license)
* Removes evaluationDependsOn(xpackModule('core')) from build.gradle files under x-pack
* Removes a place holder test in favor of disabling the test task (in the async plugin)
2020-08-03 10:15:12 -05:00
Dan Hermann b459257278
Un-mute data stream REST test (#60120) 2020-08-03 07:43:11 -05:00
Rene Groeschke dd74be0f83
Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
2020-08-03 12:07:41 +02:00
Armin Braun c99cac4e26
Refactor SnapshotsInProgress State Transitions (#60517)
The copy constructors previously used were hard to read and the exact state changes
were not obvious at all.
Refactored those into a number of named constructors instead, added additional assertions
and moved the snapshot abort logic into `SnapshotsInProgress`.
2020-08-03 12:05:36 +02:00
Andrei Dan ac2184c461
Data streams: throw ResourceAlreadyExists exception (#60518)
For consistency reasons (and reducing the overload of IllegalArgumentException)
this changes the exception thrown when trying to create a data stream
that already exists.
2020-07-31 17:42:59 +01:00
James Baiera 0e4f493ac4
Track backing indices in data streams stats from cluster state (#59817)
If shard level results are incomplete in the data streams stats call, it is possible to get inaccurate 
counts of the number of backing indices, despite this data being accurate and available in the 
cluster state.
2020-07-21 13:58:54 -04:00
Lee Hinman ebcf5d525d
Fix retrieving data stream stats for a DS with multiple backing indices (#59806)
* Fix retrieving data stream stats for a DS with multiple backing indices

This API incorrectly had `allowAliasesToMultipleIndices` set to false in the default options for the
request. This changes it from `false` to `true` and enhances a test to exercise the functionality.

Resolves #59802

* Fix test for wording change
2020-07-17 13:52:44 -06:00
Martijn van Groningen 261250d616
Replaced _data_stream_timestamp meta field's 'path' option with 'enabled' option (#59503)
and adjusted exception messages.

Relates to #59076
2020-07-16 21:31:45 +02:00
Dan Hermann 902c1fa80a
Move REST specs for data streams (#59634) 2020-07-16 08:56:58 -05:00
Alan Woodward bf12ac1333
Convert DateFieldMapper to parametrized format (#59429)
This commit makes DateFieldMapper extend ParametrizedFieldMapper,
declaring its parameters explicitly. As well as changes to DateFieldMapper
itself, there are some changes to dynamic mapping code to ensure that
dynamically detected date formats are passed through to new date mapper
builders.
2020-07-16 10:15:56 +01:00
James Baiera 1fb4a5ed31
Remove unneeded rest params from Data Stream Stats (#59575)
This PR removes the expand_wildcards and forbid_closed_indices parameters from the Data 
Streams Stats REST endpoint. These options are required for broadcast requests, but are not 
needed for anything in terms of resolving data streams. Instead, we just set a default set of 
IndicesOptions on the transport request.
2020-07-15 14:24:15 -04:00
Martijn van Groningen 80f238b041
Move data stream transport and rest action to xpack (#59525)
* Actions are moved to xpack core.
* Transport and rest actions are moved the data-streams module.
* Removed data streams methods from Client interface.
* Adjusted tests to use client.execute(...) instead of data stream specific methods.
* only attempt to delete all data streams if xpack is installed in rest tests
* Now that ds apis are in xpack and ESIntegTestCase
no longers deletes all ds, do that in the MlNativeIntegTestCase
class for ml tests.
2020-07-15 10:09:23 +02:00
Martijn van Groningen c27dc5f786
Remove data stream feature flag, (#59504)
so that it can used in the next minor release (7.9.0).

Closes #53100
2020-07-14 22:12:30 +02:00
James Baiera 589bb1f26c
Data Stream Stats API (#58707)
This API reports on statistics important for data streams, including the number of data 
streams, the number of backing indices for those streams, the disk usage for each data 
stream, and the maximum timestamp for each data stream
2020-07-14 14:51:32 -04:00
Dan Hermann b2974aeaab
Reenable BWC tests for data streams (#59483) 2020-07-14 07:23:23 -05:00
Andrei Dan 5609353c5d
Default to @timestamp in composable template datastream definition (#59317)
This makes the data_stream timestamp field specification optional when
defining a composable template.
When there isn't one specified it will default to `@timestamp`.
2020-07-14 11:45:48 +01:00
Martijn van Groningen 7833c3ec1a
Also set system property when running test task. (#59499)
Closes #59488
2020-07-14 10:27:36 +02:00
Andrei Dan 4e72f43d62
Composable templates: add a default mapping for @timestamp (#59244)
This adds a low precendece mapping for the `@timestamp` field with
type `date`.
This will aid with the bootstrapping of data streams as a timestamp
mapping can be omitted when nanos precision is not needed.
2020-07-14 09:19:00 +01:00
Igor Motov 12c61e0d80
Change 7.9.99 -> 7.99.99 in tests (#59469)
Since we most likely going to have 7.10 we should update version
in tests skips to 7.99.99.
2020-07-13 17:26:15 -04:00
Lee Hinman 45da8df79d Fix license header for DataStreamRestIT 2020-07-13 12:58:11 -06:00
Lee Hinman d543c27223
Add telemetery for data streams (#59433)
This commit adds data stream info to the `/_xpack` and `/_xpack/usage` APIs. Currently the usage is
pretty minimal, returning only the number of data streams and the number of indices currently
abstracted by a data stream:

```
  ...
  "data_streams" : {
    "available" : true,
    "enabled" : true,
    "data_streams" : 3,
    "indices_count" : 17
  }
  ...
```
2020-07-13 12:12:22 -06:00
Martijn van Groningen 40b9fd49e0
Make data streams a basic licensed feature. (#59293)
* Create new data-stream xpack module.
* Move TimestampFieldMapper to the new module,
  this results in storing a composable index template
  with data stream definition only to work with default
  distribution. This way data streams can only be used
  with default distribution, since a data stream can
  currently only be created if a matching composable index
  template exists with a data stream definition.
* Renamed `_timestamp` meta field mapper 
   to `_data_stream_timestamp` meta field mapper.
* Add logic to put composable index template api
  to fail if `_data_stream_timestamp` meta field mapper
  isn't registered. So that a more understandable
  error is returned when attempting to store a template
  with data stream definition via the oss distribution.

In a follow up the data stream transport and
rest actions can be moved to the xpack data-stream module.
2020-07-13 11:43:42 +02:00