We have to filter the intermediate metadata here so that indices
and datastreams in it are consistent. Otherwise we throw an NPE
in production when restoring global state but not all datastreams
and trip an assertion in tests.
Closes#68357
There were a number of issues around data streams rolling over during
a snapshot that are fixed in this PR:
1. If partial snapshots cause a data stream to not be fully snapshotted
because an index gets concurrently deleted we must not add it to the
resulting cluster state (otherwise we trip assertions on either snapshot itself
or a later restore, depending on whether or not the complete global state is
snapshotted).
2. When a non-partial snapshot is running we must not allow a datastream rollover,
otherwise we cannot finish the datastream snapshot correctly because the newly
created write index has not become part of the snapshot.
3. If any part of a datastream fails snapshotting or gets concurrently deleted during
a partial snapshot we must not add it to `SnapshotInfo` same as we do not add concurrently
deleted indices to it.
Index aliases are not allowed to refer to backing indices of data streams.
Adding an alias that points to a backing index results into a validation error.
However when defining aliases and a data stream definition on an index template,
it is still possible to for aliases that refer to backing indices to be created
when a data stream or new backing index is created.
This change add additional validation that prevents defining aliases and data
stream definition together in a composable index template or component templates
that are referred by an composable index template. This should fix the mentioned
bug.
This checks only enables this validation when adding/updating a composable index
and component template. There may be templates in the wild that have both
data stream and aliases definitions, so we currently can't fail cluster states
that contain aliases that refer to backing indices. So instead a warning header
is emitted.
Closes#67730
We recently rewrote the `TransportRolloverAction` code to encapsulate the rollover into a single
cluster state. This is great, but we can do a bit better when there are concurrent rollover requests
that occur, specifically, we should meet both of the following criteria:
- Multiple concurrent unconditional rollovers should generate multiple rollovers without any concurrent
modification exceptions.
- Multiple concurrent rollovers with conditions should roll over exactly once, assuming the
condition is met only once
This commit changes the `TransportRolloverAction` code to reach these goals. It does it by treating
the cluster state submittion as a pseudo-"synchronized" block. This means that we calculate the
source, destination, and conditions met *prior* to performing the cluster state update. If the
conditions are met, we invoke a cluster state update. Inside the cluster state update (serialized,
so essentially inside our "synchronized" block) we recalculate the source, destination, and
contitions met, and only proceed with the rollover if the conditions on met.
As a byproduct of this, it also fixes issues where rollover could happen but the response returned
incorrect information such as conditions that weren't actually met or an incorrect
source/destination index.
I've also tried to document (with comments) the thought process and how this code works for future
generations.
This also adds a test with multiple threads all trying to invoke a rollover that ensures that only a
single rollover happens in the case conditions are specified.
Resolves#67836
Resloves #64921
We decided to rename `QueryShardContext` to clarify that it supports all parts
of search request execution. Before there was confusion over whether it should
only be used for building queries, or maybe only used in the query phase. This
PR also updates the javadocs.
Closes#64740.
A number of metadata field mapper tests are still extending ESSingleNodeTestCase.
This commit changes them to use MapperServiceTestCase instead, and adds a new
base class for those metadata fields that have parameters subject to conflict checks.
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.
This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.
It also addresses a number of tests that would fail in approved only mode
Mainly:
Tests that use PBKDF2 with a password less than 112 bits (14char). We
elected to change the passwords used everywhere to be at least 14
characters long instead of mandating
the use of pbkdf2_stretch because both pbkdf2 and
pbkdf2_stretch are supported and allowed in fips mode and it makes sense
to test with both. We could possibly figure out the password algorithm used
for each test and adjust password length accordingly only for pbkdf2 but
there is little value in that. It's good practice to use strong passwords so if
our docs and tests use longer passwords, then it's for the best. The approach
is brittle as there is no guarantee that the next test that will be added won't
use a short password, so we add some testing documentation too.
This leaves us with a possible coverage gap since we do support passwords
as short as 6 characters but we only test with > 14 chars but the
validation itself was not tested even before. Tests can be added in a followup,
outside of fips related context.
Tests that use a PKCS12 keystore and were not already muted.
Tests that depend on running test clusters with a basic license or
using the OSS distribution as FIPS 140 support is not available in
neither of these.
Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
When a data stream is being auto followed then a rollover in a local cluster can break auto following,
if the local cluster performs a rollover then it creates a new write index and if then later the remote
cluster rolls over as well then that new write index can't be replicated, because it has the same name
as in the write index in the local cluster, which was created earlier.
If a data stream is managed by ccr, then the local cluster should not do a rollover for those data streams.
The data stream should be rolled over in the remote cluster and that change should replicate to the local
cluster. Performing a rollover in the local cluster is an operation that the data stream support in ccr should
perform.
To protect against rolling over a replicated data stream, this PR adds a replicate field to DataStream class.
The rollover api will fail with an error in case a data stream is being rolled over and the targeted data stream is
a replicated data stream. When the put follow api creates a data stream in the local cluster then the replicate flag
is set to true. There should be a way to turn a replicated data stream into a regular data stream when for example
during disaster recovery. The newly added api in this pr (promote data stream api) is doing that. After a replicated
data stream is promoted to a regular data stream then the local data stream can be rolled over, so that the new
write index is no longer a follower index. Also if the put follow api is attempting to update this data stream
(for example to attempt to resume auto following) then that with fail, because the data stream is no longer a
replicated data stream.
Today with time based indices behind an alias, the is_write_index property isn't replicated from remote cluster
to the local cluster, so when attempting to rollover the alias in the local cluster the rollover fails, because the
alias doesn't have a write index. The added replicated field in the DataStream class and added validation
achieve the same kind of protection, but in a more robust way.
A followup from #61993.
This PR simplifies how the document source is passed to each fetch subphase. A summary of the strategy:
* For each document, we try to eagerly load the source and store it on `HitContext`. Most subphases that access source, like source filtering and highlighting, use `HitContext`. For nested hits, we filter the parent source and also store this source on `HitContext`.
* Only for non-nested documents, we also store the loaded source on `QueryShardContext#lookup`. This allows subphases that access source through `SearchLookup` to use the pre-loaded source when possible. This is now a common occurrence, since runtime fields are supported in the 'fields' option and may soon be supported in highlighting.
There is no longer a special `SearchLookup` just for the fetch phase. This was not necessary and was mostly caused by a misunderstanding of how `QueryShardContext` should be used.
Addresses #62511.
Metadata fields are always instantiated at the root of a document,
so they don't need to take the ContentPath in their build() methods.
Also converts a couple of metadata parsers from Configurable to
Fixed, as they don't have any parameters.
Mapper.BuilderContext is a simple wrapper around two objects, some
IndexSettings and a ContentPath. The IndexSettings are the same as
those provided in the ParserContext, so we can simplify things here
by removing them and just passing ContentPath directly to
Mapper.Builder#build()
The signature of MappedFieldType#valueFetcher requires MapperService as an argument which is unfortunate as that is one of the reasons why FetchContext exposes the whole MapperService.
Such use of MapperService can be replaced with exposing the QueryShardContext which encapsulates the MapperService.
This commit adds support data stream support to CCR's auto following by making the following changes:
* When the auto follow coordinator iterates over the candidate indices to follow,
the auto follow coordinator also checks whether the index is part of a data stream and
if the name of data stream also matches with the auto follow pattern then the index
will be auto followed.
* When following an index, the put follow api also checks whether that index is part
of a data stream and if so then also replicates the data stream definition to the
local cluster.
* In order for the follow index api to determine whether an index is part of a data
stream, the cluster state api was modified to also fetch the data stream definition
of the cluster state if only the state is queried for specific indices.
When a data stream is auto followed, only new backing indices are auto followed.
This is in line with how time based indices patterns are replicated today. This
means that the data stream isn't copied 1 to 1 into the local cluster. The local
cluster's data stream definition contains the same name, timestamp field and
generation, but the list of backing indices may be different (depending on when
a data stream was auto followed).
Closes#56259
Now that all our FieldMapper implementations extend ParametrizedFieldMapper,
we can collapse the two classes together, and remove a load of cruft from
FieldMapper that is unused. In particular:
* we no longer need the lucene FieldType field on FieldMapper
* we no longer use clone() for merging, so we can remove it from all impls
* the serialization code in FieldMapper that assumes we're looking at text fields can go
It's confusing and slightly error prone (see #64116) to handle the timeouts
via overrides but the priority via a field. This simplifies the code to to avoid future
issues and save over 100 LOC.
Also this fixes a bug in `TransportVotingConfigExclusionsAction` where trying to instantiate a time value with a negative time could throw and unexpected exception and as a result leak a listener.
Composable index template may hold custom metadata. This change adds behaviour that
when a data stream gets created the custom metadata of the matching composable index
template is copied to new data stream. The get data stream api can then be used to
view the custom metadata.
Example:
```
PUT /_index_template/my-logs-template
{
"index_patterns": [ "logs-*" ],
"data_stream": { },
"template": {
"settings": {
"index.number_of_replicas": 0
}
},
"_meta": {
"managed": true
}
}
PUT /_data_stream/logs-myapp
GET /_data_stream
```
The get data stream api then yields the following response:
```
{
"data_streams": [
{
"name": "logs-myapp",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-logs-myapp-000001",
"index_uuid": "3UaBxM3mQXuHR6qx0IDVCw"
}
],
"generation": 1,
"_meta": {
"managed": true
},
"status": "GREEN",
"template": "my-logs-template"
}
]
}
```
Closes#59195
Closes#20640.
This PR introduces a new parameter to v2 templates, `allow_auto_create`,
which allows templates to override the cluster setting `auto_create_index`.
Notes:
* `AutoCreateIndex` now looks for a matching v2 template, and if its
`allow_auto_create` setting is true, it overrides the usual logic.
* `TransportBulkAction` previously used `AutoCreateIndex` to check
whether missing indices should be created. We now rely on
`AutoCreateAction`, which was already differentiating between creating
indices and creating data streams. I've updated `AutoCreateAction` to
use `AutoCreateIndex`. Data streams are also influenced by
`allow_auto_create`, in that their default auto-create behaviour can
be disabled with this setting.
* Most of the Java file changes are due to introducing an extra
constructor parameter to `ComposableIndexTemplate`.
* I've added the new setting to various x-pack templates
* I added a YAML test to check that watches can be created even when
`auto_create_index` is `false`.
MapperService allows to retrieve its internal DocumentMapperParser instance. Such method is only used in tests, and always to parse mappings which is already exposed by MapperService through a specific parse method.
This commit removes the getter for DocumentMapperParser from MapperService in favour of calling MapperService#parse
* Removing some more duplication and redundant logic.
* Aligning all timeouts to 30s (60s or even 10 minute timeouts should be unnecessary, if they aren't we should figure out why)
* Remove some usage of `actionGet()` in tests (it's just evil to suppress the stack-trace)
`DocumentMapper` holds on to a reference of `MapperService` so that its `merge` method, which creates and return a new `DocumentMapper` instance, can provide it to its own constructor. On the other hand `MapperService` has a volatile reference to `DocumentMapper`, which gets updated by calling `merge`.
Given that only three components out of `MapperService` are needed, `DocumentMapper` can instead hold on to those specific objects: `IndexSettings`, `DocumentMapperParser` and `IndexAnalyzers`.
1. It is confusing and unnecessary to handle response deserialization via inheritance
and request deserialization via composition. Consistently using composition saves
hundreds of lines of code and as a matter of fact also removes some indirection in
transport master node action since we pass a reader down to the response handling anyway.
2. Defining `executor` via inheritance but then assuming the return of the method is a constant
is confusing and again not in line with how we handle the `executor` definition for other transport
actions so this was simplified away as well.
Somewhat relates to the dry-up in #63335
1. `AcknowledgedResponse` should really be two singletons to make things clearer and save some objects and code size.
2. We were duplicating reading this type over and over in a bunch of transport master node actions so I dried that up
This PR adds deprecation warnings when accessing System Indices via the REST layer. At this time, these warnings are only enabled for Snapshot builds by default, to allow projects external to Elasticsearch additional time to adjust their access patterns.
Deprecation warnings will be triggered by all REST requests which access registered System Indices, except for purpose-specific APIs which access System Indices as an implementation detail a few specific APIs which will continue to allow access to system indices by default:
- `GET _cluster/health`
- `GET {index}/_recovery`
- `GET _cluster/allocation/explain`
- `GET _cluster/state`
- `POST _cluster/reroute`
- `GET {index}/_stats`
- `GET {index}/_segments`
- `GET {index}/_shard_stores`
- `GET _cat/[indices,aliases,health,recovery,shards,segments]`
Deprecation warnings for accessing system indices take the form:
```
this request accesses system indices: [.some_system_index], but in a future major version, direct access to system indices will be prevented by default
```
For runtime fields, we will want to do all search-time interaction with
a field definition via a MappedFieldType, rather than a FieldMapper, to
avoid interfering with the logic of document parsing. Currently, fetching
values for runtime scripts and for building top hits responses need to
call a method on FieldMapper. This commit moves this method to
MappedFieldType, incidentally simplifying the current call sites and freeing
us up to implement runtime fields as pure MappedFieldType objects.
The _index field is a special field that allows using queries against the name of an index or alias.
Data stream names were not included, this pr fixes that by changing SearchIndexNameMatcher
(which used via IndexFieldMapper) to also include data streams.
Today if a data stream is auto created, but an index with same name as the
first backing index already exists then internally that error is ignored,
which then result that later in the execution of a bulk request, the
bulk item fails due to that the data stream hasn't been auto created.
This situation can only occur if an index with same is created that
will be the backing index of a data stream prior to the creation
of the data stream.
Co-authored-by: Dan Hermann <danhermann@users.noreply.github.com>
The bulk api cache the resolved concrete indices when resolving the user provided
index name into the actual index name. The validation that prevents write ops other
than create from being executed in a data stream was only performed if the result
wasn't cached. In case of cached resolvings, the validation never occurs.
The validation would be skipped for all bulk items for a data stream after a create
operation for that same data stream. This commit ensures that the validation is always
performed for all bulk items (whether the concrete index resolution has been cached or
not cached).
Closes#62762
This commit adds the `index.routing.allocation.prefer._tier` setting to the
`DataTierAllocationDecider`. This special-purpose allocation setting lets a user specify a
preference-based list of tiers for an index to be assigned to. For example, if the setting were set
to:
```
"index.routing.allocation.prefer._tier": "data_hot,data_warm,data_content"
```
If the cluster contains any nodes with the `data_hot` role, the decider will only allow them to be
allocated on the `data_hot` node(s). If there are no `data_hot` nodes, but there are `data_warm` and
`data_content` nodes, then the index will be allowed to be allocated on `data_warm` nodes.
This allows us to specify an index's preference for tier(s) without causing the index to be
unassigned if no nodes of a preferred tier are available.
Subsequent work will change the ILM migration to make additional use of this setting.
Relates to #60848
* Configurable metadata field mappers in the _source
* Changes to support metadata fields in _source
Added test testDocumentContainsAllowedMetadataField()
* Merged DocumentParserTests from master
Fixed broken tests
* Handle non string values
* Allow metadata fields to parse values/objects/arrays/null
* Removed MetadataFieldMapper.isAllowedInSource() method
Delegated this functionality to MetadataFieldMapper.parse()
* Fixed bug that caused tests to break
* Cleanup parsing for existing metadata fields
* Cleanup parsing for existing metadata fields
* Remove doParse() method
* Fix broken test
* Lookup metadata mapper by name
Instead of linear scan
This commit changes the default allocation on the "hot" tier to allocating the newly created index
to the "hot" tier if it is part of a new or existing data stream, and to the "content" tier if it is
not part of a data stream.
Overriding any of the index.routing.allocation.(include|exclude|require).* settings continues to
cause the initial allocation not to be set (no change in behavior).
Relates to #60848
ShardClusterSnapshotRestoreIT is confusing as we already have a
very complete SharedClusterSnapshotRestoreIT test suite. This
commit removes ShardClusterSnapshotRestoreIT and folds its
unique test in DataStreamsSnapshotsIT.
Return a 404 http status code when attempting to delete a non existing data stream.
However only return a 404 when targeting a data stream without any wildcards.
Closes#62022
This commit removes `integTest` task from all es-plugins.
Most relevant projects have been converted to use yamlRestTest, javaRestTest,
or internalClusterTest in prior PRs.
A few projects needed to be adjusted to allow complete removal of this task
* x-pack/plugin - converted to use yamlRestTest and javaRestTest
* plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task
* qa/die-with-dignity - convert to javaRestTest
* x-pack/qa/security-example-spi-extension - convert to javaRestTest
* multiple projects - remove the integTest.enabled = false (yay!)
related: #61802
related: #60630
related: #59444
related: #59089
related: #56841
related: #59939
related: #55896
Moving the data stream yaml tests to xpack plugin module has the following benefits:
* The tests are ran both with security enabled (as part of xpack/plugin integTest)
and disabled (as part of xpack/plugin/data-stream/qa/rest integTest).
* and running the tests in mixed cluster qa environment.
For 1/2 the plugins in x-pack, the integTest
task is now a no-op and all of the tests are now executed via a test,
yamlRestTest, javaRestTest, or internalClusterTest.
This includes the following projects:
async-search, autoscaling, ccr, enrich, eql, frozen-indicies,
data-streams, graph, ilm, mapper-constant-keyword, mapper-flattened, ml
A few of the more specialized qa projects within these plugins
have not been changed with this PR due to additional complexity which should
be addressed separately.
A follow up PR will address the remaining x-pack plugins (this PR is big enough as-is).
related: #61802
related: #56841
related: #59939
related: #55896
Currently, validation of mappers (checking that cross-references are correct, limits on
field name lengths and object depths, multiple definitions, etc) is performed by the
MapperService. This means that any mapper-specific validation, for example that done
on the CompletionFieldMapper, needs to be called specifically from core server code,
and so we can't add validation to mappers that live in plugins.
This commit reworks the validation framework so that mapper-specific validation is
done on the Mapper itself. Mapper gets a new `validate(MappingLookup)`
method (already present on `MetadataFieldMapper` and now pulled up to the parent
interface), which is called from a new `DocumentMapper.validate()` method. All
the validation code currently living on `MapperService` moves either to individual
mapper implementations (FieldAliasMapper, CompletionFieldMapper) or into
`MappingLookup`, an altered `DocumentFieldMappers` which now knows about
object fields and can check for duplicate definitions, or into DocumentMapper
which handles soft limit checks.
This commit does three things:
* Removes all Copyright/license headers for the build.gradle files under x-pack. (implicit Apache license)
* Removes evaluationDependsOn(xpackModule('core')) from build.gradle files under x-pack
* Removes a place holder test in favor of disabling the test task (in the async plugin)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
The copy constructors previously used were hard to read and the exact state changes
were not obvious at all.
Refactored those into a number of named constructors instead, added additional assertions
and moved the snapshot abort logic into `SnapshotsInProgress`.
For consistency reasons (and reducing the overload of IllegalArgumentException)
this changes the exception thrown when trying to create a data stream
that already exists.
If shard level results are incomplete in the data streams stats call, it is possible to get inaccurate
counts of the number of backing indices, despite this data being accurate and available in the
cluster state.
* Fix retrieving data stream stats for a DS with multiple backing indices
This API incorrectly had `allowAliasesToMultipleIndices` set to false in the default options for the
request. This changes it from `false` to `true` and enhances a test to exercise the functionality.
Resolves#59802
* Fix test for wording change
This commit makes DateFieldMapper extend ParametrizedFieldMapper,
declaring its parameters explicitly. As well as changes to DateFieldMapper
itself, there are some changes to dynamic mapping code to ensure that
dynamically detected date formats are passed through to new date mapper
builders.
This PR removes the expand_wildcards and forbid_closed_indices parameters from the Data
Streams Stats REST endpoint. These options are required for broadcast requests, but are not
needed for anything in terms of resolving data streams. Instead, we just set a default set of
IndicesOptions on the transport request.
* Actions are moved to xpack core.
* Transport and rest actions are moved the data-streams module.
* Removed data streams methods from Client interface.
* Adjusted tests to use client.execute(...) instead of data stream specific methods.
* only attempt to delete all data streams if xpack is installed in rest tests
* Now that ds apis are in xpack and ESIntegTestCase
no longers deletes all ds, do that in the MlNativeIntegTestCase
class for ml tests.
This API reports on statistics important for data streams, including the number of data
streams, the number of backing indices for those streams, the disk usage for each data
stream, and the maximum timestamp for each data stream
This makes the data_stream timestamp field specification optional when
defining a composable template.
When there isn't one specified it will default to `@timestamp`.
This adds a low precendece mapping for the `@timestamp` field with
type `date`.
This will aid with the bootstrapping of data streams as a timestamp
mapping can be omitted when nanos precision is not needed.
This commit adds data stream info to the `/_xpack` and `/_xpack/usage` APIs. Currently the usage is
pretty minimal, returning only the number of data streams and the number of indices currently
abstracted by a data stream:
```
...
"data_streams" : {
"available" : true,
"enabled" : true,
"data_streams" : 3,
"indices_count" : 17
}
...
```
* Create new data-stream xpack module.
* Move TimestampFieldMapper to the new module,
this results in storing a composable index template
with data stream definition only to work with default
distribution. This way data streams can only be used
with default distribution, since a data stream can
currently only be created if a matching composable index
template exists with a data stream definition.
* Renamed `_timestamp` meta field mapper
to `_data_stream_timestamp` meta field mapper.
* Add logic to put composable index template api
to fail if `_data_stream_timestamp` meta field mapper
isn't registered. So that a more understandable
error is returned when attempting to store a template
with data stream definition via the oss distribution.
In a follow up the data stream transport and
rest actions can be moved to the xpack data-stream module.
Armin Braun
Martijn van Groningen
Lee Hinman
Julie Tibshirani
Alan Woodward
Ioannis Kakavas
Dan Hermann
Rene Groeschke
Przemko Robakowski
Luca Cavanna
Rory Hunter
Gordon Brown
Andrei Stefan
Christos Soulios
Tanguy Leroux
Yannick Welsch
Yang Wang
Jake Landis
Andrei Dan
James Baiera
Igor Motov
Lee Hinman