When create template v2. mappings will add _doc. This will cause the created template to be inconsistent with the queried template.
In template class, add mappingsEquals method, to deal with this case.
reproduced:
MetadataIndexTemplateServiceTests.testAddComponentTemplate
when mappings are not null, the case will failed.
```
Template template = new Template(
Settings.builder().build(),
new CompressedXContent("{\"properties\":{\"@timestamp\":{\"type\":\"date\"}}}"),
ComponentTemplateTests.randomAliases()
);
ComponentTemplate componentTemplate = new ComponentTemplate(template, 1L, new HashMap<>());
```
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Fixes the text field mapper and the analyzers class that also retained parameter references that go really heavy.
Makes `TextFieldMapper` take hundreds of bytes compared to multiple kb per instance.
closes#73845
This commit moves the die with dignity tests to be a test module. The
purpose of this is so the _die_with_dignity endpoint is available in
snapshot builds, for the purpose of enabling testing orchestration logic
that manages what happens to a node after it dies with an
OutOfMemoryError.
This commit updates two task names:
```
yamlRestCompatTest -> yamlRestTestV7CompatTest
transformV7RestTests -> yamlRestTestV7CompatTransform
```
`7` is the N-1 version and calculated, such that when `8` is
N-1 version the task names will be `yamlRestTestV8CompatTest` and
`yamlRestTestV8CompatTransform`
The motivation for `yamlRestCompatTest -> yamlRestTestV7CompatTest` is that
many projects have configured `yamlRestCompatTest`
but that configuration is specific to the N-1 version. For example,
if we blacklist tests when running compatibility with v7, we don't also
want to blacklist those tests when running compatibility with v8.
By introducing a version-specific identifier in the name, the task will not
even exist when bumping the version creating the need to (correctly) remove
the version-specific condition.
The motivation for `transformV7RestTests -> yamlRestTestV7CompatTransform`
is to provide more consistent naming.
The idea behind the naming is the main task people
are likely familiar with is :
`yamlRestTest` so we will use that as a base.
`yamlRestTestV7CompatTest` to run the version-specific compat tests
`yamlRestTestV7CompatTransform` to run the version-specific transformations for the compat tests
CI should be un-effected since since we introduced a lifecycle task
name `checkRestCompat` which is what CI should be configured to use.
The setting `index.hidden` was originally implemented as a static
setting, but when it was converted to be a dynamic setting, the docs
were never updated to reflect that.
This commit moves the docs for `index.hidden` from the section for
static settings to the section for dynamic settings.
Closes#76812. Closes#77126.
OsProbe was only capable of handle cgroup data in the v1 format.
However, Debian 11 uses cgroups v2 by default, and Elasticsearch isn't
capable of reporting any cgroup information. Therefore, add support for
the v2 layout.
Note that we have to open access to all of /sys/fs/cgroup because with
cgroups v2, the files we need are in an unpredictably location.
Currently we configure per-field postings formats by asking the MapperService
for the MappedFieldType of the field in question, and then checking to see if it
is a completion field. If no MappedFieldType is available, we emit a warning.
However, MappedFieldTypes are for search fields only, and so we end up emitting
warnings for hidden sub-fields that have no corresponding field type, such as
prefix or phrase accelerator fields on text mappers.
This commit reworks things so that the MappingLookup is responsible for defining
per-field postings formats, and will detect CompletionFieldMappers at build time.
All fields that are not mapped to a completion field will just get the default postings
format. This also means that we no longer need a logger instance on CodecService.
Fixes#77183
It is beneficial to sort segments within a datastream's index
by desc order of their max timestamp field, so
that the most recent (in terms of timestamp) segments
will be first.
This allows to speed up sort query on @timestamp desc field,
which is the most common type of query for datastreams,
as we are mostly concerned with the recent data.
This patch addressed this for writable indices.
Segments' sorter is different from index sorting.
An index sorter by itself is concerned about the order of docs
within an individual segment (and not how the segments are organized),
while the segment sorter is only used during search and allows
to start docs collection with the "right" segment,
so we can terminate the collection faster.
This PR adds a property to IndexShard `isDataStreamIndex` that
shows if a shard is a part of datastream.
This new parameter is a boolean parameter that allows
users to put in a compressed model without it having
to be inflated on the master node during the put
request
This is useful for system/module set up and then later
having the model validated and fully parsed when it
is being loaded on a node for usage
The test failure originates from an unhandled exception being thrown in the ml_utility threadpool.
This exception was thrown by the results persister service, I found a place where the exception thrown was not ultimately handled.
closes#77182
`AbstractFilteringTestCase` had a ton of carefully formatted
`XContentBuilder` calls that would have been totally unreadable after
the formatter worked its magic. So I formatted a bunch of them by hand
and extracted the rest to json files.
When every index that a transform is configured to search has
remained completely unchanged between checkpoints the transform
should not do a search at all.
Following #75839 there was a problem where the scenario of all
indices being unchanged between checkpoints could cause an empty
list of indices to be searched, which Elasticsearch treats as
meaning _all_ indices. This change should prevent that happening
in future.
Fixes#77137
There are a few failures handled in `OpenJobPersistentTasksExecutor`
while the job is opened that are not handled properly. Currently,
the persistent task is marked as failed which effectively completes
the task. Thus the job appears as `closed` and the failure reason
is lost.
This commit changes `OpenJobPersistentTasksExecutor` so that when
failures occur during job opening the job task is not completed.
Instead, the job task's state is updated to `failed` with a
corresponding reason.
Just like #77131 but for the `MatchOnlyTextFieldMapper`. Also, cleaned up a few
other minor things in it to make the constructor code for this class easier to follow.
Today ConfiguredHostsResolver and TransportAddressConnector are declared
within PeerFinder, for no particular reason. This commit lifts them to
the top level.
Today it's possible to open two connections to a node, and then we
notice when registering the second connection and close it instead.
Fixing #67873 will require us to keep tighter control over the identity
and lifecycle of each connection, and opening redundant connections gets
in the way of this. This commit adds a check for an existing connection
_after_ marking the connection as pending, which guarantees that we
don't open those redundant connections.
Today `AbstractRefCounted` has a `name` field which is only used to
construct the exception message when calling `incRef()` after it's been
closed. This isn't really necessary, the stack trace will identify the
reference in question and give loads more useful detail besides. It's
also slightly irksome to have to name every single implementation.
This commit drops the name and the constructor parameter, and also
introduces a handy factory method for use when there's no extra state
needed and you just want to run a method or lambda when all references
are released.
The `$ES_PATH_CONF` variable isn't set at this point when users are first starting Elasticsearch, so changing mentions to the `config/` directory for clarity.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Previously, we handled the case of a write request to a system index
alias without a backing index by auto-creating the primary index. This
had the unfortunate side-effect of making it impossible to auto-create
non-primary system indices. This commit fixes the bug so that we can
handle both cases.
* Add internal cluster test for system index auto create
* Allow auto-creation of non-primary indices for a system index pattern
* Use primary index if autocreate is called with system index alias name
The SearchHit.getFields() methods returns both document and metadata
fields commingled. This commit adds new methods to retrieve them
separately.
Fixes#77171
When doing out of order finalizations we would leak shard level metadata blobs at times.
This commit enhances the cleanup logic after finalization to catch these leaked blobs
and adds a test that would without this fix trip the leaked blobs assertion in the test
infrastructure.
In certain concurrent indexing scenarios where there are deletes
executed and then a new indexing operation, the following engine
considers those as updates breaking one of the assumed invariants.
Closes#72527
Investigating the heap use of mapper instances I found this.
It seems quite a bit of overhead for these instances goes into
the builder field. In other mappers we retain the script service
and the script outright, so I did the same thing here to make these
instances a little smaller.
The enrollment token contains a list of addresses where the node is
listening for requests in the HTTP layer. This change makes it so
we order that list before serializing it in the token prioritizing
IPv4 addresses vs IPv6 addresses. The reasoning is that
- it is usually more probable for IPv4 addresses to be reachable
so in the case where the consumer of the token needs to iterate
through available addresses, this might lead to getting a response
with fewer attempts
- When persisted to configuration files and/or shown in UIs, IPv4
addresses look more familiar (less surprising) to users
Introduce an internal action that the execute policy action delegates to.
This to ensure that the actual policy execution is never executed on the elected master node
or dedicated master nodes. In case the cluster consists out of a single node then
the internal action will attempt to execute on the current/local node.
The actual enrich policy execution is encapsulated in the `EnrichPolicyRunner` class.
This class manages the execution of several API calls, so this itself isn't doing anything heavy.
However the coordination of these api calls (in particular the reindex api call) may involve
some non-neglectable work/overhead and this shouldn't be performed on the elected master
or any other dedicated master node.
Closes#70436
randomArray sometimes generates duplicate entries and the subsequent
conversion to a Map hence fails due to the duplicate key. This PR fixes
the failure by ensure array elements are distinct before converting to a
Map. The alternative is to generate distinct array entries from the
beginning. But given how the code is used in the context, a simple
distinct call is easier and sufficient.
Resolves: #77127
This commit allows the compatible REST API tests to execute on Windows.
They were previously excluded from Windows due to a command line limit
when defining a very large exclusion list. That exclusion list is much
smaller now and they will now execute properly on Windows.
Also, an empty exclusion list has been removed from the build config.
weizijun
Armin Braun
Jason Tedor
Jake Landis
Benjamin Trent
Gordon Brown
Rory Hunter
Alan Woodward
Mayya Sharipova
Nik Everett
Howard
István Zoltán Szabó
Seth Michael Larson
Oleg Smirnov
David Roberts
Andrei Stefan
Rene Groeschke
Martijn van Groningen
Dimitris Athanasiou
Philip Krauss
David Turner
Ignacio Vera
Adam Locke
William Brafford
Henning Andersen
Francisco Fernández Castaño
Artem Prigoda
Ioannis Kakavas
Yang Wang
Bogdan Pintea