Commit Graph

7590 Commits

Author SHA1 Message Date
weizijun e321fb024e
Fix template equals when mappings are wrapped (#77008)
When create template v2. mappings will add _doc. This will cause the created template to be inconsistent with the queried template.

In template class, add mappingsEquals method, to deal with this case.

reproduced:
MetadataIndexTemplateServiceTests.testAddComponentTemplate
when mappings are not null, the case will failed.

```
        Template template = new Template(
            Settings.builder().build(),
            new CompressedXContent("{\"properties\":{\"@timestamp\":{\"type\":\"date\"}}}"),
            ComponentTemplateTests.randomAliases()
        );
        ComponentTemplate componentTemplate = new ComponentTemplate(template, 1L, new HashMap<>());
```

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-09-03 14:22:56 -06:00
Armin Braun 096b8ccc26
Fix TextFieldMapper Retaining a Reference to its Builder (#77251)
Fixes the text field mapper and the analyzers class that also retained parameter references that go really heavy.
Makes `TextFieldMapper` take hundreds of bytes compared to multiple kb per instance.

closes #73845
2021-09-03 18:44:11 +02:00
Rory Hunter d23837499d
Handle cgroups v2 in `OsProbe` (#77128)
Closes #76812. Closes #77126.

OsProbe was only capable of handle cgroup data in the v1 format.
However, Debian 11 uses cgroups v2 by default, and Elasticsearch isn't
capable of reporting any cgroup information. Therefore, add support for
the v2 layout.

Note that we have to open access to all of /sys/fs/cgroup because with
cgroups v2, the files we need are in an unpredictably location.
2021-09-03 15:49:56 +01:00
Alan Woodward 385b97f92b
Choose postings format from FieldMapper instead of MappedFieldType (#77234)
Currently we configure per-field postings formats by asking the MapperService
for the MappedFieldType of the field in question, and then checking to see if it
is a completion field. If no MappedFieldType is available, we emit a warning.
However, MappedFieldTypes are for search fields only, and so we end up emitting
warnings for hidden sub-fields that have no corresponding field type, such as
prefix or phrase accelerator fields on text mappers.

This commit reworks things so that the MappingLookup is responsible for defining
per-field postings formats, and will detect CompletionFieldMappers at build time.
All fields that are not mapped to a completion field will just get the default postings
format. This also means that we no longer need a logger instance on CodecService.

Fixes #77183
2021-09-03 14:54:43 +01:00
Mayya Sharipova f18b9d5ac8
Add segment sorter for data streams (#75195)
It is beneficial to sort segments within a datastream's index
by desc order of their max timestamp field, so
that the most recent (in terms of timestamp) segments
will be first.

This allows to speed up sort query on @timestamp desc field,
which is the most common type of query for datastreams,
as we are mostly concerned with the recent data.
This patch addressed this for writable indices.

Segments' sorter is different from index sorting.
An index sorter by itself is  concerned about the order of docs
within an individual segment (and not how the segments are organized),
while the segment sorter is only used during search and allows
to start docs collection with the "right" segment,
so we can terminate the collection faster.

This PR adds a property to IndexShard `isDataStreamIndex` that
shows if a shard is a part of datastream.
2021-09-03 09:42:48 -04:00
Howard 962e050d5e
Only check replicas in cancelling existing recoveries. (#60564)
Minor performance improvement when only primaries are initializing.
2021-09-03 14:59:45 +02:00
Nik Everett aa2aa9b1bf
Format `AbstractFilteringTestCase` (#77217)
`AbstractFilteringTestCase` had a ton of carefully formatted
`XContentBuilder` calls that would have been totally unreadable after
the formatter worked its magic. So I formatted a bunch of them by hand
and extracted the rest to json files.
2021-09-03 08:59:00 -04:00
Oleg Smirnov f9b38406f0
Prevent unnecessary boxing, improve code clarity. (#76808)
Use BooleanSupplier, OptionalInt, Consumer and Predicate in places
where more generic types were used.
2021-09-03 13:45:29 +02:00
David Turner 45bb29f408
Move PeerFinder inner interfaces to top level (#77194)
Today ConfiguredHostsResolver and TransportAddressConnector are declared
within PeerFinder, for no particular reason. This commit lifts them to
the top level.
2021-09-03 08:36:36 +01:00
David Turner 1f15764e43
Avoid redundant connections in ClusterConnectionManager (#77196)
Today it's possible to open two connections to a node, and then we
notice when registering the second connection and close it instead.
Fixing #67873 will require us to keep tighter control over the identity
and lifecycle of each connection, and opening redundant connections gets
in the way of this. This commit adds a check for an existing connection
_after_ marking the connection as pending, which guarantees that we
don't open those redundant connections.
2021-09-03 08:09:12 +01:00
David Turner bfcc93a042
Anonymize AbstractRefCounted (#77208)
Today `AbstractRefCounted` has a `name` field which is only used to
construct the exception message when calling `incRef()` after it's been
closed. This isn't really necessary, the stack trace will identify the
reference in question and give loads more useful detail besides. It's
also slightly irksome to have to name every single implementation.

This commit drops the name and the constructor parameter, and also
introduces a handy factory method for use when there's no extra state
needed and you just want to run a method or lambda when all references
are released.
2021-09-03 07:59:44 +01:00
Ignacio Vera 062276f84d
Refactor GeoBoundingBoxQuery integration tests (#77103)
This commit breaks the test to separate the testing of legacy geo_shape field.
2021-09-03 08:51:52 +02:00
Armin Braun b7c5293228
Make DataStreamTimestampFieldMapper a Singleton (#77209)
No need to have multiple instances of this one, it's either
disabled or enabled and that's it.
2021-09-02 21:59:10 +02:00
William Brafford e04e5eba65
AutoCreate should work for non-primary system indices (#77045)
Previously, we handled the case of a write request to a system index
alias without a backing index by auto-creating the primary index. This
had the unfortunate side-effect of making it impossible to auto-create
non-primary system indices. This commit fixes the bug so that we can
handle both cases.

* Add internal cluster test for system index auto create
* Allow auto-creation of non-primary indices for a system index pattern
* Use primary index if autocreate is called with system index alias name
2021-09-02 10:55:02 -04:00
Alan Woodward f70123ff9f
Add methods to retrieve doc and/or metadata fields only from SearchHit (#77192)
The SearchHit.getFields() methods returns both document and metadata
fields commingled. This commit adds new methods to retrieve them
separately.

Fixes #77171
2021-09-02 15:42:22 +01:00
Armin Braun e60f7ed96f
Fix Temporarily Leaking Shard Level Metadata Blobs in some Cases (#76562)
When doing out of order finalizations we would leak shard level metadata blobs at times.
This commit enhances the cleanup logic after finalization to catch these leaked blobs
and adds a test that would without this fix trip the leaked blobs assertion in the test
infrastructure.
2021-09-02 13:08:35 +02:00
Francisco Fernández Castaño b7838d680f
Fix FollowingEngineTests#testOptimizeMultipleVersions (#75583)
In certain concurrent indexing scenarios where there are deletes
executed and then a new indexing operation, the following engine
considers those as updates breaking one of the assumed invariants.

Closes #72527
2021-09-02 11:12:14 +02:00
Artem Prigoda a9747988d6
Add a unit test verifying that the IPv6 scope id is optional for node ips (#76889)
Closes #72091
Ref #60172
2021-09-02 11:11:25 +02:00
Armin Braun c082c2578a
Fix NumberFieldMapper Referencing its Own Builder (#77131)
Investigating the heap use of mapper instances I found this.
It seems quite a bit of overhead for these instances goes into
the builder field. In other mappers we retain the script service
and the script outright, so I did the same thing here to make these
instances a little smaller.
2021-09-02 10:26:41 +02:00
Armin Braun 8b9c52ebe3
Fix Queued Snapshot Clone not Starting after Data Node Drops Out (#77111)
We have to account for queued up clones when dealing with nodes dropping out
and start them when they become ready to execute because of a node leaving the cluster.
Added test to reproduce the issue in #77101 and another test to verify that the more complex
case of clone queued after snapshot queued after clone still works correctly as well.
The solution here is the most direct fix I could think of and the by far easiest to backport.
That said, I added a TODO that asks for a follow-up that should allow for completely removing
the duplicate code across handling shard updates and external changes. The difference between
the two ways of updating the state is a left-over from the time before we had concurrent
operations and has become a needless complexity nowadays.

closes #77101
2021-09-01 20:45:22 +02:00
Martijn van Groningen 4a4d604463
Add 7.14.2 version. 2021-09-01 18:30:51 +02:00
Nik Everett dde7d418a8
Add matching pattern to error in fields (#76903)
This adds the pattern into the error message returned when trying to
fetch fields. So this:
```
POST _search {
  "fields": [ { "field": "*", "format": "date_time" } ]
}
```

Will return an error message like
```
error fetching [foo] which matches [*]: Field [foo] of type [keyword] doesn't support formats
```
2021-09-01 12:15:54 -04:00
Alan Woodward 5d48fdc741
Replace Lucene DataInput/DataOutput with Elasticsearch StreamInput/StreamOutput (#77118)
In a number of places, we read and write binary data into byte arrays using lucene's
DataInput and DataOutput abstractions. In lucene 9 these abstractions are changing
the endianness of their read/writeInt methods. To avoid dealing with this formatting
change, this commit changes things to use elasticsearch StreamInput/StreamOutput
abstractions instead, which have basically the same API but will preserve endianness.

Relates to #73324
2021-09-01 16:21:26 +01:00
Rory Hunter 33c4129a60 Revert "Reapply "Handle cgroups v2 in `OsProbe` (#76883)" (#77106)"
This reverts commit 54d4737b23.
2021-09-01 14:32:24 +01:00
Nik Everett 429beba517
Centralize doc values checking (#77089)
This adds two utility methods for to validate the parameters to the
`docValueFormat` method and replaces a pile of copy and pasted code with
calls to them. They just emit a standard error message if the any
unsupported parameters are provided.
2021-09-01 09:06:24 -04:00
Benjamin Trent 100f222650
Adds support for the rate aggregation under a composite agg (#76992)
rate aggregation should support being a sub-aggregation
of a composite agg.

The catch is that the composite aggregation source
must be a date histogram. Other sources can be present
but their must be exactly one date histogram source
otherwise the rate aggregation does not know which
interval to compare its unit rate to.

closes https://github.com/elastic/elasticsearch/issues/76988
2021-09-01 07:29:13 -04:00
Armin Braun 0920e21445
Implement Sort By Repository Name in Get Snapshots API (#77049)
This one is the last sort column not yet implemented but used by Kibana.
2021-09-01 13:01:58 +02:00
Rory Hunter 54d4737b23
Reapply "Handle cgroups v2 in `OsProbe` (#76883)" (#77106)
Re-apply #76883. Somehow a line was missed from security.policy.
2021-09-01 11:46:49 +01:00
Rory Hunter 54a9c3ba05 Revert "Handle cgroups v2 in `OsProbe` (#76883)"
This reverts commit e827ec6078.
2021-09-01 09:42:45 +01:00
Rory Hunter e827ec6078
Handle cgroups v2 in `OsProbe` (#76883)
Closes #76812.

`OsProbe` was only capable of handle cgroup data in the v1 format.
However, Debian 11 uses cgroups v2 by default, and Elasticsearch isn't
capable of reporting any cgroup information. Therefore, add support for
the v2 layout.
2021-09-01 09:29:01 +01:00
Ignacio Vera 07715438b5
Refactor of GeoShape integration tests (#77052)
This commit joins GeoFilterIT and GeoShapeIntegrationIT into one test case called GeoShapeIntegTestCase 
which is moved into the test framework.
2021-09-01 07:21:15 +02:00
Ignacio Vera 648a7aefbe
Replace Lucene DataInput/DataOutput with Elasticsearch StreamInput/StreamOutput when reading/writing geo_shape doc values (#76162)
This commit introduces a new ByteArrayStreamInput that allows reusability and random reads.
2021-09-01 07:05:56 +02:00
Christos Soulios 707dd497e4
Add multiple validators to Parameters (#77073)
This PR implements support for multiple validators to a FieldMapper.Parameter.

The Parameter#setValidator method was replaced by Parameter#addValidator that can be called multipled times
to add validation to a parameter.

All validators of a parameter will be executed in the same order as they have been added and if any of them fails all validation will failed.
2021-08-31 21:28:14 +03:00
David Turner ead0020497
Tidy up ClusterApplierService (#76837)
This commit cleans up some cruft left over from older versions of the
`ClusterApplierService`:

- `UpdateTask` doesn't need to implement lots of interfaces and give
  access to its internals, it can just pass appropriate arguments to
  `runTasks()`.
- No need for the `runOnApplierThread` override with a default priority,
  just have callers be explicit about the priority.
- `submitStateUpdateTask` takes a config which never has a timeout, may
  as well just pass the priority and remove the dead code
- `SafeClusterApplyListener` doesn't need to be a
  `ClusterApplyListener`, may as well just be an `ActionListener<Void>`.
- No implementations of `ClusterApplyListener` care about the source
  argument, may as well drop it.
- Adds assertions to prevent `ClusterApplyListener` implementations from
  throwing exceptions since we just swallow them.
- No need to override getting the current time in the
  `ClusterApplierService`, we can control this from the `ThreadPool`.
2021-08-31 17:35:32 +01:00
Armin Braun 37516daa56
Make some Mappers Singletons (#77067)
Just some obvious singletons we can use to save a little memory/cache
that I found during other experiments.
2021-08-31 17:25:05 +02:00
Armin Braun f89eda5f9d
Fix Snapshot BwC Version Randomization Behavior (#77057)
The randomization of the repo version often wasn't used because of the repository cache.
Force re-creating the repository every time we manually mess with the versions.
2021-08-31 15:28:57 +02:00
markharwood 2d0773cd3c
Add support for QueryString-parsed wildcard queries on runtime keyword fields. (#76867)
The QueryStringQuery parser assumes that wildcard queries should use normalized values in queries.
The KeywordScriptFieldType did not support this so was throwing an error. Given there is currently no concept of normalisation in scripted fields I assume it is safe to just add support for this in the same way un-normalized wildcard queries are handled - it feels right that they should behave the same rather than throw an error.
Added a test too.

Closes #76838
2021-08-31 11:35:54 +01:00
Mark Tozzi d715449766
Apply spotless to aggregations (#76682)
* spotless apply for core analytics code

* spotless apply for x-pack:plugin:analytics

* spotless apply for matrix stats

* don't format huge lists of HLL constants

* Spotless for Rollups

* Don't exclude HLLPP, we added narrower inline exclusions
2021-08-30 13:42:20 -04:00
Gordon Brown 7c5bd03738
Add `replace` shutdown type (#75908)
This PR adds the `REPLACE` shutdown type. As of this PR, `REPLACE` behaves identically to `REMOVE`.

Co-authored-by: Lee Hinman <lee@writequit.org>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-30 11:34:42 -06:00
Julie Tibshirani 061253ee0e
Break up and simplify TransportFieldCapabilitiesAction (#76958)
`TransportFieldCapabilitiesAction` currently holds a lot of logic. This PR
breaks it up into smaller pieces and simplifies its large `doExecute` method.
Simplifying the class will help before we start to make field caps
optimizations.

Changes:
* Factor some methods out of `doExecute` to reduce its length
* Pull `AsyncShardAction` out into its own class to simplify and better match
the code structure in 7.x
2021-08-30 10:14:42 -07:00
Armin Braun 111100ebfd
Adjust GetSnapshotsRequest BwC Serialization after Backport (#77019)
Once #77018 is merged, we can merge this to enabled the new sort columns
in 7.16+.
2021-08-30 14:59:04 +02:00
Armin Braun 48f3784a6d
Add Sort By Shard Count and Failed Shard Count to Get Snapshots API (#77011)
It's in the title. As requested by the Kibana team, adding these two additional sort columns.

relates #74350
2021-08-30 13:39:51 +02:00
Stuart Tettemer 3c737341fe
Script: Fields API converter tests (#76900) 2021-08-26 09:18:42 -05:00
Przemyslaw Gomulka 0e50d2041c
Expand the minimum utc timestamp used in fetching timezone transitions (#75584)
when rounding UTC timestamps we convert a timestamp from UTC to local, round this to closest midnight, then we convert back to UTC.
This means, that for a timestamp close to a DST we need to make sure we collected a transition that will be needed when converting back to UTC.
To do this, we decrease the minUtcMillis by 2* unit to sure that the additional transition that could affect the timestamp is also fetched and the correct minimum is used in further lookups

closes #73995
2021-08-26 09:05:19 +02:00
Nhat Nguyen 25984d159d
Ensure norms is non-zero in IndexDiskUsageAnalyzerIT (#76894)
The test fails when norms have a singleton value and is loaded eagerly. 
This commit makes sure that we index more than one norm value.

Closes #76173
2021-08-25 09:43:37 -04:00
Nhat Nguyen 7b618a01d9
Relax disk usage test assertions (#76895)
This commit allows up to 2K data structures that are loaded
before we can track bytes read.

Closes #76643
2021-08-25 09:41:17 -04:00
Nikola Grcevski 98f0f4b9ec
[TEST] Implement HotThreads unit tests (#76857)
* [TEST] Implement HotThreads unit tests

Add unit tests for the internal HotThreads logic for calculating and
sorting threads by CPU, Wait and Blocked "hotness". Also adds tests
for identifying certain threads as idle, as well as supported report
types (e.g. cpu, wait, blocked).

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-24 18:14:49 -04:00
Armin Braun 98bf4eb3d5
Fix Test Failure in testMasterFailOverWithQueuedDeletes (#76865)
We can have a freak situation here where running the get snapshots
request concurrently with a delete produces a missing snapshot exception
if a snasphot is deleted just as its metadata is fetched.
This is a known issue and a fix is tricky (SLM etc. work around this issue
in tests and prod code by using the ignore-unavailable flag for example).
In this test we can easily fix the problem by just using the deterministic waiting
on cluster state before asserting that the snapshots are gone from the repo.

closes #76549
2021-08-24 13:48:53 +02:00
David Turner 007b5aec40
Fix testMasterStatsOnSuccessfulUpdate (#76844)
This test advances time in surprising places, so we have to allow a
little longer for publication or stabilisation to complete.

Closes #76840
2021-08-23 15:02:16 -04:00
David Turner 1ebc378e78 AwaitsFix for #76840 2021-08-23 19:07:05 +01:00