We have recently split DocumentMapper creation from parsing Mapping. There was one method leftover that exposed parsing mapping into DocumentMapper, which is generally not needed. Either you only need to parse into a Mapping instance, which is more lightweight, or like in some tests you need to apply a mapping update for which you merge new mappings and get the resulting document mapper. This commit addresses this and removes the method.
Today we track a couple of values for each snapshot in the top-level
`RepositoryData` blob: the overall snapshot state and the version of
Elasticsearch which took the snapshot. In the blob these values are
fields of the corresponding snapshot object, but in code they're kept in
independent maps. In the near future we would like to track some more
values in the same way, but adding a new field for every tracked value
is a little ugly. This commit refactors these values into a single
object, `SnapshotDetails`, so that it can be more easily extended in
future.
In the corner case of uploading a large (>5MB) metadata blob we did not set content validation
requirement on the upload request (we automatically have it for smaller requests that are not resumable
uploads). This change sets the relevant request option to enforce a MD5 hash check when writing
`BytesReference` to GCS (as is the case with all but data blob writes)
closes#72018
This adds a new `match_only_text` field, which indexes the same data as a `text`
field that has `index_options: docs` and `norms: false` and uses the `_source`
for positional queries like `match_phrase`. Unlike `text`, this field doesn't
support scoring.
#71696 introduced a regression to the various shape field mappers,
where they would no longer handle null values. This commit fixes
that regression and adds a testNullValues method to MapperTestCase
to ensure that all field mappers correctly handle nulls.
Fixes#71874
This commit adds support for system data streams and also the first use
of a system data stream with the fleet action results data stream. A
system data stream is one that is used to store system data that users
should not interact with directly. Elasticsearch will manage these data
streams. REST API access is available for external system data streams
so that other stack components can store system data within a system
data stream. System data streams will not use the system index read and
write threadpools.
This drops a few properties from the reproduction info printed when a
test fails because it is implied by the build:
* `tests.security.manager`
* `tests.rest.suite`
* `tests.rest.blacklist`
The two `tests.rest` properties a set by the build *and* duplicate the
`--test` output!
Closes#71290
The frozen tier partially downloads shards only. This commit
introduces an autoscaling decider that scales the total storage
on the tier according to a configurable percentage relative to
the total data set size.
As we started thinking about applying on_script_error to runtime fields, to handle script errors at search time, we would like to use the same parameter that was recently introduced for indexed fields. We decided that continue or fail gives a better indication of the behaviour compared to the current ignore or reject which is too specific to indexing documents.
This commit applies such rename.
Some functionality will no longer work with multiple data paths and in
order to run integration tests for that, we need the capability to
force a single data path for those tests.
Relates #71844
This commit adds some per-index statistics to the `SnapshotInfo` blob:
- number of shards
- total size in bytes
- maximum number of segments per shard
It also exposes these statistics in the get snapshot API.
This commit allows to use the include_type_name parameter with the compatible rest api.
The support for include_type_name was previously removed in #48632
relates #51816
types removal meta issue #54160
Adds support for close_to assertion to yaml tests. The assertion can be called
the following way:
```
- close_to: { get.fields._routing: { value: 5.1, error: 0.00001 } }
```
Closes#71303
This PR makes sure MockSearchService collects ARS statistics. Before, if we
randomly chose to use MockSearchService then ARS information would be missing
and the test would fail.
Also makes the following fixes:
* Removes a test workaround for the bug #71022, which is now fixed.
* Handle the case where nodes have same rank, to prevent random failures.
This speeds up the `terms` aggregation when it can't take the fancy
`filters` path, there is more than one segment, and any of those
segments have only a single value for the field. These three things are
super common.
Here are the performance change numbers:
```
| 50th percentile latency | date-histo-string-terms-via-global-ords | 3414.02 | 2632.01 | -782.015 | ms |
| 90th percentile latency | date-histo-string-terms-via-global-ords | 3470.91 | 2756.88 | -714.031 | ms |
| 100th percentile latency | date-histo-string-terms-via-global-ords | 3620.89 | 2875.79 | -745.102 | ms |
| 50th percentile service time | date-histo-string-terms-via-global-ords | 3410.15 | 2628.87 | -781.275 | ms |
| 90th percentile service time | date-histo-string-terms-via-global-ords | 3467.36 | 2752.43 | -714.933 | ms | 20%!!!!
| 100th percentile service time | date-histo-string-terms-via-global-ords | 3617.71 | 2871.63 | -746.083 | ms |
```
This works by hooking global ordinals into `DocValues.unwrapSingleton`.
Without this you could unwrap singletons *if* the segment's ordinals
aligned exactly with the global ordinals. If they didn't we'd return an
doc values iterator that you can't unwrap. Even if the segment ordinals
were singletons.
That speeds up the terms aggregator because we have a fast path we can
take if we have singletons. It was previously only working if we had a
single segment. Or if the segment's ordinals lined up exactly. Which,
for low cardinality fields is fairly common. So they might not benefit
from this quite as much as high cardinality fields.
Closes#71086
Revamps the integration tests for the `filter` agg to be more clear and
builds integration tests for the `fitlers` agg. Both of these
integration tests are fairly basic but they do assert that the aggs
work.
This PR introduces a new query called `combined_fields` for searching multiple
text fields. It takes a term-centric view, first analyzing the query string
into individual terms, then searching for each term any of the fields as though
they were one combined field. It is based on Lucene's `CombinedFieldQuery`,
which takes a principled approach to scoring based on the BM25F formula.
This query provides an alternative to the `cross_fields` `multi_match` mode. It
has simpler behavior and a more robust approach to scoring.
Addresses #41106.
Normally there are only 3 parts to a YAML REST test
`api/name/test section name` where `api` is sourced
from the filesystem, a relative path from the root of
the tests. `name` is the filename of the test minus the `.yml`
and the `test section name` is from inside the .yml file`
Some tests have use multiple directories to represent the `api`
for example `foo/bar/10_basic/My test Name` where foo/bar is the
relative path from the root of the tests. All works fine in both
*nix and Windows. Except for when you need to reference that `api`
(aka path from root) under Windows. Under Windows that relative path
uses backslashes to represent the `api`. This means that under Windows
you need to `foo\bar/10_basic/My test Name` to reproduce\execute a test.
Additionally, due to how the regex matching is done for blacklisting tests
the backslash will never match, so it is not possible to
blacklist a 4+ path YAML REST test for Windows.
This commit simply ensures that the API part is always represented as a
forward slash. This commit also removes a prior naive attempt to blacklist
on Windows.
closes#71475
This commit removes the ability for plugins to add roles. Roles are
fairly tightly coupled with the behavior of the system (as evidenced by
the fact that some roles from the default distribution leaked behavior
into the OSS distribution). We previously had this plugin extension
point so that we could support a difference in the set of roles between
the OSS and default distributions. We no longer need to maintain that
differentiation, and can therefore remove this plugin extension
point. This was technical debt that we were willing to accept to allow
the default distribution to have additional roles, but now we no longer
need to be encumbered with that technical debt.
copy_to is currently implemented at document parse time, and does not
work with values generated from index-time scripts. We may want to add
this functionality in future, but for now this commit ensures that we throw
an exception if copy_to and script are both set on a field mapper.
* Warn users if security is implicitly disabled
Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
clusters.
This change introduces clear warnings when security features are
implicitly disabled.
- a warning header in each REST response if security is implicitly
disabled;
- a log message during cluster boot.
This fixes the `global` aggregator when `profile` is enabled. It does so
by removing all of the special case handling for `global` aggs in
`AggregationPhase` and having the global aggregator itself perform the
scoped collection using the same trick that we use in filter-by-filter
mode of the `filters` aggregation.
Closes#71098
This adds a few tests for runtime field queries applied to
"filter-by-filter" style aggregations. We expect to still be able to
use filter-by-filter aggregations to speed up collection when the top
level query is a runtime field. You'd think that filter-by-filter would
be slow when the top level query is slow, like it is with runtime
fields, but we only run filter-by-filter when we can translate each
aggregation bucket into a quick query. So long as the results of those
queries don't "overlap" we shouldn't end up running the slower top level
query more times than we would during regular collection.
This also adds some javadoc to that effect to the two places where we
chose between filter-by-filter and a "native" aggregation
implementation.
Elasticsearch enumerates Lucene files extensions for various
purposes: grouping files in segment stats under a description,
mapping files in memory through HybridDirectory or adjusting
the caching strategy for Lucene files in searchable snapshots.
But when a new extension is handled somewhere(let's say,
added to the list of files to mmap) it is easy to forget to add it
in other places. This commit is an attempt to centralize in a
single place all known Lucene files extensions in Elasticsearch.
Multifields are built at the same time as their parent fields, using
a positioned xcontent parser to read information. Fields with index
time scripts are built entirely differently, and it does not make sense
to combine the two.
This commit adds a base test to MapperScriptTestCase that ensures
a field mapper defined with both multifields and a script parameter
throws a parse error.
This change adds additional test to GeoIpDownloaderIT which tests that artifacts produces by GeoIP CLI tool can be consumed by cluster the same way as from our original service.
It does so by running the tool from fixture which then simply serves the generated files (this is exactly the way users are supposed to use the tool as well).
Relates to #68920
When we added scripts to long and double mapped fields, we added tests
for the general scripting infrastructure, and also specific tests for those two
field types. This commit extracts those type-specific tests out into a new base
test class that we can use when adding scripts to more field mappers.
Adds support for "Default Application Credentials" for GCS repositories, making it easier to set up a repository on GCP,
as all relevant information to connect to the repository is retrieved from the environment, not necessitating complicated
keystore setups.
In some scenarios where the read timeout is too tight it's possible
that the http request times out before the response headers have
been received, in that case an URLHttpClientIOException is thrown.
This commit adds that exception type to the expected set of read timeout
exceptions.
Closes#70931
The test currently generates a list of random values and checks whether
retrieval of these values via doc values is equivallent to fetching them with a
value fetcher from source. If the random value array contains a duplicate value,
we will only get one back via doc values, but fetching from source will return
both, which is a case we should probably avoid in this test.
Closes#71053
Today when creating an internal test cluster, we allow the test to
supply the node settings that are applied. The extension point to
provide these settings has a single integer parameter, indicating the
index (zero-based) of the node being constructed. This allows the test
to make some decisions about the settings to return, but it is too
simplistic. For example, imagine a test that wants to provide a setting,
but some values for that setting are not valid on non-data nodes. Since
the only information the test has about the node being constructed is
its index, it does not have sufficient information to determine if the
node being constructed is a non-data node or not, since this is done by
the test framework externally by overriding the final settings with
specific settings that dicate the roles of the node. This commit changes
the test framework so that the test has information about what settings
are going to be overriden by the test framework after the test provide
its test-specific settings. This allows the test to make informed
decisions about what values it can return to the test framework.
This PR sets the default value of `action.destructive_requires_name`
to `true.` Fixes#61074. Additionally, we set this value explicitly in
test classes that rely on wildcard deletions to clear test state.
This commit moves the data tier roles to server. It is no longer
necessary to separate these roles from server as we no longer build
distributions that would not contain these roles. Moving these roles
will simplify many things. This is deliberately the smallest possible
commit that moves these roles. Other aspects related to the data tiers
can move in separate, also small, commits.
Air-gapped environments can't simply use GeoIp database service provided by Infra, so they have to either use proxy or recreate similar service themselves.
This PR adds tool to make this process easier. Basic workflow is:
download databases from MaxMind site to single directory (either .mmdb files or gzipped tarballs with .tgz suffix)
run the tool with $ES_PATH/bin/elasticsearch-geoip -s directory/to/use [-t target/directory]
serve static files from that directory (for example with docker run -v directory/to/use:/usr/share/nginx/html:ro nginx
use server above as endpoint for GeoIpDownloader (geoip.downloader.endpoint setting)
to update new databases simply put new files in directory and run the tool again
This change also adds support for relative paths in overview json because the cli tool doesn't know about the address it would be served under.
Relates to #68920
This commit adds a script parameter to long and double fields that makes
it possible to calculate a value for these fields at index time. It uses the same
script context as the equivalent runtime fields, and allows for multiple index-time
scripted fields to cross-refer while still checking for indirection loops.
Runtime fields currently live in their own java package. This is really
a leftover from when they were in their own module; now that they are
in core they should instead live in the common packages for classes of
their kind.
This commit makes the following moves:
org.elasticsearch.runtimefields.mapper => org.elasticsearch.index.mapper
org.elasticsearch.runtimefields.fielddata => org.elasticsearch.index.fielddata
org.elasticsearch.runtimefields.query => org.elasticsearch.search.runtime
The XFieldScript fields are moved out of the `mapper` package into
org.elasticsearch.scripts, and the `PARSE_FROM_SOURCE` default scripts
are moved from these Script classes directly into the field type classes that
use them.
We have to ship COPYRIGHT.txt and LICENSE.txt files alongside .mmdb files for legal compliance. Infra will pack these in single .tgz (gzipped tar) archive provided by GeoIP databases service.
This change adds support for that format to GeoIpDownloader and DatabaseRegistry
We use the `getDataPath` method to convert from a resource
location to a filesystem path in anticipation of eventually moving
the json files to a top-level directory. However, we were constructing
the resource locations using a filesystem concatenation, which,
on Windows, put backslashes in the path instead of slashes.
We will use a simple string concatenation to fix the Windows tests.
We've had a few bugs in the fields API where is doesn't behave like we'd
expect. Typically this happens because it isn't obvious what we expct. So
we'll try and use randomized testing to ferret out what we want. This adds
a test for most field types that asserts that `fields` works similarly
to `docvalues_fields`. We expect this to be true for most fields.
It does so by forcing all subclasses of `MapperTestCase` to define a
method that makes random values. It declares a few other hooks that
subclasses can override to further randomize the test.
We skip the test for a few field types that don't have doc values:
* `annotated_text`
* `completion`
* `search_as_you_type`
* `text`
We should come up with some way to test these without doc values, even
if it isn't as nice. But that is a problem for another time, I think.
We skip the test for a few more types just because I wanted to cut this
PR in half so we could get to reviewing it earlier. We'll get to those
in a follow up change.
I've filed a few bugs for things that are inconsistent with
`docvalues_fields`. Typically that means that we have to limit the
random values that we generate to those that *do* round trip properly.
Today nothing prevents CCR's auto-follow patterns to pick
up snapshot backed indices on a remote cluster. This can
lead to various errors on the follower cluster that are not
obvious to troubleshoot for a user (ex: multiple engine
factories provided).
This commit adds verifications to CCR to make it fail faster
when a user tries to follow an index that is backed by a
snapshot, providing a more obvious error message.
The types removal effort has removed the type from Index API in #47671 and from Get API in #46587
This commit allows to use 'typed' endpoints for the both Index and Get APIs
relates compatible types-removal meta issue #54160
TypeParser.ParserContext exposes a ScriptService allowing type parsers to
compile scripts from runtime fields (and in future, index-time scripts). However,
ScriptService itself is a fairly heavyweight object, and you can only add script
parsers to it via the Plugin mechanism.
To make testing easier, this commit extracts a ScriptCompiler interface and makes
that available from ParserContext instead.
* Don't try to invoke delete component/index templates APIs if there are no templates to delete.
* Don't delete deprecation templates by marking these as xpack templates.
Relates to #69973
Low-effort fix for #70482 that makes sure that no two slices get created with the
same name but different byte ranges in a single test run which lead to conflicts with
the assumption that cfs file ranges are fixed for given names for the frozen index input.
closes#70482
Rest API specs define the API's used at the rest level, however these specs
only define the endpoint and the parameters. We miss definitions for the
body, especially when it comes to rich bodies like they are used in ML.
This change introduces an abstract testcase for json schema validation. This
allows developers to validate any object that is serializable to JSON - using
the `ToXContent` - to be tested against a json schema. You can use it for REST
input and outputs, but also for internal objects(documents) and
`ToXContentFragments`.
As the overall goal is to ensure it validates properly, the testcase enforces
strictness. A schema file must spec all properties. This will ensure that once
a schema test has been added, it won't go out of sync. Every change to the
pojo enforces a schema update as otherwise the test would fail.
Schemas can load sub-schemas from extra files. That way you can re-use schemas
e.g. in hierarchies or re-use a schema for similar but not same interfaces.
If you send `percentiles` or `percentiles_ranks` over the transport
client or over cross cluster search it breaks internal components
subtly. They mostly work so we hadn't yet noticed the break. But if you
send the request to the slow log then it'll fail to log. This fixes the
subtle internal break.
Add support to delete component templates api to specify multiple template
names separated by a comma.
Change the cleanup template logic for rest tests to remove all component templates via a single delete component template request. This to optimize the cleanup logic. After each rest test we delete all templates. So deleting templates this via a single api call (and thus single cluster state update) saves a lot of time considering the number of rest tests.
Older versions don't support component / composable index templates
and/or data streams. Yet the test base class tries to remove objects
after each test, which adds a significant number of lines to the
log files (which slows the tests down). The ESRestTestCase will
now check whether all nodes have a specific version and then decide
whether data streams and component / composable index templates will
be deleted.
Also ensured that the logstash-index-template and security-index-template
aren't deleted between tests, these templates are builtin templates that
ES will install if missing. So if tests remove these templates between tests
then ES will add these template back almost immediately. These causes
many log lines and a lot of cluster state updates, which slow tests down.
Relates to #69973
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Add support to delete composable index templates api to specify multiple template
names separated by a comma.
Change to cleanup template logic for rest tests to remove all composable index templates via a single delete composable index template request. This to optimize the cleanup logic. After each rest test we delete all templates. So deleting templates this via a single api call (and thus single cluster state update) saves a lot of time considering the number of rest tests.
If this pr is accepted then I will do the same change for the delete component template api.
Relates to #69973
This commit removes the metadata field _parent_join
that was needed to ensure that only one join field is used in a mapping.
It is replaced with a validation at the field level.
This change also fixes in [bug](https://github.com/elastic/kibana/issues/92960) in the handling of parent join fields in _field_caps.
This metadata field throws an unexpected exception in [7.11](https://github.com/elastic/elasticsearch/pull/63878)
when checking if the field is aggregatable.
That's now fixed since this unused field has been removed.
A #68808 introduced a possibility to declare fields which will be only available to parsing when a compatible API was used.
This commit replaces deprecated log with compatible logging when a 'compatible only' field was used. Also includes a refactoring of LoggingDeprecationHandler method names
relates #51816
This fixes an issue that `fields` has with dates sent in the
`###.######` format.
If you send a date in the format `#####.######` we'll parse the bit
before the decimal place as the number of milliseconds since epoch and
we'll parse the bit after the decimal as the number of nanoseconds since
the start of that millisecond. This works and is convenient for some
folks. Sadly, the code that back the `fields` API for dates doesn't work
with the string format in this case - it works with a `double`. `double`
is bad for two reasons:
1. It's default string representation is scientific notation and our
parsers don't know how to deal with that.
2. It loses precision relative to the string representation. `double`
only has 52 bits of mantissa which can precisely store the number of
nanoseconds until about 6am on April 15th, 1970. After that it starts
to lose precision.
This fixed the first issue, getting us the correct string
representation is a "quick and dirty" way. It just converts the `double`
back to a string. But we still lose precision. Fixing that would require
a larger change.....
We only used this method in tests and it's somewhat needless to have a potentially
slow `notifyAll` in the hot path for assertions only when we can just busy assert in tests instead.
closes#69963
Before this change upon wiping the cluster, we would get a list of all legacy and index component templates. For each template first attempt to delete it as legacy template if that returned a 404 then remove it as composable index template. In the worst case this means that we would make double the amount of delete requests for templates then is necessary.
This change first gets all composable index templates (if exist and if the cluster supports it) and then deletes these composable index templates. After this separately get a list of all legacy templates and then delete those legacy templates.
Relates to #69973
Today in tests we often use a utility method that creates and starts a
transport service, and then we start it again in the tests anyway. This
commit removes this unnecessary code and asserts that we only ever call
`TransportService#acceptIncomingRequests` once.
Unwrapping the `byte[]` in a `BytesArray` can be generalized
so that releasable bytes references can make use of the optimizations
that use the bytes directly as well.
Relates #67502 but already saves a few cycles here and there around
`BytesRequest` (publish and recovery actions)
Search cancellation currently does not work well in the context of searchable snapshot shards, as it requires search
tasks to fully enter the query phase (i.e. start execution on the node, loading up the searcher, which means loading up
the index on FrozenEngine and doing some actual work) to detect cancellation, which can take a while in the frozen tier,
blocking on file downloads.
Since #69861 CFS files read from FrozenIndexInput create
dedicated frozen shared cache files when they are sliced.
This does not play well with some tests that use the
randomReadAndSlice to read files: this method can create
overlapping slice/clone reads operations which makes it
difficult to assert anything about CFS files with partial cache.
This commit prevent the tests to generate a .cfs file name
when the partial cache is randomly picked up. As a follow
up we should rework those test to make them more realistic
with the new behavior.
Closes#70000
The blob store cache is used to cache a variable length of the
begining of Lucene files in the .snapshot-blob-cache system
index. This is useful to speed up Lucene directory opening
during shard recovery and to limit the number of bytes
downloaded from the blob store when a searchable snapshot
shard must be rebuilt.
This commit adds support for compound files segment (.cfs)
when they are partially cached (ie, Storage.SHARED_CACHE)
so that the files they are composed of can also be cached in
the blob store cache index.
Co-authored-by: Yannick Welsch yannick@welsch.lu
Today every `BulkProcessor` creates two scheduler threads, both called
`[node-name][scheduler][T#1]`, which is also the name of the main
scheduler thread for the node. The duplicated thread names make it
harder to interpret a thread dump.
This commit makes the names of these threads distinct.
Closes#68470
Reading ops from the translog snapshot must not run on the transport thread.
When sending more than one batch of ops the listener (and thus `run`) would be
invoked on the transport thread for all but the first batch of ops.
=> Forking to the generic pool like we do for sending ops during recovery.
This change updates GeoIP database service URL to the new https://geoip.elastic.co/v1/database and removes (now optional) key/UUID parameter.
It also fixes geoip-fixture to provide 3 different test databases (City, Country and ASN).
It also unmutes GeoIpDownloaderIT. testGeoIpDatabasesDownload with additional logging and increased timeouts which tries to address #69594
MapperRegistry is the only class under the indices.mapper package. It fits better under index.mapper together with MapperService and friends, hence this commit moves it there and removes the indices.mapper package.
This commit adds support for two new REST test features.
warnings_regex and allowed_warnings_regex.
This is a near mirror of the warnings and allowed_warnings
warnings feature where the test can be instructed to allow
or require HTTP warnings. The difference with these new features
is that is allows the match to be based on a regular expression.
This commit introduces system index types that will be used to
differentiate behavior. Previously system indices were all treated the
same regardless of whether they belonged to Elasticsearch, a stack
component, or one of our solutions. Upon further discussion and
analysis this decision was not in the best interest of the various
teams and instead a new type of system index was needed. These system
indices will be referred to as external system indices. Within external
system indices, an option exists for these indices to be managed by
Elasticsearch or to be managed by the external product.
In order to represent this within Elasticsearch, each system index will
have a type and this type will be used to control behavior.
Closes#67383
In 2a04118e88 we moved `ensureGreen()`
from `IndexingIT` to `ESRestTestCase`, including its `70s` timeout. This
timeout makes sense in the context of an `AbstractRollingTestCase` which
has a client timeout of `90s` (#26781) but general-purpose REST tests
only have a `60s` client timeout, so if `ensureGreen()` fails then it
fails with a `SocketTimeoutException`, bypassing the useful exception
handling that log the cluster state at time of failure.
This commit reduces the `ensureGreen()` timeout for most tests, leaving
it at `70s` only for `AbstractRollingTestCase`.
We should not be ignoring and suppressing exceptions on releasing
network resources quietly in these spots.
Co-authored-by: David Turner <david.turner@elastic.co>
If a shard is reassigned to a node, but it has open searches (could be
scrolls even), the current behavior is to throw a
ShardLockObtainFailedException. This commit changes the behavior to
close the search contexts, likely failing some of the searches. The
sentiment is to prefer restoring availability over trying to complete
those searches. A situation where this can happen is when master(s) are
restarted, which is likely to cause similar search issues anyway.
This speeds up the `terms` agg in a very specific case:
1. It has no child aggregations
2. It has no parent aggregations
3. There are no deleted documents
4. You are not using document level security
5. There is no top level query
6. The field has global ordinals
7. There are less than one thousand distinct terms
That is a lot of restirctions! But the speed up pretty substantial because
in those cases we can serve the entire aggregation using metadata that
lucene precomputes while it builds the index. In a real rally track we
have we get a 92% speed improvement, but the index isn't *that* big:
```
| 90th percentile service time | keyword-terms-low-cardinality | 446.031 | 36.7677 | -409.263 | ms |
```
In a rally track with a larger index I ran some tests by hand and the
aggregation went from 2200ms to 8ms.
Even though there are 7 restrictions on this, I expect it to come into
play enough to matter. Restriction 6 just means you are aggregating on
a `keyword` field. Or an `ip`. And its fairly common for `keyword`s to
have less than a thousand distinct values. Certainly not everywhere, but
some places.
I expect "cold tier" indices are very very likely not to have deleted
documents at all. And the optimization works segment by segment - so
it'll save some time on each segment without deleted documents. But more
time if the entire index doesn't have any.
The optimization builds on #68871 which translates `terms` aggregations
against low cardinality fields with global ordinals into a `filters`
aggregation. This teaches the `filters` aggregation to recognize when
it can get its results from the index metadata. Rather, it creates the
infrastructure to make that fairly simple and applies it in the case of
the queries generated by the terms aggregation.
When we test the REST actions we assumed that they'd produce a result
but one of the mocking/verification mechanisms didn't. This forces it to
produce a result. It uses some generics dancing to force the calling
code to mock things with the appropriate type even though we don't its
only a compile time guarantee. So long as callers aren't rude its safe.
This commit adds leak tracking infrastructure that enables assertions
about the state of objects at GC time (simplified version of what Netty
uses to track `ByteBuf` instances).
This commit uses the infrastructure to improve the quality of leak
checks for page recycling in the mock nio transport (the logic in
`org.elasticsearch.common.util.MockPageCacheRecycler#ensureAllPagesAreReleased`
does not run for all tests and tracks too little information to allow for debugging
what caused a specific leak in most cases due to the lack of an equivalent of the added
`#touch` logic).
Co-authored-by: David Turner <david.turner@elastic.co>
Makes sure that we assert every expected message in this test, because
if we don't then we might shut the appender down too early.
Reverts #68267Closes#66630
This change adds query parameter confirming that we accept ToS of GeoIP database service provided by Infra.
It also changes integration test to use lower timeout when using local fixture.
Relates to #68920
This change adds component that will download new GeoIP databases from infra service
New databases are downloaded in chunks and stored in .geoip_databases index
Downloads are verified against MD5 checksum provided by the server
Current state of all stored databases is stored in cluster state in persistent task state
Relates to #68920
This adds best effort `toString` rendering the various wrapping
action listeners to make `TRACE` logging, that will currently only print the
top level listener `toString` which isn't helpful to find the original of a listener
in case of wrapped listeners, more useful (e.g. when logging rejected executions).
Also this change makes the `delegateX` methods less verbose to use and makes use of them
in a few spots where they weren't yet used.
We can't enable the sequential access optimization for stored fields of
changes snapshots used in peer recoveries because they are accessed by
multiple threads.
Relates to #68961
Opening a Lucene index that supports soft-deletes currently creates the liveDocs bitset eagerly. This requires scanning
the doc values to materialize the liveDocs bitset from the soft-delete doc values. In order for searchable snapshot shards
to be available for searches as quickly as possible (i.e. on recovery, or in case of FrozenEngine whenever a search comes
in), they should read as little as possible from the Lucene files.
This commit introduces a LazySoftDeletesDirectoryReaderWrapper, a variant of Lucene's
SoftDeletesDirectoryReaderWrapper that loads the livedocs bitset lazily on first access. It is special-tailored to
ReadOnlyEngine / FrozenEngine as it only operates on non-NRT readers.
This commit makes a start on moving runtime fields to server.
The runtime field types are now built-in. The dynamic fields builder extension (needed to make dynamic:runtime work) is removed: further simplifications are needed but are left for a follow-up. It is still possible to plug in custom runtime field types through the same extension point that the xpack plugin used to use.
The runtime fields script contexts are now also moved to server, and the corresponding whitelists are part of painless-lang with the exception of grok and dissect which will require additional work.
The runtime fields xpack plugin still exists to hold the grok and dissect extensions needed, which depend on grok and dissect lib and we will address as a follow-up. Also, the qa tests and yaml tests are yet to be moved.
Despite the need for additional work, runtime fields are fully functional and users should not be impacted by this change.
This adds yet another terms aggregator that uses `term` filters to run
in similar speed to specialized low cardinality terms aggregator. It is
mostly useful as a stepping stone for some additional optimizations that
we can make later. So it doesn't have to be faster on its own. Just not
*slower*. And its about the same speed.
Currently in the abstract simple transport test case the response
handlers are added after the request is submitted. This introduces a
race condition where the handler could be executed on the test thread
(if the request is finished before the handler is added). This commit
resolves the issue by adding the handlers first.
Currently, the value fetcher framework handles ignored fields by reading
the stored values of the _ignored metadata field, and passing these through
on calls to fetchValues(). However, this means that if a document has multiple
values indexed for a field, and one malformed value, then the fields API will
ignore everything, including the valid values, and return an empty list for this
document.
If a document source contains a malformed value, then it must have been
ignored at index time. Therefore, we can safely assume that if we get an
exception parsing values from source at fetch time, they were also ignored
at index time and they can be skipped. This commit moves this exception
handling directly into SourceValueFetcher and ArraySourceValueFetcher,
removing the need to inspect the _ignored metadata and fixing the case
of mixed valid and invalid values.
In order to support compatible fields when parsing XContent additional information has to be set during ParsedField declaration.
This commit adds a set of RestApiCompatibleVersion on a ParsedField in order to specify on which versions a field is supported. By default ParsedField is allowed to be parsed on both current and previous major versions.
ObjectParser - which is used for constructing objects using 'setters' - has a modified fieldParsersMap to be Map of Maps. with key being RestApiCompatibility. This allows to choose set of field-parsers as specified on a request.
Under RestApiCompatibility.minimumSupported key, there is a map that contains field-parsers for both previous and current versions.
Under RestApiCompatibility.current there will be only current versions field (compatible fields not a present)
ConstructingObjectParser - which is used for constructing objects using 'constructors' - is modified to contain a map of Version To constructorArgInfo , declarations of fields to be set on a constructor depending on a version
relates #51816
The response to an `IndicesSegmentsAction` might be large, perhaps 10s
of MBs of JSON, and today it is serialized on a transport thread. It
also might take so long to respond that the client times out, resulting
in the work needed to compute the response being wasted.
This commit introduces the `DispatchingRestToXContentListener` which
dispatches the work of serializing an `XContent` response to a
non-transport thread, and also makes `TransportBroadcastByNodeAction`
sensitive to the cancellability of its tasks.
It uses these two features to make the `RestIndicesSegmentsAction`
serialize its response on a `MANAGEMENT` thread, and to abort its work
more promptly if the client's channel is closed before the response is
sent.
Today we represent block IDs sent to Azure using the URL-safe base-64
encoding. This makes sense: these IDs appear in URLs. It turns out that
Azure rejects this encoding for block IDs and instead demands that they
are represented using the regular, URL-unsafe, base-64 encoding instead,
then further wrapped in %-encoding to deal with the URL-unsafe
characters that inevitably result.
Relates #66489
This changes adds a new QA test that runs a smoke test on a node that
has been configured with one realm of each type.
Not all of the realms work, because some of them would depend on
external fixtures (LDAP, SAML, etc) and this particularly test suite
is intended to be as stable as possible and have no external
dependencies.
The primary purpose of this test is to catch any issues that prevent
a node from starting with particular realms configurd (e.g. security
manager or classpath issues). We don't depend on external fixtures
becaused we want this to be a smoke test that clearly indicates when a
(seemingly unrelated) change in Elasticsearch has unintended
consequences on realms. The use of external dependencies would
increase the number of things that could go wrong and move this from a
smoke test to a potentially noisy integration test.
This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.
This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):
```
POST /_snapshot/my_repository/snapshot_2/_restore
{
"indices": "index_1",
"include_global_state": true,
"feature_states": ["watcher", "security"]
}
```
If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.
The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```
which returns a response of the form:
```
{
"features": [
{
"name": "tasks",
"description": "Manages task results"
},
{
"name": "kibana",
"description": "Manages Kibana configuration and reports"
}
]
}
```
Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.
Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
{
"feature_name": "tasks",
"indices": [
".tasks"
]
}
],
"indices": [
".tasks",
"test1",
"test2"
]
```
Co-authored-by: William Brafford <william.brafford@elastic.co>
This partially reverts #64016 and and adds #67839 and adds
additional tests that would have caught issues with the changes
in #64016. It's mostly Nik's code, I am just cleaning things up
a bit.
Co-authored-by: Nik Everett <nik9000@gmail.com>
A common pattern today is to set up a sequence of `StepListener` objects
which ultimately notify an outer `ActionListener`. Today we do this as
follows:
step.whenComplete(listener::onResponse, listener::onFailure);
Since this is such a common pattern, this commit exposes a method that
that adds the listener directly.
As of #53261 today `PlainListenableActionFuture` is the sole
implementation of `ListenableActionFuture`, the reason for the
`Plain...` prefix is now lost. It also heavily uses variable shadowing
in its implementation which makes it quite hard to read, uses a mix of
`volatile` fields and mutexes, and keeps hold of the deferred listeners
even after completion. Finally it does not have many tests.
This commit drops the unnecessary interface and renames the class to
simply `ListenableActionFuture`. It reworks the implementation to avoid
shadowing variables, drops the deferred listeners on completion, and
moves to an entirely mutex-based implementation. Finally, it adds
another test that it works as advertised.
Part 10 (and hopefully the last one).
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.
We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
Part 9.
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.
We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
DocumentMapperParser parses xcontent mappings into a DocumentMapper, through DocumentMapper.Builder. Most of the work is done to construct a Mapping instance, that then gets provided to the DocumentMapper.
Moving forward, we would like to rely more on Mapping and less on the entire DocumentMapper, but currently it is cumbersome to create Mapping without creating an instance of DocumentMapper.
This commit removes DocumentMapperParser and DocumentMapper.Builder in favor of a new class called MappingParser that given xcontent mappings returns a Mapping instance, which can be used on its own or provided to DocumentMapper at its construction time. This will help with further refactorings as well as to possibly have more tests that don't rely on the entire MapperService/DocumentMapper but rather only on the needed components.
This reverts commit 5fe0d67ade.
I benchmarked an improvement in CCR and found that changes introduced in
#67190 made CCR 10 times slower (1816 seconds to 26515 seconds).
"total_read_remote_exec_time_millis" : 26515182
"total_read_remote_exec_time_millis" : 1816094
With concurrent indexing and Lucene merges, documents in segments are no
longer sorted by sequence numbers. And if the index sorting is
specified, documents are never sorted by sequence numbers. Using a
mergeInstance stored field reader will decompress the whole block, which
we will not use.
Reverts #67190
At the moment, the ‘fields’ API handles nested fields the same way I handles non-nested object arrays: it just returns them in a flat list. However, the relationship between nested fields is something we should try to preserve, since this is the main purpose of mapping something as “nested” instead of just using an object.
This PR changes this by returning grouped field values that are inside a nested object according to the nested object they initially appear in. Any further object structures inside a nested object are again returned as a flattened list. Fields inside nested fields don’t appear in the flattened response outside of the nested path any more. The grouping of fields inside nested objects is applied recursively if nested mappings are defined inside another nested mapping.
Closes#63709
A frozen tier is backed by an external object store (like S3) and caches only a
small portion of data on local disks. In this way, users can reduce hardware
costs substantially for infrequently accessed data. For the frozen tier we only
pull in the parts of the files that are actually needed to run a given search.
Further, we don't require the node to have enough space to host all the files.
We therefore have a cache that manages which file parts are available, and which
ones not. This node-level shared cache is bounded in size (typically in relation
to the disk size), and will evict items based on a LFU policy, as we expect some
parts of the Lucene files to be used more frequently than other parts. The level
of granularity for evictions is at the level of regions of a file, and does not
require evicting full files. The on-disk representation that was chosen for the
cold tier is not a good fit here, as it won't allow evicting parts of a file.
Instead we are using fixed-size pre-allocated files and have implemented our own
memory management logic to map regions of the shard's original Lucene files onto
regions in these node-level shared files that are representing the on-disk
cache.
This PR adds the core functionality to searchable snapshots to power such a
frozen tier:
- It adds the node-level shared cache that evicts file regions based on a LFU
policy
- It adds the machinery to dynamically download file regions into this cache and
serve their contents when searches execute.
- It extends the mount API with a new parameter, `storage`, which selects the
kind of local storage used to accelerate searches of the mounted index. If set
to `full_copy` (default, used for cold tier), each node holding a shard of the
searchable snapshot index makes a full copy of the shard to its local storage.
If set to `shared_cache`, the shard uses the newly introduced shared cache,
only holding a partial copy of the index on disk (used for frozen tier).
Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
Co-authored-by: Armin Braun <me@obrown.io>
Co-authored-by: David Turner <david.turner@elastic.co>
Prevent failures of the test framework in mixed version tests
when different Java versions report the name of a particular
OS version differently.
Fixes#68346
Adds a multi_terms aggregation support. The multi terms aggregation works
very similarly to the terms aggregation but supports multiple terms. The goal
of this PR is to add the basic functionality so it is not optimized at the
moment. It will be done in follow up PRs.
Closes#65623
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
This PR removes support for index metadata upgrades:
* Stop using the `index.version.upgrade` setting and deprecate it.
* Remove `MetadataIndexUpgradeService` and other references to upgrades.
In addition to supporting upgrades, `MetadataIndexUpgradeService` verified
certain aspects of the metadata, like index version compatibility. This logic
is important to keep, so `MetadataIndexUpgradeService` was reworked to
`IndexMetadataVerifier` instead of being removed completely.
Closes#66143.
when parsing xContent objects named object can be registered and need to
be parsed as per N-1 rules.
named objects are declared on a parser (PARSER.declareNamedObject) and
the corresponding parsing logic is registered in NamedObjectRegistry.
There is a need to use an old N-1 "configuration" of the registry in
order to support the compatible parsing logic.
This commit extends the NamedObjectRegistry with additional set of
compatible entries, which provide the compatible parsing logic. Those
entries are then in turn used in parseNamedObject when xContentParser is
using compatibility.
relates #51816
A blob store repository can be put in readonly mode by setting
`readonly: true` in its settings. In the codebase the setting key is
just the literal string `"readonly"` wherever it's used and it takes
some effort to determine what the right setting name is, in particular
to check each time that it's not spelled `"read_only"`.
This commit replaces those literal `"readonly"` strings with the
`BlobStoreRepository#READONLY_SETTING_KEY` constant to reduce this
trappiness.
This drops the `-Xlint:-rawtypes` suppression from the test framework
build because it makes it easier to write code with "funny" assumptions.
For the most part I made the assumptions explicit rather than implicit.
Some of them are still "funny", but at least they are a little more
obvious when you read the code.
I also had to touch a pile of agg tests due to a signature that had a raw
type as well.
SearchLookup has two intermediate classes, DocMap and StoredFieldsLookup, that
are simple factories for their Leaf implementations. They are never accessed
outside SearchLookup, with the exception of two calls on DocMap that can be
easily refactored. This commit removes them, making SearchLookup.getLeafSearchLookup
directly responsible for creating the leaf lookups.
Currently the logic for parsing vendor specific headers - i.e. application/vnd.elasticsearch+json;compatible-with=7 is within rest-compatibility module.
This commit is removing the rest-compatibility plugin and moving the version parsing logic
into server module. It no longer needs to be injected from xpack into server, therefore can be directly used from RestRequest.
relates #51816
Record the clusterUUID of the last cluster to write
to a repository in the `RepositoryData` and use it for more
meaningful logging when running into a concurrent modification
issue.
Today if you upgrade a cluster containing invalid cluster settings and a
read-only block then you are stuck: you cannot remove the now-archived
settings because of the read-only block, but you cannot set any settings
to remove the read-only block because of the archived settings.
This commit adds functionality to permit removing both the read-only
block and the archived settings in a single step.
Closes#62920
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.
We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
The cardinality agg delays calculating stuff until just before it is
needed. Before #64016 it used the `postCollect` phase to do this work
which was perfect for the `terms` agg but we decided that `postCollect`
was dangerous because some aggs, notably the `parent` and `child` aggs
need to know which children to build and they *can't* during
`postCollect`. After #64016 we built the cardinality agg results when we
built the buckets. But we if you sort on the cardinality agg then you
need to do the `postCollect` stuff in order to know which buckets
to build! So you have a chicken and egg problem. Sort of.
This change splits the difference by running the delayed cardinality agg
stuff as soon as you *either* try to build the buckets *or* read the
cardinality for use with sorting. This works, but is a little janky and
feels wrong. It feels like we could make a structural fix to the way we
read metric values from aggs before building the buckets that would make
this sort of bug much more difficult to cause. But any sort of solution
to this is a larger structural change. So this fixes the bug in the
quick and janky way and we hope to do a more structural fix to the way
we read metrics soon.
Closes#67782
This fixes a left over from #65042 where if a clone
was initially put in the cluster state before another clone
but then initialized after the other clone already finished its shard clones,
a broken snapshot state would be reached where the first clone
would be queued before the almost finished clone.
closes#67871
Today a snapshot repository does not have a well-defined identity. It
can be reregistered with a different cluster under a different name, and
can even be registered with multiple clusters in readonly mode.
This presents problems for cases where we need to refer to a specific
snapshot in a globally-unique fashion. Today we rely on the repository
being registered under the same name on every cluster, but this is not a
safe assumption.
This commit adds a UUID that can be used to uniquely identify a
repository. The UUID is stored in the top-level index blob, represented
by `RepositoryData`, and is also usually copied into the
`RepositoryMetadata` that represents the repository in the cluster
state. The repository UUID is exposed in the get-repositories API; other
more meaningful consumers will be added in due course.
A trap for the uninitiated: only the master should be calling
`getRepositoryData()`, but today this isn't checked anywhere so there's
a risk that we inadvertently introduce some code that gets the
repository data on other nodes too. This commit introduces an assertion
to catch that.
Second attempt at #67780 which was reverted due to test failures.
This PR caused test failures after merge, one of which was subsequently
muted. It's quite possible that this isn't the only test that's now
broken so this commit reverts it (and the muting) pending further
investigation.
This reverts commit e8ed1b441b.
This reverts commit b467595564.
A trap for the uninitiated: only the master should be calling
`getRepositoryData()`, but today this isn't checked anywhere so there's
a risk that we inadvertently introduce some code that gets the
repository data on other nodes too. This commit introduces an assertion
to catch that.
Closes#64824. Introduce the concept of categories to deprecation
logging. Every location where we log a deprecation message must now
include a deprecation category.
- Port UrlFixture to test fixture plugin
- Avoid exposing PID and PORt for http fixture when not required
- Make AbstractHttpFixture work inside and outside docker
- Check directories when running UrlFixture
Today when testing loggers we use `MockLogAppender` which is configured
to ignore exceptions encountered during logging. These exceptions are
captured by the `StatusLogger` and ultimately cause tests to fail, but
the failure lacks the context we need to diagnose the problem.
This commit sets the `ignoreExceptions` flag to `false` on all
`MockLogAppender` objects so that exceptions are thrown to the caller
for a more informative debugging experience.
Relates #66630
Relates #67117
We decided to rename `QueryShardContext` to clarify that it supports all parts
of search request execution. Before there was confusion over whether it should
only be used for building queries, or maybe only used in the query phase. This
PR also updates the javadocs.
Closes#64740.
Part of the fixes for #66419, this commit permits nodes to emit the
deprecation warning regarding not specifying `?wait_for_active_shards`
when closing an index in 7.x versions for x ≥ 12. This change is
required on `master` too since the BWC tests encounter these warnings.
Relates #67246, which is the 7.x part of this change.
Add a getOutputFieldNames method to aggregationbuilder to retrieve the names of the fields the
aggregation produces. Implementation is optional and provided for a set of multi value aggregations
as a start.
This uses the mapping snapshot that we built for the search phase
in #66295 for fetching nested documents. This is simpler to reason about
because the mapping snapshot is immutable.
The selective muting implemented for autoscaling in #67159
is extended to the ML tests that also fail when machine
memory is reported as 0.
Most of the logic to determine when memory will not be
accurately reported is now in a utility method in the
base class.
Relates #66885
Today if a client requests a cluster state and then closes the
connection then we still do all the work of computing and serializing
the cluster state before finally dropping it all on the floor.
With this commit we introduce checks to make sure that the HTTP channel
is still open before starting the serialization process. We also make
the tasks themselves cancellable and abort any ongoing waiting if the
channel is closed (mainly to make the cancellability testing easier).
Finally we introduce a more detailed description of the task to help
identify cases where clients are inefficiently requesting more
components of the cluster state than they need.
As part of #66295 we made QueryShardContext perform mapping lookups through MappingLookup rather than MapperService. That helps as MapperService relies on DocumentMapper which may change througout the execution of the search request. At search time, the percolate query also needs to parse documents, which made us add a parse method to MappingLookup.Such parse method currently relies on calling DocumentMapper#parseDocument through a function, but we would like to rather make this easier to follow. (see https://github.com/elastic/elasticsearch/pull/66295/files#r544639868)
We recently removed the need to provide the entire DocumentMapper to DocumentParser#parse, opening the possibility for using DocumentParser directly when needing to parse a document at query time. This commit adds everything that is needed (namely Mapping, IndexSettings and IndexAnalyzers) to MappingLookup so that it can parse a document through DocumentParser without relying on DocumentMapper.
As a bonus, given that MappingLookup holds a reference to these three additional objects, we can make DocumentMapper rely on MappingLookup to retrieve those and not hold its own same references to them.
Along the same lines, given that MappingLookup holds all that's necessary to parse a document, the signature of DocumentParser#parse can be simplified by replacing most of its arguments with MappingLookup and retrieving what is needed from it.
Fixes a bug where nested documents that match a filter in the `filters`
agg will be counted as matching the filter. Usually nested documents
only match if you explicitly ask to match them. Worse, we only mach them
in the "filter by filter" mode that we wrote to speed up date_histogram.
The `filters` agg is fairly rare, but with #63643 we run
`date_histogram` and `range` aggregations using `filters.
VND types like application/vnd.elasticsearch+json
(XContentType.VND_JSON) when compared with
application/json (XContentType.JSON) should use `canonnical`
closes#67064
A number of metadata field mapper tests are still extending ESSingleNodeTestCase.
This commit changes them to use MapperServiceTestCase instead, and adds a new
base class for those metadata fields that have parameters subject to conflict checks.
We were copying any byte arrays we would read streams that are backed by
a `BytesReference`. This would mean copying e.g. 500k chunks during
file based recovery, large chunks during CCR file copy, or many MB when
receiving a large cluster state for the first time.
This PR adds the ability to read shared bytes from streams that support it,
adds reference counting to transport messages that make use of these bytes
and makes use of the new functionality in recovery, CCR and CS publication.
This should reduce memory usage by all 3 adjusted transport APIs significantly
and could be extended to other very large messages that currently get copied
during deserialization like e.g. bulk requests.
When reading sequential data from compressed stored fields it is
more efficient to use a CodecReader's merge instance, as this holds
decompressed blocks in memory and avoids re-doing the
decompression for every document. When restoring the version map
during a recovery, we read id fields from all documents beyond the global
checkpoint, and this is precisely the situation where this performance
enhancement should help.
This commit updates the IdOnlyFieldsVisitor to use a merge instance,
and in the process tidies up the API a bit; we now have a package-private
IdStoredFieldLoader that handles fetching merge instances and loading
ids for each document.
Autoscaling would respond with no memory info in current capacity and
ML non-scaling events. However, this does not play nicely with trying
to short circuit the decision made by autoscaling capacity API clients,
it is reasonable to compare required to current to determine if a
scale up is necessary.
This commit adds a service that asks all nodes for their info and
ensures that current capacity includes this information when available.
This commit reworks the InternalClusterInfoService to run
asynchronously, using timeouts on the stats requests instead of
implementing its own blocking timeouts. It also improves the logging of
failures by identifying the nodes that failed or timed out. Finally it
ensures that only a single refresh is running at once, enqueueing later
refresh requests to run immediately after the current refresh is
finished rather than racing them against each other.
This removes the return values from `CircuitBreaker`'s
`addEstimateBytesAndMaybeBreak` and `addWithoutBreaking` because they
are not supported by the new pre-allocating `CircuitBreaker` and they
aren't used for anything other than tests.
This commit allows returning a correct requested response content-type - it did not work for versioned media types.
It is done by adding new vendor specific instances to XContent and TextFormat enums. These instances can then "format" the response content type string when provided with parameters. This is similar to what SQL plugin does with its media types.
#51816
This lowers the contention on the `REQUEST` circuit breaker when building
many aggregations on many threads by preallocating a chunk of breaker
up front. This cuts down on the number of times we enter the busy loop
in `ChildMemoryCircuitBreaker.limit`. Now we hit it one time when building
aggregations. We still hit the busy loop if we collect many buckets.
We let the `AggregationBuilder` pick size of the "chunk" that we
preallocate but it doesn't have much to go on - not even the field types.
But it is available in a convenient spot and the estimates don't have to
be particularly accurate.
The benchmarks on my 12 core desktop are interesting:
```
Benchmark (breaker) Mode Cnt Score Error Units
sum noop avgt 10 1.672 ± 0.042 us/op
sum real avgt 10 4.100 ± 0.027 us/op
sum preallocate avgt 10 4.230 ± 0.034 us/op
termsSixtySums noop avgt 10 92.658 ± 0.939 us/op
termsSixtySums real avgt 10 278.764 ± 39.751 us/op
termsSixtySums preallocate avgt 10 120.896 ± 16.097 us/op
termsSum noop avgt 10 4.573 ± 0.095 us/op
termsSum real avgt 10 9.932 ± 0.211 us/op
termsSum preallocate avgt 10 7.695 ± 0.313 us/op
```
They show pretty clearly that not using the circuit breaker at all is
faster. But we can't do that because we don't want to bring the node
down on bad aggs. When there are many aggs (termsSixtySums) the
preallocation claws back much of the performance. It even helps
marginally when there are two aggs (termsSum). For a single agg (sum)
we see a 130 nanosecond hit. Fine.
But these values are all pretty small. At best we're seeing a 160
microsecond savings. Not so on a 160 vCPU machine:
```
Benchmark (breaker) Mode Cnt Score Error Units
sum noop avgt 10 44.956 ± 8.851 us/op
sum real avgt 10 118.008 ± 19.505 us/op
sum preallocate avgt 10 241.234 ± 305.998 us/op
termsSixtySums noop avgt 10 1339.802 ± 51.410 us/op
termsSixtySums real avgt 10 12077.671 ± 12110.993 us/op
termsSixtySums preallocate avgt 10 3804.515 ± 1458.702 us/op
termsSum noop avgt 10 59.478 ± 2.261 us/op
termsSum real avgt 10 293.756 ± 253.854 us/op
termsSum preallocate avgt 10 197.963 ± 41.578 us/op
```
All of these numbers are larger because we're running all the CPUs
flat out and we're seeing more contention everywhere. Even the "noop"
breaker sees some contention, but I think it is mostly around memory
allocation. Anyway, with many many (termsSixtySums) aggs we're looking
at 8 milliseconds of savings by preallocating. Just by dodging the busy
loop as much as possible. The error in the measurements there are
substantial. Here are the runs:
```
real:
Iteration 1: 8679.417 ±(99.9%) 273.220 us/op
Iteration 2: 5849.538 ±(99.9%) 179.258 us/op
Iteration 3: 5953.935 ±(99.9%) 152.829 us/op
Iteration 4: 5763.465 ±(99.9%) 150.759 us/op
Iteration 5: 14157.592 ±(99.9%) 395.224 us/op
Iteration 1: 24857.020 ±(99.9%) 1133.847 us/op
Iteration 2: 24730.903 ±(99.9%) 1107.718 us/op
Iteration 3: 18894.383 ±(99.9%) 738.706 us/op
Iteration 4: 5493.965 ±(99.9%) 120.529 us/op
Iteration 5: 6396.493 ±(99.9%) 143.630 us/op
preallocate:
Iteration 1: 5512.590 ±(99.9%) 110.222 us/op
Iteration 2: 3087.771 ±(99.9%) 120.084 us/op
Iteration 3: 3544.282 ±(99.9%) 110.373 us/op
Iteration 4: 3477.228 ±(99.9%) 107.270 us/op
Iteration 5: 4351.820 ±(99.9%) 82.946 us/op
Iteration 1: 3185.250 ±(99.9%) 154.102 us/op
Iteration 2: 3058.000 ±(99.9%) 143.758 us/op
Iteration 3: 3199.920 ±(99.9%) 61.589 us/op
Iteration 4: 3163.735 ±(99.9%) 71.291 us/op
Iteration 5: 5464.556 ±(99.9%) 59.034 us/op
```
That variability from 5.5ms to 25ms is terrible. It makes me not
particularly trust the 8ms savings from the report. But still,
the preallocating method has much less variability between runs
and almost all the runs are faster than all of the non-preallocated
runs. Maybe the savings is more like 2 or 3 milliseconds, but still.
Or maybe we should think of hte savings as worst vs worst? If so its
19 milliseconds.
Anyway, its hard to measure how much this helps. But, certainly some.
Closes#58647
Since we introduced (#65583) we are storing the timestamp range in the
mappings resolution so the qualifier can be misleading, for that
reason we rename all related variables to remove any references
to milliseconds.
LuceneChangesSnapshot currently takes a field type lookup lambda so that
it can call `FieldsVisitor#postProcess()` once it has read a document from
its lucene index. However, this `postProcess()` call is unnecessary - the only
fields loaded for the document are `_source`, `_id` and `_routing`, none of
which require post-processing. Deleting this call allows us to remove the
fieldtype lookup from a number of places, with the result that various Engine
tests no longer need to create a MapperService (quite a heavy operation) to
test the transaction log.
This finishes porting all tasks created in gradle build scripts and plugins to use
the task avoidance api (see #56610)
* Port all task definitions to task avoidance api
* Fix last task created during configuration
* Fix test setup in :modules:reindex
* Declare proper task inputs
The client-side encrypted repository is a new type of snapshot repository that
internally delegates to the regular variants of snapshot repositories (of types
Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted
repository is set up, it is transparent to the snapshot and restore APIs (i.e. all
snapshots stored in the encrypted repository are encrypted, no other parameters
required).
The encrypted repository is protected by a password stored on every node's
keystore (which must be the same across the nodes).
The password is used to generate a key encrytion key (KEK), using the PBKDF2
function, which is used to encrypt (using the AES Wrap algorithm) other
symmetric keys (referred to as DEK - data encryption keys), which themselves
are generated randomly, and which are ultimately used to encrypt the snapshot
blobs.
For example, here is how to set up an encrypted FS repository:
------
1) make sure that the cluster runs under at least a "platinum" license
(simplest test configuration is to put `xpack.license.self_generated.type: "trial"`
in the elasticsearch.yml file)
2) identical to the un-encrypted FS repository, specify the mount point of the
shared FS in the elasticsearch.yml conf file (on all the cluster nodes),
e.g. `path.repo: ["/tmp/repo"]`
3) store the repository password inside the elasticsearch.keystore, *on every cluster node*.
In order to support changing password on existing repository (implemented in a follow-up),
the password itself must be names, e.g. for the "test_enc_key" repository password name:
`./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password`
*type in the password*
4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling:
`
curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d'
{
"type": "encrypted",
"settings": {
"location": "/tmp/repo/enc",
"delegate_type": "fs",
"password_name": "test_enc_pass"
}
}
'
`
5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g.
` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"`
Related: #49896#41910#50846#48221#65768
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.
This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.
It also addresses a number of tests that would fail in approved only mode
Mainly:
Tests that use PBKDF2 with a password less than 112 bits (14char). We
elected to change the passwords used everywhere to be at least 14
characters long instead of mandating
the use of pbkdf2_stretch because both pbkdf2 and
pbkdf2_stretch are supported and allowed in fips mode and it makes sense
to test with both. We could possibly figure out the password algorithm used
for each test and adjust password length accordingly only for pbkdf2 but
there is little value in that. It's good practice to use strong passwords so if
our docs and tests use longer passwords, then it's for the best. The approach
is brittle as there is no guarantee that the next test that will be added won't
use a short password, so we add some testing documentation too.
This leaves us with a possible coverage gap since we do support passwords
as short as 6 characters but we only test with > 14 chars but the
validation itself was not tested even before. Tests can be added in a followup,
outside of fips related context.
Tests that use a PKCS12 keystore and were not already muted.
Tests that depend on running test clusters with a basic license or
using the OSS distribution as FIPS 140 support is not available in
neither of these.
Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
This makes sure that we only serve a hit from the request cache if it
was build using the same mapping and that the same mapping is used for
the entire "query phase" of the search.
Closes#62033
ESRestTestCase rest clients could only be configured to trust
the certificate authorities that were contained in a truststore. In
certain cases (like in fips mode where JKS/PKCS12 keystores) cannot
be used, it's beneficial to be able to trust specific certificate
authorities (indicated by the CA PEM endoded certificate)
The commit id introduced in #63963 does not work well with searchable
snapshots as we create a new index commit when restoring from snapshots.
This change revises an approach that generates an id using the ids of the
segments of an index commit.
Relates #63963
This commit introduces a new sort field called `_shard_doc` that
can be used in conjunction with a PIT to consistently tiebreak
identical sort values. The sort value is a numeric long that is
composed of the ordinal of the shard (assigned by the coordinating node)
and the internal Lucene document ID. These two values are consistent within
a PIT so this sort criteria can be used as the tiebreaker of any search
requests.
Since this sort criteria is stable we'd like to add it automatically to any
sorted search requests that use a PIT but we also need to expose it explicitly
in order to be able to:
* Reverse the order of the tiebreaking, useful to search "before" `search_after`.
* Force the primary sort to use it in order to benefit from the `search_after` optimization when sorting by index order (to be released in Lucene 8.8.
I plan to add the documentation and the automatic configuration for PIT in a follow up since this change is already big.
Relates #56828
Except when writing actual segment files to the blob store
we always write `BytesReference` instead of a stream.
Only having the stream API available forces needless copies
on us. I fixed the straight-forward needless copying for
HDFS and FS repos in this PR, we could do similar fixes for
GCS and Azure as well and thus significantly reduce the peak
memory use of these writes on master nodes in particular.
Drops `AggregatorTestCase#mapperServiceMock` because it is getting in
the way of other work I'm doing for runtime fields. It was only
overridden to test the `parent` and `child` aggregation to add the
`MappedFieldType`s for join fields in the backdoor. Those aggregations
can just as easily add those fields in the normal method calls.
Running multiple hdfs fixtures in parallel for running integration tests requires
a dynamic port assignment in order to avoid port clashes. This introduces
the ability to assign port ranges to gradle projects that can be used
to allocate dynamically ports used by these projects.
We apply this dynamic port setup for hdfs fixtures used in
:x-pack:plugin:searchable-snapshots:qa only at the moment as
tests sources (rest tests) in :plugins:repository-hdfs still rely on
hard coded ports.
This is a simplified version of fixtures I created before on the gradle codebase
to deal with similar issues.
Fixes#66377
This cleans up some tests around aggregations, specifically because I
bumped into them while working on something else and was quite confused.
This removes support for passing `null` `MappedFieldTypes` to
`AggregatorTestCase` and instead passes an empty array.
It also removes `getFieldAliases` from `AggregatorTestCase` and instead
aliases all fields to `<name>-alias` to encourage everyone to test
aliases all the time. We override it several times, always copy and
pasting the code. Node we just do that by default.
The proactive storage decider looks at past indexing in data streams and
calculates a required capacity that ensures that the set of nodes
governed by the policy can handle adding a similar amount of data. The
time looked back is called the forecast window and defaults to 30
minutes.
This initial version only ensures that enough storage is available for
primaries.
Luca Cavanna
David Turner
Armin Braun
Adrien Grand
Alan Woodward
Hendrik Muhs
Jay Modi
Nik Everett
Henning Andersen
Przemyslaw Gomulka
Dan Hermann
Igor Motov
Julie Tibshirani
Jake Landis
Jason Tedor
Lyudmila Fokina
Tanguy Leroux
Nhat Nguyen
Przemko Robakowski
Yannick Welsch
Francisco Fernández Castaño
Christoph Büscher
Ryan Ernst
Yash Jipkate
Ignacio Vera
Mark Vieira
William Brafford
Benjamin Trent
Varsha Muzumdar
Martijn van Groningen
Jim Ferenczi
Joe Gallo
Tim Brooks
Tim Vernum
Gordon Brown
Rory Hunter
David Kyle
David Roberts
Andrei Dan
Howard
Rene Groeschke
Mark Tozzi
Albert Zaharovits
Ioannis Kakavas