Commit Graph

2893 Commits

Author SHA1 Message Date
Armin Braun 096b8ccc26
Fix TextFieldMapper Retaining a Reference to its Builder (#77251)
Fixes the text field mapper and the analyzers class that also retained parameter references that go really heavy.
Makes `TextFieldMapper` take hundreds of bytes compared to multiple kb per instance.

closes #73845
2021-09-03 18:44:11 +02:00
Jake Landis 26dfe02a0b
Update task names for rest compatiblity (#75267)
This commit updates two task names:
```
yamlRestCompatTest -> yamlRestTestV7CompatTest
transformV7RestTests -> yamlRestTestV7CompatTransform 
```

`7` is the N-1 version and calculated, such that when `8` is 
N-1 version the task names will be  `yamlRestTestV8CompatTest` and 
`yamlRestTestV8CompatTransform`

The motivation for `yamlRestCompatTest -> yamlRestTestV7CompatTest`  is that 
many projects have configured `yamlRestCompatTest` 
but that configuration is specific to the N-1 version. For example, 
if we blacklist tests when running compatibility with v7, we don't also
want to blacklist those tests when running compatibility with v8.

By introducing a version-specific identifier in the name, the task will not
even exist when bumping the version creating the need to (correctly) remove
the version-specific condition.

The motivation for `transformV7RestTests -> yamlRestTestV7CompatTransform` 
is to provide more consistent naming. 

The idea behind the naming is the main task people
are likely familiar with is :

`yamlRestTest` so we will use that as a base.
`yamlRestTestV7CompatTest` to run the version-specific compat tests
`yamlRestTestV7CompatTransform` to run the version-specific transformations for the compat tests

CI should be un-effected since since we introduced a lifecycle task 
name `checkRestCompat` which is what CI should be configured to use.
2021-09-03 11:26:11 -05:00
David Turner bfcc93a042
Anonymize AbstractRefCounted (#77208)
Today `AbstractRefCounted` has a `name` field which is only used to
construct the exception message when calling `incRef()` after it's been
closed. This isn't really necessary, the stack trace will identify the
reference in question and give loads more useful detail besides. It's
also slightly irksome to have to name every single implementation.

This commit drops the name and the constructor parameter, and also
introduces a handy factory method for use when there's no extra state
needed and you just want to run a method or lambda when all references
are released.
2021-09-03 07:59:44 +01:00
Christos Soulios 707dd497e4
Add multiple validators to Parameters (#77073)
This PR implements support for multiple validators to a FieldMapper.Parameter.

The Parameter#setValidator method was replaced by Parameter#addValidator that can be called multipled times
to add validation to a parameter.

All validators of a parameter will be executed in the same order as they have been added and if any of them fails all validation will failed.
2021-08-31 21:28:14 +03:00
Rene Groeschke 35ec6f348c
Introduce simple public yaml-rest-test plugin (#76554)
This introduces a basic public yaml rest test plugin that is supposed to be used by external 
elasticsearch plugin authors. This is driven by #76215

- Rename yaml-rest-test to intern-yaml-rest-test
- Use public yaml plugin in example plugins

Co-authored-by: Mark Vieira <portugee@gmail.com>
2021-08-31 08:45:52 +02:00
Rory Hunter a6f2a4df8b
Introduce Cloud docker variant (#74980)
Closes #74795.

Introduce two Docker image variants for Cloud. The first bundles
(actually installs) the S3, Azure and GCS repository plugins. The
second bundles all official plugins, but only installs the repository
plugins.

Both images also bundle Filebeat and Metricbeat.

The testing utils have been refactored to introduce a `docker`
sub-package. This allows the static `Docker.containerId` to be
shared without needing all the code in one big class. The code for
checking file ownership / permissions has also been refactored to
a more Hamcrest style, using a custom Docker file matcher.
2021-08-20 20:11:05 +01:00
Przemyslaw Gomulka 127015e54c
[Rest Api Compatibility] Clean up blocklist (#76179)
v7compatibilityNotSupportedTests was introduced to make it easier to
track tests that have been identified as not needing compatible changes
and those that still need to be checked.
We have checked all tests now and the separate list is no longer needed.

relates #51816
relates #73912
2021-08-19 10:09:46 +02:00
Rory Hunter d01efa4fd6
Changes to keep Checkstyle happy after reformatting (#76464)
* Reformatting to keep Checkstyle after formatting

* Configure spotless everywhere, and disable the tasks if necessary

* Add XContentBuilder helpers, fix test

* Tweaks

* Add a TODO

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-18 07:15:55 -04:00
Armin Braun b01cd8ff87
Radomize BlobContainer Path in Retries Tests (#76303)
Follow up to #76273 adding some randomization across all retries tests.
2021-08-11 22:31:04 +02:00
Armin Braun da668f9cb0
Refactor Snapshot Finalization Method (#76005)
This refactors the signature of snapshot finalization. For one it allows removing
a TODO about being dependent on mutable `SnapshotInfo` which was not great but
more importantly this sets up a follow-up where state can be shared between the
cluster state update at the end of finalization and subsequent old-shard-generation
cleanup so that we can resolve another open TODO about leaking shard generation files
in some cases.
2021-08-10 16:41:21 +02:00
Armin Braun 734f72714c
Remove Apache Http Client from GCE Discovery Plugin (#76291)
This was fixed for the repository plugin at some point as well.
We don't need the Apache http client here since we're using the
net http one.
=> One less dependency on our system wide http dependency to think about.
2021-08-10 16:30:20 +02:00
Armin Braun 46ebb2298c
Fix S3 Streaming Writes Ignoring Relative Paths for Large Writes (#76273)
It's in the title, we were not accounting for relative paths at all
here and only saved by the fact that we mostly short-circuit to
non-streaming writes.
Extended testing to catch this case for S3 and would do a follow-up
to extend it for the other implementations as well.
2021-08-10 13:36:25 +02:00
Armin Braun 873fbf7b65
Fix Leaking Http Channel Objects when Http Client Stats are Disabled (#76257)
We have to remove the channel from the internal collection of channels when stats are disabled.

Closes #76183
2021-08-10 12:39:12 +02:00
Rene Groeschke b323726ebd
Resolve system properties in build scripts via provider factory (#76199)
This allows tracking system properties used in the build configuration and brings us
one step closer to be gradle configuration cache compliant.
2021-08-09 09:39:30 +02:00
Armin Braun d510fa4841
Upgrade to Netty 4.1.66 (#76135)
Upgrade to 4.1.66 which might bring some helpful
improvements to how TLS exceptions are propagated.
2021-08-05 09:24:52 +02:00
Stuart Tettemer 6c02a6c657
Script: Fields API for Sort and Score scripts (#75863)
Adds minimal fields API support to sort and score scripts.

Example: `field('myfield').getValue(123)` where `123` is the default if the field has no values.

Refs: #61388
2021-08-04 10:11:12 -05:00
Keith Massey 0ae9e77637
Not throwing FileNotFoundException if BlobContainer has been deleted (#75991)
If the underlying directory for an HdfsBlobContainer has been deleted (such as by calling HdfsBlobContainer.delete()) then listBlobsByPrefix() was throwing a FileNotFoundException. This change makes listBlobsByPrefix() return an empty array instead, which is inline with the behavior of FsBlobContainer. It also adds HdfsSnapshotRepoTestKitIT, which runs the repo analyzer against the HDFS repo.
Closes #73708
2021-08-04 08:17:56 -05:00
Francisco Fernández Castaño f21c864634
Take into account base path on AzureBlobStoreRepositoryTests#testNotFoundErrorMessageContainsFullKey (#75934) 2021-08-02 14:17:37 +02:00
Francisco Fernández Castaño 09a5db364b
Include full key in Azure read operations exceptions (#75920)
Closes #75916
2021-08-02 11:38:37 +02:00
Jake Landis 4553bf03d5
Remove MavenFilteringHack (#73637) 2021-07-20 17:05:16 -04:00
Przemyslaw Gomulka 71e05838a6
[Rest Api Compatibility] Enable tests after types and cat api fixed (#75179)
Some tests are fixed after typed api is available with compatible api.
Also cat api returning text fixed some tests
relates #51816
2021-07-14 08:37:38 +02:00
Nhat Nguyen 5fb4c05095
Deprecate SimpleFS and replace it with NIOFS (#75156) (#75196)
SimpleFS is deprecated and will be removed in Lucene 9. This commit
deprecates SimpleFS in 7.x and uses NIOFS for SimpleFS in Elasticsearch
7.15 or later as it offers superior or equivalent performance to
SimpleFS.
2021-07-09 18:22:41 -04:00
Dan Hermann 76ada61d8c
Upgrade Tika to 1.27 for ingest (#75191) 2021-07-09 14:03:02 -05:00
Armin Braun 01872e8ed5
Fix GCS Keystore Handling in FIPS Mode (#75028)
In FIPS mode loading the `.p12` keystore used by the new SDK version is not supported
because of "PBE AlgorithmParameters not available". Fortunately, the SDK still includes
the old jks trust store so we can just manually load it the same way it was loaded by
the previous version to fix things.
Also, fixed `SocketAccess` to properly rethrow this kind of exception and not run into
a class cast issue.

Closes #75023

relates https://github.com/googleapis/google-api-java-client/pull/1738
2021-07-07 15:19:02 +02:00
Luca Cavanna c6641bf00c
Rename ParseContext to DocumentParserContext (#74963)
ParseContext is used to parse documents. It was easily confused with ParserContext (now renamed to MappingParserContext) which is instead used to parse mappings.

To remove any confusion, this commit renames ParseContext to DocumentParserContext and adapts its subclasses accordingly.
2021-07-06 09:15:59 -04:00
Armin Braun d41dfc9903
Upgrade GCS SDK to 1.117.1 (#74938)
We're behind the ugprade schedule by quite a bit here, upgrading to the latest version
and adjusting our test fixture accordingly.
2021-07-05 21:57:30 +02:00
Armin Braun 85f0a029c2
Optimize Away Small Resumable Uploads in GCS Repository (#74813)
Similar to ChunkedOutputStream we can optimize away small resumable uploads
to speed up small meta writes (and improve stability with JDK-8 in 7.x).
2021-07-01 12:38:33 +02:00
Nhat Nguyen 174f65ef82
Upgrade to Lucene 8.9.0 (#74729) 2021-06-30 08:49:53 -04:00
Armin Braun f50ea6a014
Simplify Stacktrace in GCS Streaming Upload Logic (#74716)
Aside from simplifying the code (the close is unnecessary) this change fixes
access control checks breaking on Java 8 in 7.x.
2021-06-30 09:13:19 +02:00
Mark Vieira 4ef48e8db5 Fix thirdPartyAudit task configuration 2021-06-29 15:43:05 -07:00
Armin Braun 8947c1e980
Save Memory on Large Repository Metadata Blob Writes (#74313)
This PR adds a new API for doing streaming serialization writes to a repository to enable repository metadata of arbitrary size and at bounded memory during writing. 
The existing write-APIs require knowledge of the eventual blob size beforehand. This forced us to materialize the serialized blob in memory before writing, costing a lot of memory in case of e.g. very large `RepositoryData` (and limiting us to `2G` max blob size).
With this PR the requirement to fully materialize the serialized metadata goes away and the memory overhead becomes completely bounded by the outbound buffer size of the repository implementation. 

As we move to larger repositories this makes master node stability a lot more predictable since writing out `RepositoryData` does not take as much memory any longer (same applies to shard level metadata), enables aggregating multiple metadata blobs into a single larger blobs without massive overhead and removes the 2G size limit on `RepositoryData`.
2021-06-29 11:29:55 +02:00
Armin Braun cbf48e0633
Flatten Get Snapshots Response (#74451)
This PR returns the get snapshots API to the 7.x format (and transport client behavior) and enhances it for requests that ask for multiple repositories.
The changes for requests that target multiple repositories are:
* Add `repository` field to `SnapshotInfo` and REST response
* Add `failures` map alongside `snapshots` list instead of returning just an exception response as done for single repo requests
* Pagination now works across repositories instead of being per repository for multi-repository requests

closes #69108
closes #43462
2021-06-24 16:58:33 +02:00
Rene Groeschke b79dd52c1b
Cleanup QA projects build scripts (#74428)
Aiming for configuring less during the build,
this removes non required configuration from qa build scripts that do not
contain any sources. We also remove a few non required afterEvaluate hooks
2021-06-23 11:35:47 +02:00
Christoph Büscher 437805b800
Add keyword fields above `ignore_above` to `_ignored` (#74418)
Currently the `_ignore` field indexes and stores the names of every field in a document that has been ignored
because eg. it was malformed. The `ignore_above` option for keyword-type fields
serves a somewhat similar purpose, so this change add logix that  adds these
fields to the "_ignored" field as well for `keyword`, `wildcard` and
`icu_collation_keyword` fields.

Closes #74228
2021-06-22 16:13:22 +02:00
Alan Woodward 7d665616da
Deprecate setting version on analyzers (#74073)
The version field on all lucene Analyzers is unused, and is being removed
in lucene 9. This commit deprecates setting a version on an analyzer in
index settings and removes the related calls to Analyzer.setVersion()

Relates to #74057
2021-06-16 09:40:41 +01:00
Igor Motov d17a834a9e
Upgrade commons-math3 library and fix license and notice files (#74032)
Upgrades commons-math3 library in analytics plugin and updates all
notice and license files to match commons-math3 3.6.1 distribution.
2021-06-14 12:53:33 -10:00
Armin Braun 4fa99f58e2
Remove S3 Eventual Consistency Related Tests (#74015)
S3 list, update etc. are consistent now => no need to have these tests around any longer.
2021-06-10 20:50:28 +02:00
Armin Braun 5249540a5c
Simplify Blobstore Consistency Check in Tests (#73992)
With work to make repo APIs more async incoming in #73570
we need a non-blocking way to run this check. This adds that async
check and removes the need to manually pass executors around as well.
2021-06-10 16:12:26 +02:00
David Turner 319286664a
Fix store-smb split package (#73970)
Moves the implementation of this plugin from `o.e.i.store` and misc
other places into `o.e.i.store.smb`.
2021-06-10 07:41:39 +01:00
Ryan Ernst ab1a2e4a84
Add precommit task for detecting split packages (#73784)
Modularization of the JDK has been ongoing for several years. Recently
in Java 16 the JDK began enforcing module boundaries by default. While
Elasticsearch does not yet use the module system directly, there are
some side effects even for those projects not modularized (eg #73517).
Before we can even begin to think about how to modularize, we must
Prepare The Way by enforcing packages only exist in a single jar file,
since the module system does not allow packages to coexist in multiple
modules.

This commit adds a precommit check to the build which detects split
packages. The expectation is that we will add the existing split
packages to the ignore list so that any new classes will not exacerbate
the problem, and the work to cleanup these split packages can be
parallelized.

relates #73525
2021-06-08 15:04:23 -07:00
Ryan Ernst 63012c8a40
Move ParseField to o.e.c.xcontent (#73923)
ParseField is part of the x-content lib, yet it doesn't exist under the
same root package as the rest of the lib. This commit moves the class to
the appropriate package.

relates #73784
2021-06-08 13:32:14 -07:00
Ryan Ernst 68817d7ca2
Rename o.e.common in libs/core to o.e.core (#73909)
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.

relates #73784
2021-06-08 09:53:28 -07:00
Ryan Ernst f98b374cf6
Revert "Upgrade Azure SDK and Jackson (#72833) (#72995)" (#73837)
The recent upgrade of the Azure SDK has caused a few test failures that
have been difficult to debug and do not yet have a fix. In particular, a
change to the netty reactor resolving
(https://github.com/reactor/reactor-netty/issues/1655). We need to wait
for a fix for that issue, so this reverts commit
6c4c4a0ecb.

relates #73493
2021-06-07 10:20:46 -07:00
Ryan Ernst 64054de1ac
Rename bootstrap package in core jar (#73788)
The org.elasticsearch.bootstrap package exists in server with classes
for starting up Elasticsearch. The elasticsearch-core jar has a handful
of classes that were split out from there, namely java version parsing
and jarhell. This commit moves those classes to a new
org.elasticsearch.jdk package so as to not split the server owned
bootstrap package.

relates #73784
2021-06-07 08:14:44 -07:00
Ryan Ernst 8cba213dfc
Explicitly set illegal-access to deny for tests (#72588)
Since Java 16, the default value for illegal-access is deny. This means
the latest release of Elasticsearch, and all current integration tests,
run with deny (since we don't explicitly set it in jvm options). Yet
tests run with illegal-access=warn, for legacy reasons. #71908
proposed to remove the setting from test jvms, but concerns were raised
there about whether this would cause some test failures.

This commit explicitly sets tests to deny. This has the added benefit
that any failures will be caught even when running tests with older
jvms.
2021-06-02 16:09:27 -07:00
Nhat Nguyen 5efb6eaba6
Update Lucene to 8.9.0-snapshot-ddc238e5df8 (#73568)
Just include LUCENE-9980, which fixes #39591.

Closes #39591
2021-05-31 13:49:29 -04:00
Ryan Ernst e9970aa7e3
Upgrade netty to 4.1.63 (#73011) 2021-05-27 15:51:35 -07:00
Ryan Ernst 6c4c4a0ecb
Upgrade Azure SDK and Jackson (#72833) (#72995)
This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The
Jackson upgrade must happen at the same time due to Azure depending on
this new version of Jackson.

closes #66555
closes #67214

Co-authored-by: Francisco Fernández Castaño <francisco.fernandez.castano@gmail.com>
2021-05-27 07:55:18 -07:00
Tamara Braun 35584d9fdd
Make RBACEngine.resolveAuthorizedIndicesFromRole Return Set for Lookups (#72598)
This changes the result of AuthorizationEngine.loadAuthorizedIndices
(and dependent methods) from List<String> to Set<String>.

This has the following performance benefits:
1. `contains` checks are faster
2.  RBACEngine always formed this collections as a Set, so this
    change reduces unnecessary copying.

An additional performance improvement was added when resolve authorized
index names for data streams.
2021-05-26 16:25:53 +10:00
Nhat Nguyen 1764e8ba15
Upgrade to Lucene-8.9.0-SNAPSHOT-efdc43fee18 (#73130)
Upgrades to Lucene-8.9 snapshot which includes:

- LUCENE-9507: Custom order for leaves (/cc @mayya-sharipova)
- LUCENE-9935: Enable bulk merge for stored fields with index sort
2021-05-17 09:37:20 -04:00