Commit Graph

2799 Commits

Author SHA1 Message Date
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
David Turner dd519a9eba
Introduce string constant for readonly setting (#68291)
A blob store repository can be put in readonly mode by setting
`readonly: true` in its settings. In the codebase the setting key is
just the literal string `"readonly"` wherever it's used and it takes
some effort to determine what the right setting name is, in particular
to check each time that it's not spelled `"read_only"`.

This commit replaces those literal `"readonly"` strings with the
`BlobStoreRepository#READONLY_SETTING_KEY` constant to reduce this
trappiness.
2021-02-01 15:43:37 +00:00
Ignacio Vera 747773d5af
Upgrade to Lucene 8.8.0 (#68272) 2021-02-01 13:36:03 +01:00
Mark Vieira 413e6bac07
Disable secureHdfs fixture when testing on JDK 16 (#68182) 2021-01-28 18:25:59 -08:00
Armin Braun 77162071f5
Add ClusterUUID to RepositoryData (#68002)
Record the clusterUUID of the last cluster to write
to a repository in the `RepositoryData` and use it for more
meaningful logging when running into a concurrent modification
issue.
2021-01-28 12:38:15 +01:00
Rory Hunter ad1f876daa
Replace NOT operator with explicit `false` check (#67817)
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.

We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
2021-01-26 14:47:09 +00:00
David Turner e5a15d4fcb
Introduce repository UUIDs (#67829)
Today a snapshot repository does not have a well-defined identity. It
can be reregistered with a different cluster under a different name, and
can even be registered with multiple clusters in readonly mode.

This presents problems for cases where we need to refer to a specific
snapshot in a globally-unique fashion. Today we rely on the repository
being registered under the same name on every cluster, but this is not a
safe assumption.

This commit adds a UUID that can be used to uniquely identify a
repository. The UUID is stored in the top-level index blob, represented
by `RepositoryData`, and is also usually copied into the
`RepositoryMetadata` that represents the repository in the cluster
state. The repository UUID is exposed in the get-repositories API; other
more meaningful consumers will be added in due course.
2021-01-25 12:17:52 +00:00
Jim Ferenczi e77c523bd9
Upgrade to a new lucene 8.8.0 snapshot (#67691)
This change upgrades to the latest Lucene 8.8.0 snapshot.
It also restores the compression on binary doc values that was lost in the last snapshot upgrade.
The compression is now configurable on binary doc values but we don't expose this functionality yet so this commit ensures that we pick the same compression mode as previous releases (BEST_COMPRESSION).
2021-01-19 13:33:19 +01:00
Armin Braun 6d025d3a27
Log Slowness on Sending Transport Messages (#67664)
Similar to #62444 but for the outbound path.

This does not detect slowness in individual transport handler logic,
this is done via the inbound handler logging already, but instead
warns if it takes a long time to hand off the message to the relevant
transport thread and then transfer the message over the wire.
This gives some visibility into the stability of the network
connection itself and into the reasons for slow network
responses (if they are the result of slow networking on the sender).
2021-01-19 12:19:32 +01:00
Rory Hunter 1a05a5ac24
Introduce deprecation categories (#67443)
Closes #64824. Introduce the concept of categories to deprecation
logging. Every location where we log a deprecation message must now
include a deprecation category.
2021-01-18 16:16:54 +00:00
Julie Tibshirani 5852fbedf5
Rename QueryShardContext -> SearchExecutionContext. (#67490)
We decided to rename `QueryShardContext` to clarify that it supports all parts
of search request execution. Before there was confusion over whether it should
only be used for building queries, or maybe only used in the query phase. This
PR also updates the javadocs.

Closes #64740.
2021-01-14 09:11:59 -08:00
David Turner bc1f50c523
Permit wait_for_active_shards warnings in master (#67498)
Part of the fixes for #66419, this commit permits nodes to emit the
deprecation warning regarding not specifying `?wait_for_active_shards`
when closing an index in 7.x versions for x ≥ 12. This change is
required on `master` too since the BWC tests encounter these warnings.

Relates #67246, which is the 7.x part of this change.
2021-01-14 15:55:43 +00:00
Nik Everett 7b0b09dfd7
Help eclipse compilation (#67403)
Eclipse wasn't seeing the special shadow jars we were making for
repository-azure and repository-hdfs so it wasn't able to compile those
plugins. This points Eclipse at the project that we use to build the
shadow jar which gets it compiling. The tests don't pass because we
aren't pointing at the shadow jars but at least we compile.
2021-01-13 13:37:46 -05:00
Francisco Fernández Castaño 4b9f2e94bd
Increase Azure client timeout on tests (#67210)
Additionally, this commit improves the error messages provided as
previously we weren't including the blob name on
deletion failures.

Closes #67119
2021-01-13 13:57:13 +01:00
Francisco Fernández Castaño 78ad79a87f
Remove assertion that checks the exception message on AzureBlobContainerRetriesTests#testRetryUntilFail (#67258)
The error message might change depending on the timing when we try
to read from the stream. Since we already check that we're not able
to read any data this assertion doesn't add much value.
2021-01-12 14:05:28 +01:00
Ignacio Vera 604ee06a3b
Upgrade to lucene-8.8-snapshot-f73f6b1 (#67228) 2021-01-12 08:03:00 +01:00
Dan Hermann 8b05edaeb5
Fix attachment processor test that fails on Windows (#67156) 2021-01-08 07:10:04 -06:00
Mark Vieira 22a6811802 Mute AzureBlobStoreRepositoryTests.testLargeBlobCountDeletion 2021-01-07 11:54:05 -08:00
Francisco Fernández Castaño ac63c6dcf5
Fix AzureBlobContainerRetriesTests#testRetryUntilFail (#67077)
We were too agressive with retries and in certain scenarios (CI) it
was possible that when the SDK had retried n times the http handler
had some pending backlog that didn't account for all the performed
requests.

Closes #66865
2021-01-06 13:47:21 +01:00
Francisco Fernández Castaño 9950cd24be
Add Ability to Write a BytesReference to Azure BlobContainer (#66683) 2021-01-06 12:01:11 +01:00
Francisco Fernández Castaño f1ebe1195c
Avoid early task cancellation during azure parallel blob deletions (#66929)
Closes #66633
2021-01-05 11:08:16 +01:00
Albert Zaharovits a184486362
Fix azure repo stream exhaust check for multipart uploads (#66769)
This PR fixes the validation of the conversion from an input stream to a flux in the
AzureBlobStore's multipart update logic, which erroneously checked that the upload
input stream is exhausted after each part's flux is completed.
2021-01-04 17:49:40 +02:00
markharwood aa01af882e
Annotated text plugin highlighter causes "array_index_out_of_bounds_exception" (#66593)
Recent changes to the way Analyzers and field mappings are managed revealed a bug in the AnnotatedHighlighterAnalyzer class.
Old sequences of calls avoided the issue but under the new scheme a counter reset was required between documents being highlighted.
Closes #66535
2021-01-04 15:41:49 +00:00
Rene Groeschke eee6e11883
Port all task definitions to task avoidance api (#66738)
This finishes porting all tasks created in gradle build scripts and plugins to use 
the task avoidance api (see #56610)

* Port all task definitions to task avoidance api
* Fix last task created during configuration
* Fix test setup in  :modules:reindex
* Declare proper task inputs
2021-01-04 12:32:19 +01:00
Armin Braun f0459f63f2
Fix S3ClientSettings Class Loading (#66886)
This is motivated by the inability to run
`org.elasticsearch.repositories.encrypted.EncryptedS3BlobStoreRepositoryIntegTests`
in isolation without this workaround. The way integration tests load classes
otherwise leads to a load order which doesn't load the plugin class first,
thus fails to apply the jackson workaround before further S3 classes are loaded
but depend on our Jackson workaround.
2021-01-04 12:30:34 +01:00
Mark Tozzi e26c9bbd52
Rename BYTES ValuesSourceType to reflect intended usage (#66762) 2020-12-30 12:39:17 -05:00
Albert Zaharovits cd72f45c33
Client-side encrypted snapshot repository (feature flag) (#66773)
The client-side encrypted repository is a new type of snapshot repository that
internally delegates to the regular variants of snapshot repositories (of types
Azure, S3, GCS, FS, and maybe others but not yet tested). After the encrypted
repository is set up, it is transparent to the snapshot and restore APIs (i.e. all
snapshots stored in the encrypted repository are encrypted, no other parameters
required).
The encrypted repository is protected by a password stored on every node's
keystore (which must be the same across the nodes).
The password is used to generate a key encrytion key (KEK), using the PBKDF2
function, which is used to encrypt (using the AES Wrap algorithm) other
symmetric keys (referred to as DEK - data encryption keys), which themselves
are generated randomly, and which are ultimately used to encrypt the snapshot
blobs.

For example, here is how to set up an encrypted  FS repository:
------
 1) make sure that the cluster runs under at least a "platinum" license
(simplest test configuration is to put `xpack.license.self_generated.type: "trial"`
in the elasticsearch.yml file)
 2) identical to the un-encrypted FS repository, specify the mount point of the
shared FS in the elasticsearch.yml conf file (on all the cluster nodes),
e.g. `path.repo: ["/tmp/repo"]`
 3) store the repository password inside the elasticsearch.keystore, *on every cluster node*.
In order to support changing password on existing repository (implemented in a follow-up),
the password itself must be names, e.g. for the "test_enc_key" repository password name:
`./bin/elasticsearch-keystore add repository.encrypted.test_enc_pass.password`
*type in the password*
4) start up the cluster and create the new encrypted FS repository, named "test_enc", by calling:
`
curl -X PUT "localhost:9200/_snapshot/test_enc?pretty" -H 'Content-Type: application/json' -d'
{
  "type": "encrypted",
  "settings": {
    "location": "/tmp/repo/enc",
    "delegate_type": "fs",
    "password_name": "test_enc_pass"
  }
}
'
`
5) the snapshot and restore APIs work unmodified when they refer to this new repository, e.g.
` curl -X PUT "localhost:9200/_snapshot/test_enc/snapshot_1?wait_for_completion=true"`


Related: #49896 #41910 #50846 #48221 #65768
2020-12-23 23:46:59 +02:00
Ioannis Kakavas bd873698bc
Ensure CI is run in FIPS 140 approved only mode (#64024)
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.

This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.

It also addresses a number of tests that would fail in approved only mode
Mainly:

    Tests that use PBKDF2 with a password less than 112 bits (14char). We
    elected to change the passwords used everywhere to be at least 14
    characters long instead of mandating
    the use of pbkdf2_stretch because both pbkdf2 and
    pbkdf2_stretch are supported and allowed in fips mode and it makes sense
    to test with both. We could possibly figure out the password algorithm used
    for each test and adjust password length accordingly only for pbkdf2 but
    there is little value in that. It's good practice to use strong passwords so if
    our docs and tests use longer passwords, then it's for the best. The approach
    is brittle as there is no guarantee that the next test that will be added won't
    use a short password, so we add some testing documentation too.
    This leaves us with a possible coverage gap since we do support passwords
    as short as 6 characters but we only test with > 14 chars but the
    validation itself was not tested even before. Tests can be added in a followup,
    outside of fips related context.

    Tests that use a PKCS12 keystore and were not already muted.

    Tests that depend on running test clusters with a basic license or
    using the OSS distribution as FIPS 140 support is not available in
    neither of these.

Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
2020-12-23 21:00:49 +02:00
Gordon Brown 045abd82d4
Mute AzureStorageCleanupThirdPartyTests.testCleanup (#66635)
See https://github.com/elastic/elasticsearch/issues/66633
2020-12-18 12:52:15 -07:00
Armin Braun 3819fcb582
Add Ability to Write a BytesReference to BlobContainer (#66501)
Except when writing actual segment files to the blob store
we always write `BytesReference` instead of a stream.
Only having the stream API available forces needless copies
on us. I fixed the straight-forward needless copying for
HDFS and FS repos in this PR, we could do similar fixes for
GCS and Azure as well and thus significantly reduce the peak
memory use of these writes on master nodes in particular.
2020-12-17 17:42:29 +01:00
Francisco Fernández Castaño c96b3ba9b6
Fix AzureBlobContainerRetriesTests#testRetryUntilFails (#66531)
Add a clearer approach to this test
2020-12-17 16:53:35 +01:00
Francisco Fernández Castaño 02ac68eb8b
Reduce memory usage on Azure repository implementation (#66489)
This commit moves the upload logic to the repository itself
instead of delegating into the SDK.
Multi-block uploads are done sequentially instead of in parallel
that allows to bound the outstanding memory.
Additionally the number of i/o threads and heap arenas have been 
reduced to 1, to reduce the memory overhead.

Closes #66385
2020-12-17 11:09:55 +01:00
Francisco Fernández Castaño 54fbd03052
Mute AzureBlobStoreRepositoryTests (#66389) 2020-12-15 19:06:28 +01:00
Francisco Fernández Castaño fd1d282ba9
Upgrade Azure repository SDK to v12 (#65140)
Upgrade Azure repository to the latest non blocking Azure SDK.

Closes https://github.com/elastic/elasticsearch/issues/43309

Co-authored-by: Ryan Ernst <ryan@iernst.net>
2020-12-15 11:39:06 +01:00
James Baiera 9bb6a3ad2d
Add HDFS searchable snapshot integration (#66185)
Adds a bounded read implementation on the HDFS blob store as well as integration tests to 
the searchable snapshot project that ensures functionality on both kerberos and simple 
authentication HDFS.
2020-12-14 16:04:41 -05:00
Martijn Laarman e31e3dea32
Add `visibility` the to rest-spec-api (#56104) 2020-12-14 12:23:28 +01:00
Rene Groeschke defaa93902
Avoid tasks materialized during configuration phase (#65922)
* Avoid tasks materialized during configuration phase
* Fix RestTestFromSnippet testRoot setup
2020-12-12 16:14:17 +01:00
Dan Hermann 51452d1ae3
Mute failing AttachmentProcessor.testIndexedCharsWithResourceName test (#66121) 2020-12-09 11:24:57 -06:00
Martijn Laarman 8d3def3e1f
Add Accept & Content-Type headers to rest api spec (#53979)
Co-authored-by: Russ Cam <russ.cam@elastic.co>
2020-12-09 14:43:05 +01:00
Dan Hermann 149f1f9412
Minor DRYing up of attachment processor tests (#65975) 2020-12-08 11:38:00 -06:00
Rene Groeschke 0911d04467
Make AntFixture handling task provider api compliant (#65832)
This tweaks the AntFixture handling to make it compliant with the task avoidance api.
Tasks of type StandaloneRestTestTask are now generally finalised by using the typed ant stop task
which allows us to remove of errorprone dependsOn overrides in StandaloneRestTestTask. As a result
we also ported more task definitions in the build to task avoidance api.

Next work item regarding AntFixture handling is porting AntFixture to a plain Gradle task and remove
Groovy AntBuilder will allow us to port more build logic from Groovy to Java but is out of the scope of
This PR.
2020-12-08 13:07:36 +01:00
yangyaofei 0f8476361c
Attachment ingest processor: add resource_name field (#64389) 2020-12-07 11:46:20 -06:00
Alexander Reelsen fd3d7e3368
Remove class that is part of commons-codec (#65259)
This class was copied from the trunk and is now part of the stable 
release since 2012.
2020-12-03 17:55:18 +01:00
Armin Braun 4547d3b245
Refactor ActionListener#map towards Stricter API (#65526)
Making `#map` look and feel a little nicer, optimize chains of `#map`,
and replace `#delegateFailure` calls with `#map` calls where possible
in order to enforce callbacks not throwing where possible.
2020-12-01 03:00:51 +01:00
Armin Braun 06a31a0aca
Add List Append Utility Method (#65576)
(list -> copy -> add one -> wrap immutable) is a pretty common pattern in CS
updates and tests => added a shortcut for it here and used it in easily identifyable
spots.
2020-12-01 02:47:21 +01:00
Alan Woodward 1a8ce8716d
Restore use of default search and search_quote analyzers (#65491)
In the refactoring of TextFieldMapper, we lost the ability to define
a default search or search_quote analyzer in index settings. This
commit restores that ability, and adds some more comprehensive
testing.

Fixes #65434
2020-11-26 16:57:45 +00:00
Rene Groeschke 97749a3372
Port rest integ tests to use task avoidance api (#65011)
This ports the majority of the rest integ tests tasks to use the task avoidance api.

- There are some edge cases left that we need to investigate, but we can do that separately.
2020-11-26 10:30:06 +01:00
Alan Woodward d088171a87
Use ValueFetcher when loading text snippets to highlight (#63572)
HighlighterUtils.loadFieldValues() loads values directly from the source, and
then callers have to deal with filtering out values that would have been removed
by an ignore_above filter on keyword fields. Instead, we can use the
ValueFetcher for the relevant field, which handles all this logic for us.

Closes #59931.
2020-11-24 16:09:37 +00:00
Julie Tibshirani f4a462d05e
Simplify how source is passed to fetch subphases. (#65292)
This PR simplifies how the document source is passed to each fetch subphase. A summary of the strategy:
* For each document, we try to eagerly load the source and store it on `HitContext`. Most subphases that access source, like source filtering and highlighting, use `HitContext`. For nested hits, we filter the parent source and also store this source on `HitContext`.
* Only for non-nested documents, we also store the loaded source on `QueryShardContext#lookup`. This allows subphases that access source through `SearchLookup` to use the pre-loaded source when possible. This is now a common occurrence, since runtime fields are supported in the 'fields' option and may soon be supported in highlighting.

There is no longer a special `SearchLookup` just for the fetch phase. This was not necessary and was mostly caused by a misunderstanding of how `QueryShardContext` should be used.

Addresses #62511.
2020-11-20 14:09:41 -08:00
Ryan Ernst 23a47cebf1
Add plugin permission validation (#64751)
Security manager policies within plugins currently can ask to grant any
permission (though we block some within the security manager itself at
runtime). Yet most of these permissions should never be necessary, and
some we would actively not want any plugins to be allowed to use. This
commit adds validation of plugins' policy files to restrict the
permissions allowed to be granted to a subset that is reasonable for
plugins to need. The allowed permissions are not ideal (still containing
things like suppressAccessChecks), but it is a step forward in defining
a stricter model for plugins that reduces the surface area of potential
abuse.
2020-11-19 14:21:34 -08:00