System indices should be treated as a special set of indices and not be
accessible by all users. The existing security codebase has the notion
of restricted indices, which are currently a subset of system indices.
This change unifies the two concepts by making system indices the set
of restricted indices. This means that going forward, consumers of
system indices will need access to restricted indices.
Our intention is that this will be handled internally as much as
possible. For example, restricted index access can be used implicitly
by setting a valid origin on a request or using a system index plugin.
In these cases, the security module will apply internally defined
privileges when necessary.
The main impact of this change for developers is that system index
deletions will require superuser privileges, so we have to make sure
we are using an admin role for test cleanup.
Closes#69298
Co-authored-by: William Brafford <william.brafford@elastic.co>
Co-authored-by: Albert Zaharovits <albert.zaharovits@elastic.co>
This is the final step in the removal of the X-Pack specific SSL
configuration code (replaced by libs/ssl-config)
For some time we have had two implementations of SSL Configuration
code. One was in org.elasticsearch.xpack.core.ssl, the other in
org.elasticsearch.common.ssl
These two implementations had essentially the same functionality:
- Handle settings such as '*.ssl.certificate`, `*.ssl.key` etc
- Load certificates and keys from PEM files and Keystores
- Build and configure Java class such as SSLContext, KeyManager, etc
based on the configuration and certificates.
As of this common the X-Pack version is no more, and all SSL
configuration in Elasticsearch is handled by the libs/ssl-config
version instead.
Resolves: #68719
* Reformatting to keep Checkstyle after formatting
* Configure spotless everywhere, and disable the tasks if necessary
* Add XContentBuilder helpers, fix test
* Tweaks
* Add a TODO
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Adding base RestHandler class for Enrollment APIs
This change adding an abstract RestHandler class and extends it by
enrollment API classes (node and Kibana enrollment). It will handle the
cases when `enrollment.enabled` is not set to `true`. It will return an
appropriate exception in this case.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This commit changes the Enroll Kibana API to create and return
a token for this service account, instead of setting and returning the
password of the kibana_system built-in user. Both the token name and
value are returned in the response of the API.
Removes the deprecated kibana_dashboard_only_user from the set of reserved roles in Elasticsearch as this legacy functionality is being removed from Kibana.
Relates: elastic/kibana#54755.
* Name `BulkItemResponse` ctors
`BulkItemResponse` can contain either a success or failure. This
replaces the two constructors used to build either case with named
static methods. So instead of
```
return new BulkItemResponse(0, OpType.CREATE, createResponse);
return new BulkItemResponse(0, OpType.CREATE, failure);
```
you now use
```
return BulkItemResponse.success(0, OpType.CREATE, createResponse);
return BulkItemResponse.failure(0, OpType.CREATE, failure);
```
This makes it marginally easier to read code building these things - you
don't have to know the type of the parameter to know if its a failure
or success.
* Consistent
* Mock response
Introduce inference configs for NLP models.
When a PyTorch model is put, the config now expects
a different inference config per task type. Thus, we
have a `ner`, `fill_mask`, and `sentiment_analysis`
config. In addition, the tokenization parameters have
been grouped together and are now part of the relevant
inference config objects. Thus the vocabulary can now
be on a document on its own. A new vocabulary config
object allows the user to specify the location of the
vocabulary document.
In #75617 a new setting, system_annotations_retention_days, was
added to control how long system annotations are retained for.
We now feel that this setting is redundant and that system
annotations should be retained for the same period as results.
This is intuitive and defensible, as system annotations can be
considered a type of result.
Followup to #75617
Add configuration for pruning dead split fields in anomaly detection
jobs via the `model_prune_window` field for both the job creation and
update APIs.
Relates to ml-cpp/#1962
The Get service account credentials API now returns file-backed tokens from all
nodes instead of only the local node. For each file-backed service token, we
list names of the nodes where this token is found. The response for node-local
credentials (currently only file-backed tokens) is place inside the
"nodes_credentials.file_tokens" field. There is also a nodes_credentials._nodes
field containing information about the overall request execution (it works the
same way as the _nodes field of Nodes info API, etc.) Detailed response sample
can be found in #74530
This PR also removes the beta label from the API's documentation page.
Resolves: #74530
Add an additional boolean field to a deprecation issue to indicate that a deprecation issue can only be resolved during a rolling upgrade when a node is offline (for the upgrade).
No deprecation issue has been marked as restart required as part of this change.
Closes#73091
The current `ids` option doesn't allow pinning a specific document in a
single index when searching over multiple indices. This introduces a
`documents` option, which is an array of `_id` and `_index`
fields to allow index-specific pins.
Closes https://github.com/elastic/elasticsearch/issues/67855.
When performing incremental reductions, 0 value of docCountError may mean that
the error was not previously calculated, or that the error was indeed previously
calculated and its value was 0. We end up rejecting true values set to 0 this
way. This may lead to wrong upper bound of error in result. To fix it, this PR
makes docCountError nullable. null values mean that error was not calculated
yet.
Fixes#40005
Co-authored-by: Igor Motov <igor@motovs.org>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
In 8.0, with security on by default, we store the HTTP
layer CA PrivateKeyEntry in the http.ssl keystore (along
with the node certificate) so that it is available in our
Enrollment API transport actions.
When loading a keystore, the current behavior is that the
X509ExtendedKeyManager will iterate through the PrivateKeyEntry
objects and will return the first key/certificate that satisfies
the requirements of the client and the server configuration,
and lacks any additional logic/filters.
We need the KeyManager to deterministically pick the node
certificate/key in all cases as this is the intended entry to be
used for TLS on the HTTP layer.
This change introduces filtering when creating the in-memory
keystore the KeyManager is loaded with, so that it will not
include PrivateKeyEntry objects when:
- there are more than 1 PrivateKeyEntry objects in the keystore
- The leaf certificate associated with the PrivateKeyEntry is a
CA certificate
Related: #75097
Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
Reindex and friends have tasks that start but are not ready to
rethrottle before they figured out if they are leader or worker
tasks. Now wait for the task to fully start before rethrottling.
Also added additional assertions to help see if the inability
to rethrottle is caused by some failure.
Closes#60811
This PR returns the get snapshots API to the 7.x format (and transport client behavior) and enhances it for requests that ask for multiple repositories.
The changes for requests that target multiple repositories are:
* Add `repository` field to `SnapshotInfo` and REST response
* Add `failures` map alongside `snapshots` list instead of returning just an exception response as done for single repo requests
* Pagination now works across repositories instead of being per repository for multi-repository requests
closes#69108closes#43462
During implementation we discovered that the clusters should not
necessarily have a unique name and thus we don't need to convey
this information in the response of the Enroll Node API.
The name token1 is used in both testGetServiceAccountCredentials and
testCreateServiceAccountToken. There are times when the two tests are not fully
isolated. This leads to test failure because indexing of the service token
document must be an op_create. This PR fixes the failure by using a different
token name in testGetServiceAccountCredentials.
This will allow components to add custom metadata to deprecation issues.
This make extracting additional details about deprecations more robust,
otherwise these details need to be parsed from the deprecation message field.
Adjusted the ml model snapshot deprecation to use custom metadata, and
included the job id and snapshot id as custom metadata.
Closes#73089
We barely test the correct handling of user metadata directly.
With upcoming changes to how `SnapshotInfo` is stored it would be nice
to have better test coverage. This PR adds randomized coverage of serializing
user metadata to a large number of tests that all user the shared infrastructure
that is adjusted here.
Follow-up to #73434
Ensures that High Level Rest Client is running against a verified
Elasticsearch. When the first request is send on HLRC, a request to the
info endpoint is made first to verify the product identification and
version.
The data_description of anomaly detection jobs used to accept
delimited data, although this was never documented.
This change removes the delimited option from the data_description,
and the associated functionality in post_data that handled it.
This is not a breaking change because it's removing functionality
that officially never existed. However, just in case somebody
was using it it is only removed from 8.0 and higher, so that at
least they won't find out during a patch install.
Modularization of the JDK has been ongoing for several years. Recently
in Java 16 the JDK began enforcing module boundaries by default. While
Elasticsearch does not yet use the module system directly, there are
some side effects even for those projects not modularized (eg #73517).
Before we can even begin to think about how to modularize, we must
Prepare The Way by enforcing packages only exist in a single jar file,
since the module system does not allow packages to coexist in multiple
modules.
This commit adds a precommit check to the build which detects split
packages. The expectation is that we will add the existing split
packages to the ignore list so that any new classes will not exacerbate
the problem, and the work to cleanup these split packages can be
parallelized.
relates #73525
ParseField is part of the x-content lib, yet it doesn't exist under the
same root package as the rest of the lib. This commit moves the class to
the appropriate package.
relates #73784
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
The recent upgrade of the Azure SDK has caused a few test failures that
have been difficult to debug and do not yet have a fix. In particular, a
change to the netty reactor resolving
(https://github.com/reactor/reactor-netty/issues/1655). We need to wait
for a fix for that issue, so this reverts commit
6c4c4a0ecb.
relates #73493
Martijn van Groningen
Przemysław Witek
Jay Modi
Tim Vernum
Rory Hunter
Lyudmila Fokina
Ioannis Kakavas
Yang Wang
bellengao
Devon Thomson
Nik Everett
Dimitris Athanasiou
Rene Groeschke
David Roberts
Tanguy Leroux
Ed Savage
David Harsha
Nikita Glashenko
Henning Andersen
Almog Tavor
Sylvain Wallez
Armin Braun
David Turner
Nhat Nguyen
Ryan Ernst
Przemyslaw Gomulka