The path.data setting is now a singular string, but the method
dataFiles() that gives access to the path is still an array. This commit
renames the method and makes the return type a single Path.
relates #71205
Since multiple data path support has been removed, the Setting no longer
needs to support multiple values. This commit converts the
PATH_DATA_SETTING to a String setting from List<String>.
relates #71205
We shouldn't loop over the listeners under the mutex in `done` since in most use-cases we used `DirectExecutorService`
with this class.
Also, no need to create an `AbstractRunnable` for direct execution. We use this listener on the hot path in authentication
making this a worthwhile optimization I think.
Lastly, no need to clear and thus loop over `listeners`, the list is not used again after the `done` call returns anyway
so no point in retaining it at all (especially when in a number of use cases we add listeners only after the `done` call
so we can also save the instantiation by making the field non-final).
This test broken when the geoip processor created its index before we take the snapshot
as that would cause an unexpected number of shards in the restore. Rather than excluding
global state from the snapshot (the internal index of the plugin is snapshotted as part
of global state in 7.12+ so the index filtering we use doesn't apply to it) I opted to fix this
by making the restore selective instead to keep coverage of the global state.
closes#71763
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
This change exposes the newly introduced parameter `dynamic_templates`
in ingest. This parameter can be set by a set processor or a script processor.
Relates #69948
This change removes assertion from DatabaseRegistry - we can easily loose .geoip_databases index with persistent task state still in cluster state, this is not assertion failing, this is usual failure and should be signalled as one.
This also tries to fix packaging tests by avoiding duplicates in elasticsearch.yml.
Closes#71762
This PR adds documentation for GeoIPv2 auto-update feature.
It also changes related settings names from geoip.downloader.* to ingest.geoip.downloader to have the same convention as current setting.
Relates to #68920
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
This change enables GeoIP downloader by default.
It removes feature flag but adds flag that is used by tests to disable it again (as we don't want to hammer GeoIP database service with every test cluster we spin up).
Relates to #68920
* Warn users if security is implicitly disabled
Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
clusters.
This change introduces clear warnings when security features are
implicitly disabled.
- a warning header in each REST response if security is implicitly
disabled;
- a log message during cluster boot.
System index descriptors are used to describe a system index, which are
expected to change as new versions are developed. As part of this, the
descriptors had a minimum supported version field so that the contents
within that descriptor would not be applied if there were nodes older
than that version. However, this falls short of being able to
accurately describe what a system index should look like in a given
cluster where there are mixed node versions.
This change moves us towards being able to accurately describe and
know what the system index should look like. A system index is now
able to accept a list of the prior system index descriptor objects
so that clusters with mixed versions can select the appropriate
descriptor and ensure the index is created properly. As the node
versions change during a rolling upgrade, the cluster will then be
able to adapt the system index to the most recent version once all
master and data nodes have been upgraded.
Co-authored-by: Tim Vernum <tim@adjective.org>
Co-authored-by: Yang Wang <ywangd@gmail.com>
Today by default the `MANAGEMENT` threadpool always permits 5 threads
even if the node has a single CPU, which unfairly prioritises management
activities on small nodes. With this commit we limit the size of this
threadpool to the number of processors if less than 5.
Relates #70435
Currently we don't report any exceptions occuring during field_caps requests back to the user.
This PR adds a new failure section to the response which contains exceptions per index.
In addition the response contains another field, `failed_indices`, with the number of indices that threw
an exception. If all of the requested indices fail, the whole request fails, otherwise the request succeeds
and it is up to the caller to check for potential errors in the response body.
Closes#68994
Previously, the search states are stored in ReaderContext on data nodes.
Since 7.10, we send them to the coordinating node in a QuerySearchResult
of a ShardSearchRequest and the coordinating node then sends them back
in ShardFetchSearchRequest. We must keep the search states in data nodes
unless they are sent back in the fetch phase. We used the channel version
to determine this guarantee. However, it's not correct in CCS requests in
mixed clusters.
1. The coordinating node of the local cluster on the old version sends a
ShardSearchRequest to a proxy node of the remote cluster on the new
version. That proxy node delivers the request to the data node. In this
case, the channel version between the data node and the proxy node is
>= 7.10, but we won't receive the search states in the fetch phase as they
are stripped out in the channel between the old coordinating node and
the new proxy.
```
[coordinating node v7.9] --> [proxy node v7.10] --> [data node on v7.10]
```
2. The coordinating node of the local on the new version sends a
ShardSearchRequest to a proxy node of the remote cluster on the new
version. However, the coordinating node sends a ShardFetchSearchRequest
to another proxy node of the remote cluster that is still on an old
version. The search states then are stripped out and never reach the
data node.
```
-> query phase: [coordinating node v7.10] --> [proxy node v7.10] --> [data node on v7.10]
-> fetch phase: [coordinating node v7.10] --> [proxy node v7.9] --> [data node on v7.10]
```
This commit fixes the first issue by explicitly serializing the channel version in
a ShardSearchRequest and the second by continue storing the search states
in ReaderContext unless all nodes are upgraded.
Relates #52741
Today when creating an internal test cluster, we allow the test to
supply the node settings that are applied. The extension point to
provide these settings has a single integer parameter, indicating the
index (zero-based) of the node being constructed. This allows the test
to make some decisions about the settings to return, but it is too
simplistic. For example, imagine a test that wants to provide a setting,
but some values for that setting are not valid on non-data nodes. Since
the only information the test has about the node being constructed is
its index, it does not have sufficient information to determine if the
node being constructed is a non-data node or not, since this is done by
the test framework externally by overriding the final settings with
specific settings that dicate the roles of the node. This commit changes
the test framework so that the test has information about what settings
are going to be overriden by the test framework after the test provide
its test-specific settings. This allows the test to make informed
decisions about what values it can return to the test framework.
This PR sets the default value of `action.destructive_requires_name`
to `true.` Fixes#61074. Additionally, we set this value explicitly in
test classes that rely on wildcard deletions to clear test state.
Closes#69930. Closes#69928.
The ES build currently has 2 types of Docker output - Docker images,
and Docker build contexts. At the moment, only the images are tested,
meaning that bugs in the build contexts can go unnoticed.
This PR changes how we create Docker images so that we first create
the build contexts, and then build the images using them. This does
require some sleight-of-hand - the build contexts expect to download
an Elasticsearch archive directorly from the `Dockerfile`, which
will only ever work for non-snapshot version builds. In order to
get around this, the `Dockerfile` is modified to `COPY` in a local
archive file. Any other dependency files must exist in the build
context archive.
This PR also builds and tests the Iron Bank context. We do not
currently build a Docker image for this at all, and to build an
image requires us to set some build arguments to useful values. We
also need to provide all artifacts to the build, as the `Dockerfile`
cannot download anything. As a result, the `:distribution:docker`
project now defines a GitHub repository so that Gradle will download
a `tini` binary.
Note that there will need to be corresponding changes to
`release-manager`.
This change rewrites the rolling upgrade yml test that was added for
date_nanos in Java. This is needed to allow extra logic when backporting to
7.x. date_nanos are not available in 6.x so we need to skip the upgraded test
based on the version of the old cluster.
Relates #70463
This commit updates the default format of date_nanos field
on existing and new indices to use `strict_date_optional_time_nanos` instead of
`strict_date_optional_time`.
Using `strict_date_optional_time` as the default format for date_nanos doesn't
make sense because it accepts and parses dates with nanosecond precision,
but when it formats it drops the nanoseconds.
The change should be transparent for users, these formats accept the same input.
Relates #69192Closes#67063
This is the second PR for service accounts. It adds a new CLI tool
elasticsearch-service-tokens to manage file tokens. The file tokens are stored
in the service_tokens file under the config directory. Out of the planned create,
remove and list sub-commands, this PR only implements the create function since
it is the most important one. The other two sub-commands will be handled in
separate PRs.
Creating classloaders is not something plugins really need to do. It is
only necessary for cases of loading dynamic code, which we do within
painless. This commit restricts allowing this permission to modules.
Runtime permissions in the java security manager can be specific, or use
a wildcard for the name. This commit adds a test to ensure a blanket
policy allowing all runtime permissions is denied.
A #68808 introduced a possibility to declare fields which will be only available to parsing when a compatible API was used.
This commit replaces deprecated log with compatible logging when a 'compatible only' field was used. Also includes a refactoring of LoggingDeprecationHandler method names
relates #51816
The commit #55115 removed the possibility to directly force deprecation
log to be emitted. This means that usage of deprecated fields was
throttled and only one deprecation was logged. The key was common for
all fields = "deprecated_field".
This commit appends a used deprecated field name to prevent that
throttled.
Today every `BulkProcessor` creates two scheduler threads, both called
`[node-name][scheduler][T#1]`, which is also the name of the main
scheduler thread for the node. The duplicated thread names make it
harder to interpret a thread dump.
This commit makes the names of these threads distinct.
Closes#68470
This commit introduces system index types that will be used to
differentiate behavior. Previously system indices were all treated the
same regardless of whether they belonged to Elasticsearch, a stack
component, or one of our solutions. Upon further discussion and
analysis this decision was not in the best interest of the various
teams and instead a new type of system index was needed. These system
indices will be referred to as external system indices. Within external
system indices, an option exists for these indices to be managed by
Elasticsearch or to be managed by the external product.
In order to represent this within Elasticsearch, each system index will
have a type and this type will be used to control behavior.
Closes#67383
In 2a04118e88 we moved `ensureGreen()`
from `IndexingIT` to `ESRestTestCase`, including its `70s` timeout. This
timeout makes sense in the context of an `AbstractRollingTestCase` which
has a client timeout of `90s` (#26781) but general-purpose REST tests
only have a `60s` client timeout, so if `ensureGreen()` fails then it
fails with a `SocketTimeoutException`, bypassing the useful exception
handling that log the cluster state at time of failure.
This commit reduces the `ensureGreen()` timeout for most tests, leaving
it at `70s` only for `AbstractRollingTestCase`.
This commit adds back allowing FilePermission for reading files in
plugins. This is a temporary measure until plugins are automatically
granted read permissions for files within their own configuration
directory.
closes#69464
Closes#68043.
`PackageTests.test90DoNotCloseStderrWhenQuiet` failed because of a race
between ES startup and the test making an assertion on the ES output.
Fix this by adding a wrapper to retry the assertion for a while, until
either the assertion passes or the timeout expires.
Opening a Lucene index that supports soft-deletes currently creates the liveDocs bitset eagerly. This requires scanning
the doc values to materialize the liveDocs bitset from the soft-delete doc values. In order for searchable snapshot shards
to be available for searches as quickly as possible (i.e. on recovery, or in case of FrozenEngine whenever a search comes
in), they should read as little as possible from the Lucene files.
This commit introduces a LazySoftDeletesDirectoryReaderWrapper, a variant of Lucene's
SoftDeletesDirectoryReaderWrapper that loads the livedocs bitset lazily on first access. It is special-tailored to
ReadOnlyEngine / FrozenEngine as it only operates on non-NRT readers.
This commit removes support for JAVA_HOME. As we previously deprecated
usage of JAVA_HOME to override the path for the JDK, this commit follows
up by removing support for JAVA_HOME. Note that we do not treat
JAVA_HOME being set as a failure, as it is perfectly reasonable for a
user to have JAVA_HOME configured at the system level.
This commit introduces a dedicated envirnoment variable ES_JAVA_HOME to
determine the JDK used to start (if not using the bundled JDK). This
environment variable will replace JAVA_HOME. The reason that we are
making this change is because JAVA_HOME is a common environment variable
and sometimes users have it set in their environment from other JDK
applications that they have installed on their system. In this case,
they would accidentally end up not using the bundled JDK despite their
intentions. By using a dedicated environment variable specific to
Elasticsearch, we avoid this potential for conflict. With this commit,
we introduce the new environment variable, and deprecate the use of
JAVA_HOME. We will remove support for JAVA_HOME in a future commit.
This reduces the ceremony declaring test artifacts for a project.
It also solves an issue with usage of deprecated testRuntime that
testArtifacts extendsFrom which seems not required at all and would have
broke with Gradle 7.0 anyhow
Test artifact resolution is now variant aware which allows us a more adequate
compile and runtime classpath for the consuming projects.
We also Introduce a convention method in the elasticsearch build to declare
test artifact dependencies in an easy way close to how its done by the gradle build in
test fixture plugin.
Furthermore we cleaned up some inconsistent test dependencies declarations when
relying on a project and on its test artifacts
A small followup to #67413 and #68965: the underlying actions of the
`GET /_cat/segments` API are now cancellable, so we may as well cancel
them if needed.
The response to an `IndicesSegmentsAction` might be large, perhaps 10s
of MBs of JSON, and today it is serialized on a transport thread. It
also might take so long to respond that the client times out, resulting
in the work needed to compute the response being wasted.
This commit introduces the `DispatchingRestToXContentListener` which
dispatches the work of serializing an `XContent` response to a
non-transport thread, and also makes `TransportBroadcastByNodeAction`
sensitive to the cancellability of its tasks.
It uses these two features to make the `RestIndicesSegmentsAction`
serialize its response on a `MANAGEMENT` thread, and to abort its work
more promptly if the client's channel is closed before the response is
sent.
Extending deprecation logger to also log compatible API warnings.
These warnings will be written to deprecation log file, data stream and will emit response warning header - the same as deprecation logs. In order to comply with constant keyword type of data_stream.datatype the value will have to be the same for compatible and deprecation log - elasticsearch.deprecation.
The compatible and deprecation logs will differ by elasticsearch.event.category field which will always be DeprecationCategory.COMPATIBLE_API for all compatible logs
closes#52369
This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.
This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):
```
POST /_snapshot/my_repository/snapshot_2/_restore
{
"indices": "index_1",
"include_global_state": true,
"feature_states": ["watcher", "security"]
}
```
If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.
The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```
which returns a response of the form:
```
{
"features": [
{
"name": "tasks",
"description": "Manages task results"
},
{
"name": "kibana",
"description": "Manages Kibana configuration and reports"
}
]
}
```
Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.
Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
{
"feature_name": "tasks",
"indices": [
".tasks"
]
}
],
"indices": [
".tasks",
"test1",
"test2"
]
```
Co-authored-by: William Brafford <william.brafford@elastic.co>
Today `GET _cluster/stats` can be quite expensive, and is typically
retrieved periodically by monitoring systems (e.g. Metricbeat) that
implement a client-side timeout. When the client times out it closes the
HTTP connection in use. With this commit we react to the close of the
HTTP connection by cancelling the ongoing stats request, avoiding
unnecessary duplicated work.
Relates #55550
Part 9.
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.
We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
Closes#66656.
Current, `Docker#verifyOssInstallation(...)` checks the permissions of
`elasticsearch.keystore`, however this often causes problems when a test
forgets to wait for ES and the keystore doesn't exist yet.
Instead, add a test specifically to check `elasticsearch.keystore` and
remove the check from `verifyOssInstallation()`.
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
Closes#65980. Also closes#67097 and closes#66656.
The elasticsearch-env script used a bash feature called "process
substitution", which doesn't work when `POSIXLY_CORRECT` is set in the
environment (the construct doesn't even parse). Tweak the implementation
to use something that works under `POSIXLY_CORRECT`.
Also add a test to `ArchiveTests`, which fails without this fix, and
rework `DockerTests.test085EnvironmentVariablesAreRespectedUnderDockerExec`
since a couple of issues were open against it.
A blob store repository can be put in readonly mode by setting
`readonly: true` in its settings. In the codebase the setting key is
just the literal string `"readonly"` wherever it's used and it takes
some effort to determine what the right setting name is, in particular
to check each time that it's not spelled `"read_only"`.
This commit replaces those literal `"readonly"` strings with the
`BlobStoreRepository#READONLY_SETTING_KEY` constant to reduce this
trappiness.
Record the clusterUUID of the last cluster to write
to a repository in the `RepositoryData` and use it for more
meaningful logging when running into a concurrent modification
issue.
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.
We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
This commit mostly reverts #67934, except for the change to the version
constant `REPOSITORY_UUID_IN_REPO_DATA_VERSION`.
Completes the backport of #67829 via #67899
This commit suppresses any BWC tests related to snapshots in `master` so
that #67899 can be merged to `7.x`. It will mostly be reverted after the
merge of #67899 is complete.
Relates #66431
Calling `./bin/elasticsearch-sql-cli http://localhost:9200`under
docker failed with `ERROR: E is not a recognized option` error
message.
The root cause is that under docker the environmental variables are
automatically appended to the arguments of the sql cli. `"$@"` will
contain `-Evar=value` strings added by the `elasticsearch-env`
(line 122). This change uses the original argument list (saves it
before it is modified) when the sql cli is called.
Fixes#57744
Today a snapshot repository does not have a well-defined identity. It
can be reregistered with a different cluster under a different name, and
can even be registered with multiple clusters in readonly mode.
This presents problems for cases where we need to refer to a specific
snapshot in a globally-unique fashion. Today we rely on the repository
being registered under the same name on every cluster, but this is not a
safe assumption.
This commit adds a UUID that can be used to uniquely identify a
repository. The UUID is stored in the top-level index blob, represented
by `RepositoryData`, and is also usually copied into the
`RepositoryMetadata` that represents the repository in the cluster
state. The repository UUID is exposed in the get-repositories API; other
more meaningful consumers will be added in due course.
Stored scripts can have content_type option set, however when empty they default to XContentType.JSON#mediaType(). Commit 5e74f79 has changed this in master (ES8) method to return application/json;charset=utf-8 (previously application/json; charset=UTF-8)
This means that when upgrading ES from version 7 to 8 stored script will fail when being used as the encoder is being matched with string equality (map key)
This commit address this by adding back (in addition) the old application/json; charset=UTF-8 into the encoders map.
closes#66986
In earlier Elasticsearch versions, we had an 'upgrade API' which attempted to
upgrade an index to the current major version. This action performed a merge to
upgrade Lucene segments. The upgrade API has not worked since 5.x and was
recently deprecated and removed. So the logic for upgrading Lucene segments is
unused and can also be removed.
This PR is part of an effort to clarify our approach to index compatibility by
removing old upgrade strategies that are no longer relevant.
Closes#64824. Introduce the concept of categories to deprecation
logging. Every location where we log a deprecation message must now
include a deprecation category.
Today if a client requests a cluster state and then closes the
connection then we still do all the work of computing and serializing
the cluster state before finally dropping it all on the floor.
With this commit we introduce checks to make sure that the HTTP channel
is still open before starting the serialization process. We also make
the tasks themselves cancellable and abort any ongoing waiting if the
channel is closed (mainly to make the cancellability testing easier).
Finally we introduce a more detailed description of the task to help
identify cases where clients are inefficiently requesting more
components of the cluster state than they need.
* Adds a minimum version request parameter to SearchRequest.
The minimum version helps failing a request if any shards
involved in the search do not meet the compatibility requirements
(all shards need to have a version equal or later than the minimum
version provided).
A new version of this test dependency is finally available, enabling us
to remove a hack from production code we've long carried because of a
bug in that test dependency. This commit upgrades our tests to use
jimfs-1.2.
This commit allows returning a correct requested response content-type - it did not work for versioned media types.
It is done by adding new vendor specific instances to XContent and TextFormat enums. These instances can then "format" the response content type string when provided with parameters. This is similar to what SQL plugin does with its media types.
#51816
This finishes porting all tasks created in gradle build scripts and plugins to use
the task avoidance api (see #56610)
* Port all task definitions to task avoidance api
* Fix last task created during configuration
* Fix test setup in :modules:reindex
* Declare proper task inputs
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.
This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.
It also addresses a number of tests that would fail in approved only mode
Mainly:
Tests that use PBKDF2 with a password less than 112 bits (14char). We
elected to change the passwords used everywhere to be at least 14
characters long instead of mandating
the use of pbkdf2_stretch because both pbkdf2 and
pbkdf2_stretch are supported and allowed in fips mode and it makes sense
to test with both. We could possibly figure out the password algorithm used
for each test and adjust password length accordingly only for pbkdf2 but
there is little value in that. It's good practice to use strong passwords so if
our docs and tests use longer passwords, then it's for the best. The approach
is brittle as there is no guarantee that the next test that will be added won't
use a short password, so we add some testing documentation too.
This leaves us with a possible coverage gap since we do support passwords
as short as 6 characters but we only test with > 14 chars but the
validation itself was not tested even before. Tests can be added in a followup,
outside of fips related context.
Tests that use a PKCS12 keystore and were not already muted.
Tests that depend on running test clusters with a basic license or
using the OSS distribution as FIPS 140 support is not available in
neither of these.
Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
Closes#66107.
Bootstrap plugins are not loaded in the main Elasticsearch process, but
instead take effect only when ES is starting. As such, these plugins are
skipped when ES loads all installed plugins.
As a result, it was impossible for the plugins _cat API to report
whether any bootstrap plugins are installed.
Fix this by adjusting how the loading process skips bootstrap plugins,
and then tweaking the plugins _cat API so that bootstrap plugins can
optionally be included in the response.
This commit expands our JVM egonomics to also automatically determine
appropriate heap size based on the total available system memory as well
as the roles assigned to the node. Role determination is done via a
naive parsing of elasticsearch.yml. No settings validation is done and
only the 'node.roles' setting is taken into consideration.
For heap purposes a node falls into one of four (4) categories:
1. A 'master-only' node. This is a node with only the 'master' role.
2. A 'ml-only' node. Similarly, a node with only the 'ml' role.
3. A 'data' node. This is basically the 'other' case. A node with any
set of roles other than only master or only ml is considered a 'data'
node, to include things like coordinating-only or "tie-breaker" nodes.
4. Unknown. This is the case if legacy settings are used. In this
scenario we fallback to the old default heap options of 1GB.
In all cases we short-circuit if a user provides explicit heap options
so we only ever auto-determine heap if no existing heap options exist.
Starting with this commit the default heap settings (1GB) are now
removed from the default jvm.options which means we'll start auto-
setting heap as the new default.
This commit updates the deprecation messages that are expected on
master to account for the change to the synced flush deprecation
message that was modified in #66130.
Closes#62758.
Include the Stack log4j config in the Docker image, in order to
make it possible to write logs in a container environment in the
same way as for an archive or package deployment. This is useful
in situations where the user is bind-mounting the logs directory
and has their own arrangements for log shipping.
To use stack logging, set the environment variable `ES_LOG_STYLE`
to `file`. It can also be set to `console`, which is the same as
not specifying it at all.
The Docker logging config is now auto-generated at image build time,
by running the default config through a transformer program when
preparing the distribution in an image builder step.
Also, in the docker distribution `build.gradle`, I changed a helper
closure into a class with a static method in order to fix an
issue where the Docker image was always being rebuilt, even when
there were no changes.
Currently, if you write a date range query with numeric 'to' or 'from' bounds,
they can be interpreted as years if no format is provided. We use
"strict_date_optional_time||epoch_millis" in this case that can interpret inputs
like 1000 as the year 1000 for example.
This PR change this to always interpret and parse numbers with the "epoch_millis"
parser if no other formatter was provided.
Closes#63680
(list -> copy -> add one -> wrap immutable) is a pretty common pattern in CS
updates and tests => added a shortcut for it here and used it in easily identifyable
spots.
This ports the majority of the rest integ tests tasks to use the task avoidance api.
- There are some edge cases left that we need to investigate, but we can do that separately.
Security manager policies within plugins currently can ask to grant any
permission (though we block some within the security manager itself at
runtime). Yet most of these permissions should never be necessary, and
some we would actively not want any plugins to be allowed to use. This
commit adds validation of plugins' policy files to restrict the
permissions allowed to be granted to a subset that is reasonable for
plugins to need. The allowed permissions are not ideal (still containing
things like suppressAccessChecks), but it is a step forward in defining
a stricter model for plugins that reduces the surface area of potential
abuse.
Elasticsearch plugins can add a java security policy file to grant
additional permissions. These policy files can contain permission grants
for specific jar files, which are specified through system properties.
Unfortunately the java policy parser is lenient when a system property
is missing, meaning we can't know if there is a typo or grant for a no
longer relevant jar file.
This commit adds validation to the policy parsing by overriding the
system properties and tracking when a missing system property is used.
This PR introduces the concept of "licensed" plugins. Such plugins
may only be installed on installations of the default distribution,
and this is enforced by the plugin installer. This PR also moves
the `quote-aware-fs` plugin to the `x-pack` directory, and marks
it as licensed.
Note that I didn't move the plugin source under `x-pack/plugin`
because all the existing x-pack plugins are actually bundles as
modules into the default distribution, whereas the `quota-aware-fs`
plugin needs to remain a standalone plugin.
Closes#64735.
When running the `elasticsearch-node` tool in the archive packaging
tests with a non-standard data directory, make sure we run the tool as
the same user as Elasticsearch. Some systems place extra restrictions on
`/tmp` that prevent a different user getting a lock on `/tmp/node.lock`.
* Add tests for using ES_JAVA_OPTS with windows service
* Relocate ES_JAVA_OPTS delimiter munging
* Don't use equals for -Xmx and -Xms args
* Write newlines in temporary configs
Ryan Ernst
Armin Braun
Rene Groeschke
Mark Vieira
Przemko Robakowski
Francisco Fernández Castaño
Nhat Nguyen
Dan Hermann
Lyudmila Fokina
Rory Hunter
Jay Modi
David Turner
Christoph Büscher
Jason Tedor
Yash Jipkate
Jim Ferenczi
Yang Wang
Joe Gallo
Przemyslaw Gomulka
Przemysław Witek
Akash Gupta
Yannick Welsch
Gordon Brown
Andras Palinkas
Julie Tibshirani
Andrei Stefan
Tanguy Leroux
Ioannis Kakavas
Alan Woodward
William Brafford