This introduces a basic public yaml rest test plugin that is supposed to be used by external
elasticsearch plugin authors. This is driven by #76215
- Rename yaml-rest-test to intern-yaml-rest-test
- Use public yaml plugin in example plugins
Co-authored-by: Mark Vieira <portugee@gmail.com>
just configuring common conventions on java based projects without
adding opinionated sourcesets. Reduces the configuration overhead for
yaml rest test only projects.
In the end we create less tasks and configure less for test only projects.
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
* Warn users if security is implicitly disabled
Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
clusters.
This change introduces clear warnings when security features are
implicitly disabled.
- a warning header in each REST response if security is implicitly
disabled;
- a log message during cluster boot.
this commit moves most of the coordination of setting up a rollup index
to the master node RollupAction. The indexing and broadcast action,
RollupIndexerAction, was created and split out as its own action to handle
indexing into the rollup index.
Closes#68296.
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
This commit updates the Rollup qa directory to leverage the
updated convention of running yaml tests. This also fixes a bug
wherein the qa tests were not running on `rollup:check`.
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.
This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.
It also addresses a number of tests that would fail in approved only mode
Mainly:
Tests that use PBKDF2 with a password less than 112 bits (14char). We
elected to change the passwords used everywhere to be at least 14
characters long instead of mandating
the use of pbkdf2_stretch because both pbkdf2 and
pbkdf2_stretch are supported and allowed in fips mode and it makes sense
to test with both. We could possibly figure out the password algorithm used
for each test and adjust password length accordingly only for pbkdf2 but
there is little value in that. It's good practice to use strong passwords so if
our docs and tests use longer passwords, then it's for the best. The approach
is brittle as there is no guarantee that the next test that will be added won't
use a short password, so we add some testing documentation too.
This leaves us with a possible coverage gap since we do support passwords
as short as 6 characters but we only test with > 14 chars but the
validation itself was not tested even before. Tests can be added in a followup,
outside of fips related context.
Tests that use a PKCS12 keystore and were not already muted.
Tests that depend on running test clusters with a basic license or
using the OSS distribution as FIPS 140 support is not available in
neither of these.
Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
This commit moves the ownership of tracking the rollup_index from
the RollupActionConfig to the RollupAction.Request.
This is cleaner since the config should not be concerned with the
source and rollup indices.
relates #42720.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
This ports the majority of the rest integ tests tasks to use the task avoidance api.
- There are some edge cases left that we need to investigate, but we can do that separately.
This commit adds a new endpoint for Rollup V2, a new way to rollup indices.
Instead of relying on a cron-job, this action will rollup a whole index on the spot.
When an index is rolled up using a Rollup Config, it does the following
1. check that original index is read-only
2. runs an aggregation and indexes results into a temporary hidden rollup index
3. "resizes" the temporary index into the final rollup-index (in-place segment pointer juggling)
4. adds RollupMetadata about the rollup-group (keyed by original index name) and adds
custom index-metadata with data about what the rollup index's original index is so that its
group information in RollupMetadata can be looked up
example usage:
```
POST /_rollup_vtwo/index
{
"rollup_index": "index_rolled",
"groups": {
"date_histogram": {
"field": "date",
"calendar_interval": "1M"
},
"terms": {
"fields": ["unit"]
}
},
"metrics": [
{
"field": "temperature",
"metrics": ["sum"]
}
]
}
```
relates #42720.
Rene Groeschke
Lyudmila Fokina
Joe Gallo
Mark Vieira
Tal Levy
Ioannis Kakavas