Commit Graph

21 Commits

Author SHA1 Message Date
Rene Groeschke b323726ebd
Resolve system properties in build scripts via provider factory (#76199)
This allows tracking system properties used in the build configuration and brings us
one step closer to be gradle configuration cache compliant.
2021-08-09 09:39:30 +02:00
Ryan Ernst 68817d7ca2
Rename o.e.common in libs/core to o.e.core (#73909)
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.

relates #73784
2021-06-08 09:53:28 -07:00
Rene Groeschke e609e07cfe
Remove internal build logic from public build tool plugins (#72470)
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic

This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.

It also introduces a set of internal versions of public plugins.

As part of this we also generate the plugin descriptors now.

As a follow up on this we can actually move these public used classes into 
an extra project (declared as included build)

We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
2021-05-06 14:02:35 +02:00
Rene Groeschke 5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Przemko Robakowski 308aee283d
Update GeoIP processor documentation (#71211)
This PR adds documentation for GeoIPv2 auto-update feature.
It also changes related settings names from geoip.downloader.* to ingest.geoip.downloader to have the same convention as current setting.

Relates to #68920

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2021-04-15 13:47:09 +02:00
Przemko Robakowski 39eb12a972
Enable GeoIP downloader by default (#71505)
This change enables GeoIP downloader by default.
It removes feature flag but adds flag that is used by tests to disable it again (as we don't want to hammer GeoIP database service with every test cluster we spin up).

Relates to #68920
2021-04-15 12:28:37 +02:00
Lyudmila Fokina 3b0b7941ae
Warn users if security is implicitly disabled (#70114)
* Warn users if security is implicitly disabled

Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
 clusters.
 This change introduces clear warnings when security features are
 implicitly disabled.
 - a warning header in each REST response if security is implicitly
 disabled;
 - a log message during cluster boot.
2021-04-13 18:33:41 +02:00
Yash Jipkate 60f4d22722
Change default value of `action.destructive_requires_name` to True. (#66908)
This PR sets the default value of `action.destructive_requires_name`
to `true.` Fixes #61074. Additionally, we set this value explicitly in
test classes that rely on wildcard deletions to clear test state.
2021-03-31 15:59:57 -04:00
Rene Groeschke 24adbbca41
Remove OSS specific distribution packaging (#69775)
* Remove OSS projects from distribution packaging
* Remove OSS flavor from ElasticsearchDistribution
* Remove OSS handling from docker build setup
2021-03-03 13:39:29 +01:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
Ioannis Kakavas bd873698bc
Ensure CI is run in FIPS 140 approved only mode (#64024)
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.

This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.

It also addresses a number of tests that would fail in approved only mode
Mainly:

    Tests that use PBKDF2 with a password less than 112 bits (14char). We
    elected to change the passwords used everywhere to be at least 14
    characters long instead of mandating
    the use of pbkdf2_stretch because both pbkdf2 and
    pbkdf2_stretch are supported and allowed in fips mode and it makes sense
    to test with both. We could possibly figure out the password algorithm used
    for each test and adjust password length accordingly only for pbkdf2 but
    there is little value in that. It's good practice to use strong passwords so if
    our docs and tests use longer passwords, then it's for the best. The approach
    is brittle as there is no guarantee that the next test that will be added won't
    use a short password, so we add some testing documentation too.
    This leaves us with a possible coverage gap since we do support passwords
    as short as 6 characters but we only test with > 14 chars but the
    validation itself was not tested even before. Tests can be added in a followup,
    outside of fips related context.

    Tests that use a PKCS12 keystore and were not already muted.

    Tests that depend on running test clusters with a basic license or
    using the OSS distribution as FIPS 140 support is not available in
    neither of these.

Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
2020-12-23 21:00:49 +02:00
Rene Groeschke defaa93902
Avoid tasks materialized during configuration phase (#65922)
* Avoid tasks materialized during configuration phase
* Fix RestTestFromSnippet testRoot setup
2020-12-12 16:14:17 +01:00
Rory Hunter ec350157ea
Use a tiny base image for Docker builds (#52519)
Closes #51670, closes #50838.

Introduce a tiny base image for Docker builds. It aims to create a basic filesystem with as little as possible, which is mostly glibc, busybox and bash. A statically-built curl is also provided.

We still use CentOS 8 as a base. All the fun stuff happens in the Dockerfile.
2020-10-12 14:58:01 +01:00
Rene Groeschke a72760e55b
Replace immediate task creations by using task avoidance api (#60071)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 11:29:14 +02:00
Rene Groeschke e289e7c4e7
Split internal distribution handling into separate internal plugin (#57869)
* Split internal distribution logic into separate plugin
* Remove moved func test from DistributionDownloadPluginIT
* Assert internal plugin is only appied on internal build
* Extract common abstract test specification for plugin func tests
* Provide proper failure if unexpected non jdk bundled bwc version is requested
2020-07-27 16:37:12 +02:00
Rene Groeschke 680ea07f7f
Remove deprecated usage of testCompile configuration (#57921)
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
2020-06-12 13:34:53 +02:00
Stuart Tettemer bd64da0960
Scripting: Deprecate general cache settings (#55038)
* Scripting: Deprecate general cache settings

* Add script.disable_max_compilations_rate setting

* Move construction to ScriptCache

* Use ScriptService to do updates of CacheHolder

* Remove fallbacks

* Add SCRIPT_DISABLE_MAX_COMPILATIONS_RATE_SETTING to ClusterSettings

* Node scope

* Use back compat

* 8.0 for bwc

* script.max_compilations_rate=2048/1m -> script.disable_max_compilations_rate=true in docker compose

* do not guard in esnode

* Doc update

* isSnapshotBuild() -> systemProperty 'es.script.disable_max_compilations_rate', 'true'

* Do not use snapshot in gradle to set max_compilations_rate

* Expose cacheHolder as package private

* monospace 75/5m in cbreaker docs, single space in using

* More detail in general compilation rate error

* Test: don't modify defaultConfig on upgrade
2020-04-22 12:33:33 -06:00
Rory Hunter a45b4eb809
Define aarch64 packaging test tasks (#55073)
We added tasks to build an ARM distribution and Docker image, but didn't
provide any way to run packaging tests against them. Add extra loops on
the possible Architecture values, and skip tasks that can't be run on
the current Architecture.
2020-04-15 12:12:07 +01:00
Jake Landis 810dc9fce3
Smarter copying of the rest specs and tests (#52114)
This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default 
the rest specs will get copied if the project applies 
`elasticsearch.standalone-rest-test` or `esplugin` and the project 
has rest tests or you configure the custom extension `restResources`. 

This PR also removes the need for dozens of places where the x-pack 
specs were copied by supporting copying of the x-pack rest specs too. 

The plugin/task introduced here can also copy the rest tests to the 
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of 
which rest specs are copied. Per project can be configured to include 
only a subset of the specs (or tests). Configuring a project to only 
copy the specs when actually needed should help with build cache hit 
rates since we can better define what is actually in use. 
However, project level optimizations for build cache hit rates are 
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on 
integTest task.  

The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources` 
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy 
* new extension 'restResources'
```
restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

```
2020-02-25 18:46:32 -06:00
Mark Vieira c1a1047e42
Consolidate docker availability build logic (#52548) 2020-02-21 08:11:50 -08:00
Yannick Welsch 10b7ffab2f
Add Docker-based tests for proxy connection mode (#51658)
Adds more tests for the new "proxy" remote cluster connection mode, using a Docker-based
setup, as well as testing SNI-based routing using HAProxy.

- Checks that the new proxy mode can work in situations where the publish host of the nodes in
the remote cluster are not routable.
- Checks that the new proxy mode can work with clusters where nodes are hidden behind
HAProxy.
- Checks that the new proxy mode can work with clusters where nodes are hidden behind
HAProxy, using SNI to identify the nodes/cluster behind HAProxy.

Relates #49067
2020-02-03 09:44:55 +01:00