Commit Graph

39 Commits

Author SHA1 Message Date
Rene Groeschke b323726ebd
Resolve system properties in build scripts via provider factory (#76199)
This allows tracking system properties used in the build configuration and brings us
one step closer to be gradle configuration cache compliant.
2021-08-09 09:39:30 +02:00
Rene Groeschke e609e07cfe
Remove internal build logic from public build tool plugins (#72470)
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic

This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.

It also introduces a set of internal versions of public plugins.

As part of this we also generate the plugin descriptors now.

As a follow up on this we can actually move these public used classes into 
an extra project (declared as included build)

We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
2021-05-06 14:02:35 +02:00
Lyudmila Fokina 3b0b7941ae
Warn users if security is implicitly disabled (#70114)
* Warn users if security is implicitly disabled

Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
 clusters.
 This change introduces clear warnings when security features are
 implicitly disabled.
 - a warning header in each REST response if security is implicitly
 disabled;
 - a log message during cluster boot.
2021-04-13 18:33:41 +02:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
Rene Groeschke defaa93902
Avoid tasks materialized during configuration phase (#65922)
* Avoid tasks materialized during configuration phase
* Fix RestTestFromSnippet testRoot setup
2020-12-12 16:14:17 +01:00
Rene Groeschke 97749a3372
Port rest integ tests to use task avoidance api (#65011)
This ports the majority of the rest integ tests tasks to use the task avoidance api.

- There are some edge cases left that we need to investigate, but we can do that separately.
2020-11-26 10:30:06 +01:00
Lee Hinman 1872e970ec
Make stack.templates.enabled a dynamic setting (#63764)
* Make stack.templates.enabled a dynamic setting

This change allows the setting for disabling the automatically installed stack templates (the
`logs-*-*`, `metrics-*-*`, and `synthetics-*-*` templates) to be changed dynamically.

As a byproduct, it also moves thes `IndexTemplateRegistry` to use an `initialize()` method so that
constructors are not tempted to use a `this` reference in the constructor (see #37861 for more
information about why to avoid that).

Resolves #62835
2020-10-16 10:25:46 -06:00
Rene Groeschke dd74be0f83
Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
2020-08-03 12:07:41 +02:00
Jake Landis 333a5d8cdf
Create plugin for yamlTest task (#56841)
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.

The remaining cases in modules, plugins, and x-pack will be handled in followups.

This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.

The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.

Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).

As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.

Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
2020-07-06 12:13:01 -05:00
Lee Hinman 3b68df2355
Add default composable templates for new indexing strategy (#57629)
This commit adds the component and composable templates, as well as ILM policies, for the new
default indexing strategy. It installs:

- logs-default-mappings (component)
- logs-default-settings (component)
- logs-default-policy (ilm policy)
- logs-default-template (composable template)
- metrics-default-mappings (component)
- metrics-default-settings (component)
- metrics-default-policy (ilm policy)
- metrics-default-template (composable template)

These templates and policies are managed by a new x-pack module, `stack`, and can be disabled by
setting `stack.templates.enabled` to `false`.

These ensure that patterns for the `logs-*-*` and `metrics-*-*` indices are set up to create data
streams with the proper mappings and settings.

This also makes changes to the `IndexTemplateRegistry` to support installing component and
composable templates (previously it supported only legacy templates).

Resolves #56709
2020-06-30 09:19:37 -06:00
Rene Groeschke 680ea07f7f
Remove deprecated usage of testCompile configuration (#57921)
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
2020-06-12 13:34:53 +02:00
William Brafford 92c8a73348
Deprecate disabling basic-license features (#54816)
We believe there's no longer a need to be able to disable basic-license
features completely using the "xpack.*.enabled" settings. If users don't
want to use those features, they simply don't need to use them. Having
such features always available lets us build more complex features that
assume basic-license features are present.

This commit deprecates settings of the form "xpack.*.enabled" for
basic-license features, excluding "security", which is a special case.
It also removes deprecated settings from integration tests and unit
tests where they're not directly relevant; e.g. monitoring and ILM are
no longer disabled in many integration tests.
2020-04-17 09:18:55 -04:00
Lee Hinman 984670b34d
Set feature flags for IndexTemplatesV2 and Data Streams in top… (#53897)
Resolves #53892
2020-03-20 15:11:49 -06:00
Martijn van Groningen 12046084f9
Initial data stream commit (#53666)
* Initial data stream commit

This commits adds a data stream feature flag, initial definition of a data stream and
the stubs for the data stream create, delete and get APIs. Also simple serialization
tests are added and a rest test to thest the data stream API stubs.

This is a large amount of code and mainly mechanical, but this commit should be
straightforward to review, because there isn't any real logic.

The data stream transport and rest action are behind the data stream feature flag and
are only intialized if the feature flag is enabled. The feature flag is enabled if
elasticsearch is build as snapshot or a release build and the
'es.datastreams_feature_flag_registered' is enabled.

The integ-test-zip sets the feature flag if building a release build, otherwise
rest tests would fail.

Relates to #53100

* fixed hlrc test

* ignore bwc until this change has been backported to 7.x branch

* changed data stream apis to be a cluster based action.

before this commit the data steams api were indices based actions,
but data streams aren't indices, data streams encapsulates indices,
but are indices themselves. It is a cluster level attribute, and
therefor cluster based action fits best for now.

Perhaps in the future we will have data stream based actions and
then this would be a right fit for the data stream crud apis.

* this should have been part of the previous commit

* fixed yaml test

* Also add feature flag in other modules that run the yaml test if a release build is executed

* Reverted the commits that make data stream a cluster based api

This reverts commit e362eeb669.

* Make data stream crud apis work like a indices based api.

* renamed timestamp field

* fixed compile error after merging in master

* fixed merge mistake

* moved setting system property

* applied review comments
2020-03-20 11:22:18 +01:00
Lee Hinman d47d74a558
Fix feature flag setting for ComponentTemplate APIs (#53758)
The feature flag was set for *most* of the builds, but there are a couple where it was missing.

Resolves #53708
2020-03-19 08:00:06 -06:00
Jake Landis 810dc9fce3
Smarter copying of the rest specs and tests (#52114)
This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default 
the rest specs will get copied if the project applies 
`elasticsearch.standalone-rest-test` or `esplugin` and the project 
has rest tests or you configure the custom extension `restResources`. 

This PR also removes the need for dozens of places where the x-pack 
specs were copied by supporting copying of the x-pack rest specs too. 

The plugin/task introduced here can also copy the rest tests to the 
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of 
which rest specs are copied. Per project can be configured to include 
only a subset of the specs (or tests). Configuring a project to only 
copy the specs when actually needed should help with build cache hit 
rates since we can better define what is actually in use. 
However, project level optimizations for build cache hit rates are 
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on 
integTest task.  

The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources` 
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy 
* new extension 'restResources'
```
restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

```
2020-02-25 18:46:32 -06:00
Rory Hunter 3a3e5f6176
Apply 2-space indent to all gradle scripts (#48849)
Closes #48724. Update `.editorconfig` to make the Java settings the default
for all files, and then apply a 2-space indent to all `*.gradle` files.
Then reformat all the files.
2019-11-13 10:14:04 +00:00
Alan Woodward 566e1b7d33
Remove type field from DocWriteRequest and associated Response objects (#47671)
This commit removes the type field from index, update and delete requests, and their
associated responses.

Relates to #41059
2019-10-11 10:23:55 +01:00
Ryan Ernst 7e87c4141b
Convert testclusters to use distro download plugin (#44253)
Test clusters currently has its own set of logic for dealing with
finding different versions of Elasticsearch, downloading them, and
extracting them. This commit converts testclusters to use the
DistributionDownloadPlugin.
2019-07-15 10:39:05 -07:00
Alpar Torok 01923ae514
Test clusters: convert x-pack qa tests (#43283) 2019-06-24 12:13:38 +03:00
Nhat Nguyen 8b0a74f11c
Clean up outdated skip statements in yaml tests (#41165)
These skip statements become no-ops in 8.0 for we don't support
a mixed cluster between 6.x and 8.0.

Relates #41164
2019-04-18 14:19:31 -04:00
Martijn van Groningen 7543478faf
Remove xpack dependencies from qa rest modules (#41134)
This commit removes xpack dependencies of many xpack qa modules.
(for some qa modules this will require some more work)

The reason behind this change is that qa rest modules should not depend
on the x-pack plugins, because the plugins are an implementation detail and
the tests should only know about the rest interface and qa cluster that is
being tested.

Also some qa modules rely on xpack plugins and hlrc (which is a valid
dependency for rest qa tests) creates a cyclic dependency and this is
something that we should avoid.  Also Eclipse can't handle gradle cyclic
dependencies (see #41064).

* don't copy xpack-core's plugin property into the test resource of qa
modules. Otherwise installing security manager fails, because it tries
to find the XPackPlugin class.
2019-04-15 09:28:58 +02:00
Jason Tedor 5de236e1e7
Rename ILM, ILM endpoints and drop _xpack (#32564)
This commit does the following:
 - renames index-lifecycle plugin to ilm
 - modifies the endpoints to ilm instead of index_lifecycle
 - drops _xpack from the endpoints
 - drops a few duplicate endpoints
2018-08-02 13:05:11 -04:00
Colin Goodheart-Smithe a2abc86595
Merge branch 'master' into index-lifecycle 2018-07-24 13:13:52 +01:00
Christoph Büscher fe6bb75eb4
Rename ranking evaluation `quality_level` to `metric_score` (#32168)
The notion of "quality" is an overloaded term in the search ranking evaluation 
context. Its usually used to decribe certain levels of "good" vs. "bad" of a 
seach result with respect to the users information need. We currently report the 
result of the ranking evaluation as `quality_level` which is a bit missleading.
This changes the response parameter name to `metric_score` which fits better.
2018-07-23 22:25:02 +02:00
Colin Goodheart-Smithe 5f696e15ea
Merge branch 'master' into index-lifecycle 2018-07-23 14:49:40 +01:00
Luca Cavanna 00a6ad0e9e
Remove aliases resolution limitations when security is enabled (#31952)
Resolving wildcards in aliases expression is challenging as we may end
up with no aliases to replace the original expression with, but if we
replace with an empty array that means _all which is quite the opposite.
Now that we support and serialize the original requested aliases,
whenever aliases are replaced we will be able to know what was
initially requested. `MetaData#findAliases` can then be updated to not
return anything in case it gets empty aliases, but the original aliases
were not empty. That means that empty aliases are interpreted as _all
only if they were originally requested that way.

Relates to #31516
2018-07-20 09:23:32 +02:00
Tal Levy e9cd0556c1 Merge branch 'master' into index-lifecycle 2018-07-09 19:44:12 -04:00
Konrad Beiske 2971dd56ca Enable setting client path prefix to / (#30119)
Some proxies require all requests to have paths starting with / since
there are no relative paths at the HTTP connection level. Elasticsearch
assumes paths are absolute. In order to run rest tests against a cluster
behind such a proxy, set the system property
tests.rest.client_path_prefix to /.
2018-07-01 13:42:03 -04:00
Nik Everett 232c71b6bf
QA: Create xpack yaml features (#31403)
This creates a YAML test "features" that indices if the cluster being
tested has xpack installed (`xpack`) or if it does *not* have xpack
installed (`no_xpack`). It uses those features to centralize skipping
a few tests that fail if xpack is installed.

The plan is to use this in a followup to skip docs tests that require
xpack when xpack is not installed. We *plan* to use the declaration
of required license level on the docs page to generate the required
`skip`.

Closes #30933.
2018-06-26 09:26:48 -04:00
Colin Goodheart-Smithe ce38127a16
Merge branch 'master' into index-lifecycle 2018-06-14 13:02:24 +01:00
Luca Cavanna ce245a7320
Remove RestGetAllAliasesAction (#31308)
We currently have a specific REST action to retrieve all aliaes, which
uses internally the get index API. This doesn't seem to be required
anymore though as the existing RestGetAliaesAction could as well take
the requests with no indices and aliases specified.

This commit removes the RestGetAllAliasesAction in favour of using
RestGetAliasesAction also for requests that don't specify indices nor
aliases. Similar to #31129.
2018-06-14 11:21:16 +02:00
Tal Levy 0541684df3 Merge branch 'master' into index-lifecycle 2018-06-11 17:20:05 -07:00
Jason Tedor cb952bd9ec
Enable custom credentials for core REST tests (#31235)
The core REST tests with security currently use a hardcoded username and
password. This is not amenable to running these tests in scenarios where
the user controls the creation of the cluster and owns the credentials
for this cluster. This commit enables running the core REST tests with
security with a custom username and password.
2018-06-11 16:53:40 -04:00
Tal Levy 1df6b6c65d Applying changes required for ILM after merge 2018-04-25 13:04:06 -07:00
Michael Basnight 72f57c8e72
Add comments inadvertently removed during migrate
A few files had their first comment removed even though it did not
contain a license. This re-adds those comments.
2018-04-24 13:41:09 -05:00
Jason Tedor c7c0e330b8 Rename users
This commit renames users to elasticsearch-users.
2018-04-20 15:34:01 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst 2efd22454a Migrate x-pack-elasticsearch source to elasticsearch 2018-04-20 15:29:54 -07:00