Commit Graph

612 Commits

Author SHA1 Message Date
Jake Landis 26dfe02a0b
Update task names for rest compatiblity (#75267)
This commit updates two task names:
```
yamlRestCompatTest -> yamlRestTestV7CompatTest
transformV7RestTests -> yamlRestTestV7CompatTransform 
```

`7` is the N-1 version and calculated, such that when `8` is 
N-1 version the task names will be  `yamlRestTestV8CompatTest` and 
`yamlRestTestV8CompatTransform`

The motivation for `yamlRestCompatTest -> yamlRestTestV7CompatTest`  is that 
many projects have configured `yamlRestCompatTest` 
but that configuration is specific to the N-1 version. For example, 
if we blacklist tests when running compatibility with v7, we don't also
want to blacklist those tests when running compatibility with v8.

By introducing a version-specific identifier in the name, the task will not
even exist when bumping the version creating the need to (correctly) remove
the version-specific condition.

The motivation for `transformV7RestTests -> yamlRestTestV7CompatTransform` 
is to provide more consistent naming. 

The idea behind the naming is the main task people
are likely familiar with is :

`yamlRestTest` so we will use that as a base.
`yamlRestTestV7CompatTest` to run the version-specific compat tests
`yamlRestTestV7CompatTransform` to run the version-specific transformations for the compat tests

CI should be un-effected since since we introduced a lifecycle task 
name `checkRestCompat` which is what CI should be configured to use.
2021-09-03 11:26:11 -05:00
Seth Michael Larson eaf4ac26cc
Convert 'routing' values in REST API tests to strings
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-09-03 07:33:26 -05:00
Rene Groeschke 35ec6f348c
Introduce simple public yaml-rest-test plugin (#76554)
This introduces a basic public yaml rest test plugin that is supposed to be used by external 
elasticsearch plugin authors. This is driven by #76215

- Rename yaml-rest-test to intern-yaml-rest-test
- Use public yaml plugin in example plugins

Co-authored-by: Mark Vieira <portugee@gmail.com>
2021-08-31 08:45:52 +02:00
Przemyslaw Gomulka 127015e54c
[Rest Api Compatibility] Clean up blocklist (#76179)
v7compatibilityNotSupportedTests was introduced to make it easier to
track tests that have been identified as not needing compatible changes
and those that still need to be checked.
We have checked all tests now and the separate list is no longer needed.

relates #51816
relates #73912
2021-08-19 10:09:46 +02:00
Mark Vieira 13795c4e5e
Use Adoptium builds for bundled JDK (#76631) 2021-08-18 09:52:22 -07:00
Rory Hunter d01efa4fd6
Changes to keep Checkstyle happy after reformatting (#76464)
* Reformatting to keep Checkstyle after formatting

* Configure spotless everywhere, and disable the tasks if necessary

* Add XContentBuilder helpers, fix test

* Tweaks

* Add a TODO

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-18 07:15:55 -04:00
Nik Everett e305a6bed7
Name `BulkItemResponse` ctors (#76439)
* Name `BulkItemResponse` ctors

`BulkItemResponse` can contain either a success or failure. This
replaces the two constructors used to build either case with named
static methods. So instead of
```
return new BulkItemResponse(0, OpType.CREATE, createResponse);
return new BulkItemResponse(0, OpType.CREATE, failure);
```
you now use
```
return BulkItemResponse.success(0, OpType.CREATE, createResponse);
return BulkItemResponse.failure(0, OpType.CREATE, failure);
```

This makes it marginally easier to read code building these things - you
don't have to know the type of the parameter to know if its a failure
or success.

* Consistent

* Mock response
2021-08-12 14:41:26 -04:00
Henning Andersen b19f601ae0
Remote reindex stored fields since 5.0.0 (#74914)
Test would assume stored fields were available since 5.0.0.alpha4,
while production code only enables them after 5.0.0 (final, aka 99).

Closes #74131
2021-08-04 16:06:17 +02:00
David Turner 35b016cd89
Log reindex bulk request completion (#75935)
Today we log at `DEBUG` when we receive a scroll response and send the
bulk request but do not log the completion of the bulk request or the
start of the next scroll request. This makes it impossible to tell from
the logs how long these things are taking.

This commit adds the missing logging.
2021-08-02 15:57:18 +01:00
Mark Vieira 9d14bc91d7
Set netty available processors system property for tests globally (#75699) 2021-07-27 11:21:42 -07:00
Tim Vernum 940a890958
Update "ssl-config" to support X-Pack features (#74887)
This commit upgrades the existing SSPL licensed "ssl-config" library
to include additional features that are supported by the X-Pack SSL
library.

This commit does not make any changes to X-Pack to use these new
features - it introduces them in preparation for their future use by
X-Pack.

The reindex module is updated to reflect API changes in ssl-config
2021-07-15 19:46:00 +10:00
Przemyslaw Gomulka 1f726800a6
[Rest Api Compatibility] Clean up x-pack/plugin rest compat tests (#74701)
this PR removes tests which are not meant to be fixed (ml/, vectors/) to a separate "not to be fixed list" so that we can see which compatible changes are meant to be implemented.

relates #51816
2021-07-08 11:04:55 +02:00
Przemyslaw Gomulka 744c75d34a
[Rest Api Compatibility] Rename BulkItemResponse.Failure type field (#74937)
it should return type instead of _type
relates #51816
2021-07-06 10:50:38 +02:00
Ryan Ernst ab1a2e4a84
Add precommit task for detecting split packages (#73784)
Modularization of the JDK has been ongoing for several years. Recently
in Java 16 the JDK began enforcing module boundaries by default. While
Elasticsearch does not yet use the module system directly, there are
some side effects even for those projects not modularized (eg #73517).
Before we can even begin to think about how to modularize, we must
Prepare The Way by enforcing packages only exist in a single jar file,
since the module system does not allow packages to coexist in multiple
modules.

This commit adds a precommit check to the build which detects split
packages. The expectation is that we will add the existing split
packages to the ignore list so that any new classes will not exacerbate
the problem, and the work to cleanup these split packages can be
parallelized.

relates #73525
2021-06-08 15:04:23 -07:00
Ryan Ernst 63012c8a40
Move ParseField to o.e.c.xcontent (#73923)
ParseField is part of the x-content lib, yet it doesn't exist under the
same root package as the rest of the lib. This commit moves the class to
the appropriate package.

relates #73784
2021-06-08 13:32:14 -07:00
Ryan Ernst 68817d7ca2
Rename o.e.common in libs/core to o.e.core (#73909)
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.

relates #73784
2021-06-08 09:53:28 -07:00
Ryan Ernst 64054de1ac
Rename bootstrap package in core jar (#73788)
The org.elasticsearch.bootstrap package exists in server with classes
for starting up Elasticsearch. The elasticsearch-core jar has a handful
of classes that were split out from there, namely java version parsing
and jarhell. This commit moves those classes to a new
org.elasticsearch.jdk package so as to not split the server owned
bootstrap package.

relates #73784
2021-06-07 08:14:44 -07:00
David Turner b2956b3ae7
Identify cancelled tasks in list tasks API (#72931)
This commit adds a `cancelled` flag to each cancellable task in the
response to the list tasks API, allowing users to see that a task has
been properly cancelled and will complete as soon as possible.

Closes #72907
2021-05-17 11:02:50 +01:00
Przemyslaw Gomulka 85ed9100df
[Rest Api Compatibility] Typed endpoints for search and related endpoints (#72155)
Implements a V7 compatible typed endpoints for REST for search related apis
retrofits the REST layer change removed in #41640

relates main meta issue #51816
relates types removal issue #54160
2021-05-12 21:00:19 +02:00
Armin Braun 47c77160ef
Fix ListenableFuture Resolving Listeners under Mutex (#71943) (#72087)
We shouldn't loop over the listeners under the mutex in `done` since in most use-cases we used `DirectExecutorService`
with this class.
Also, no need to create an `AbstractRunnable` for direct execution. We use this listener on the hot path in authentication
making this a worthwhile optimization I think.
Lastly, no need to clear and thus loop over `listeners`, the list is not used again after the `done` call returns anyway
so no point in retaining it at all (especially when in a number of use cases we add listeners only after the `done` call
so we can also save the instantiation by making the field non-final).
2021-04-26 19:33:34 +02:00
Rene Groeschke 5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Francisco Fernández Castaño 9d8fb9fba2
Use the remaining scroll response documents on update by query bulk requests (#71430)
In update by query requests where max_docs < size and conflicts=proceed
we weren't using the remaining documents from the scroll response in
cases where there were conflicts and in the first bulk request the
successful updates < max_docs. This commit address that problem and
use the remaining documents from the scroll response instead of
requesting a new page.

Closes #63671
2021-04-20 14:45:17 +02:00
David Turner b690798348
Reduce size of MANAGEMENT threadpool on small node (#71171)
Today by default the `MANAGEMENT` threadpool always permits 5 threads
even if the node has a single CPU, which unfairly prioritises management
activities on small nodes. With this commit we limit the size of this
threadpool to the number of processors if less than 5.

Relates #70435
2021-04-06 12:58:07 +01:00
Jason Tedor 32314493a2
Pass override settings when creating test cluster (#71203)
Today when creating an internal test cluster, we allow the test to
supply the node settings that are applied. The extension point to
provide these settings has a single integer parameter, indicating the
index (zero-based) of the node being constructed. This allows the test
to make some decisions about the settings to return, but it is too
simplistic. For example, imagine a test that wants to provide a setting,
but some values for that setting are not valid on non-data nodes. Since
the only information the test has about the node being constructed is
its index, it does not have sufficient information to determine if the
node being constructed is a non-data node or not, since this is done by
the test framework externally by overriding the final settings with
specific settings that dicate the roles of the node. This commit changes
the test framework so that the test has information about what settings
are going to be overriden by the test framework after the test provide
its test-specific settings. This allows the test to make informed
decisions about what values it can return to the test framework.
2021-04-02 10:20:36 -04:00
Mark Vieira 9dd82f8200 Format JVM option properly 2021-03-30 17:41:59 -07:00
Mark Vieira e707a6372b Workaround for running old ES fixture on certain aarch64 systems 2021-03-30 16:37:31 -07:00
Mark Vieira 6339691fe3
Consolidate REST API specifications and publish under Apache 2.0 license (#70036) 2021-03-26 16:20:14 -07:00
Przemyslaw Gomulka 91956145f0
[Rest Compatible Api] update and delete by query using size field (#69606)
A follow up after #69037 which added back size field for reindex api.
The original PR #43373 also removed the size field from update by query and delete by query APIs.
This commits allow to use size field with Compatible API for update_by_query and delete_by_query apis
2021-03-15 09:17:53 +01:00
Jay Modi 1487a5a991
Introduce system index types including external (#68919)
This commit introduces system index types that will be used to
differentiate behavior. Previously system indices were all treated the
same regardless of whether they belonged to Elasticsearch, a stack
component, or one of our solutions. Upon further discussion and
analysis this decision was not in the best interest of the various
teams and instead a new type of system index was needed. These system
indices will be referred to as external system indices. Within external
system indices, an option exists for these indices to be managed by
Elasticsearch or to be managed by the external product.

In order to represent this within Elasticsearch, each system index will
have a type and this type will be used to control behavior.

Closes #67383
2021-03-01 10:38:53 -07:00
Armin Braun 4ce162a3dc
Forbid ActionListener#onFailure to Throw in more Places (#69420)
Same assertion we have in `.map` we can extend to more places
with recent changes to prevent unexpected listener behavior going forward.
2021-02-27 21:21:09 +01:00
Joe Gallo f3aac00f5d
Apply yaml-rest-compat-test to rest-api-spec (#69462) 2021-02-23 13:15:53 -05:00
Armin Braun ad9760325c
Some Enhancements to ActionListener (#69103)
This adds best effort `toString` rendering the various wrapping
action listeners to make `TRACE` logging, that will currently only print the
top level listener `toString` which isn't helpful to find the original of a listener
in case of wrapped listeners, more useful (e.g. when logging rejected executions).
Also this change makes the `delegateX` methods less verbose to use and makes use of them
in a few spots where they weren't yet used.
2021-02-23 07:42:23 +01:00
Mark Vieira 2b5a103591
Ensure that rest compatibility test task classpath is setup correctly (#69306) 2021-02-22 10:49:33 -08:00
Mark Vieira f2b27c18c2 Mute reindex rest compatibility tests 2021-02-18 10:10:14 -08:00
Jason Tedor 0cd4863585
Introduce ES_JAVA_HOME (#68954)
This commit introduces a dedicated envirnoment variable ES_JAVA_HOME to
determine the JDK used to start (if not using the bundled JDK). This
environment variable will replace JAVA_HOME. The reason that we are
making this change is because JAVA_HOME is a common environment variable
and sometimes users have it set in their environment from other JDK
applications that they have installed on their system. In this case,
they would accidentally end up not using the bundled JDK despite their
intentions. By using a dedicated environment variable specific to
Elasticsearch, we avoid this potential for conflict. With this commit,
we introduce the new environment variable, and deprecate the use of
JAVA_HOME. We will remove support for JAVA_HOME in a future commit.
2021-02-17 12:41:23 -05:00
Przemyslaw Gomulka 587e5b4ab9
[REST Compatible API] Reindex size field (#69037)
For re-index requests, the outer most "size" field was deprecated and
renamed to "max_docs" in 7.x (#43373). With this commit, the "size" field will
remain available (but still deprecated) through out 8.x when REST API
compatibility is requested.

relates #51816
2021-02-17 07:59:31 +01:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
Gordon Brown 4fe7a612fc
Allow population of Enrich indices to work with System Index protections (#67406)
This PR does three things:
1) Tweaks existing reindex infrastructure so that different clients can be used for the "search" part and the "index" part of a reindex operation, and
2) Modifies Enrich to take advantage of this to perform the "search" part in the security context of the current user (so that DLS/FLS etc. are properly applied) while performing the "index" part in the security context of the Enrich plugin (so that access to system indices, and `.enrich-*` in particular, is allowed regardless of the permissions of the current user).
3) Adds integration tests for the above, to verify that Enrich does not leak info protected by DLS and/or FLS.

Co-authored-by: Jay Modi <jay.modi@elastic.co>
2021-01-28 10:17:26 -07:00
Rory Hunter ad1f876daa
Replace NOT operator with explicit `false` check (#67817)
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.

We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
2021-01-26 14:47:09 +00:00
Rory Hunter 1a05a5ac24
Introduce deprecation categories (#67443)
Closes #64824. Introduce the concept of categories to deprecation
logging. Every location where we log a deprecation message must now
include a deprecation category.
2021-01-18 16:16:54 +00:00
David Turner b3e550c289
Make InternalClusterInfoService async (#66993)
This commit reworks the InternalClusterInfoService to run
asynchronously, using timeouts on the stats requests instead of
implementing its own blocking timeouts. It also improves the logging of
failures by identifying the nodes that failed or timed out. Finally it
ensures that only a single refresh is running at once, enqueueing later
refresh requests to run immediately after the current refresh is
finished rather than racing them against each other.
2021-01-05 17:58:30 +00:00
Rene Groeschke eee6e11883
Port all task definitions to task avoidance api (#66738)
This finishes porting all tasks created in gradle build scripts and plugins to use 
the task avoidance api (see #56610)

* Port all task definitions to task avoidance api
* Fix last task created during configuration
* Fix test setup in  :modules:reindex
* Declare proper task inputs
2021-01-04 12:32:19 +01:00
Rene Groeschke 0911d04467
Make AntFixture handling task provider api compliant (#65832)
This tweaks the AntFixture handling to make it compliant with the task avoidance api.
Tasks of type StandaloneRestTestTask are now generally finalised by using the typed ant stop task
which allows us to remove of errorprone dependsOn overrides in StandaloneRestTestTask. As a result
we also ported more task definitions in the build to task avoidance api.

Next work item regarding AntFixture handling is porting AntFixture to a plain Gradle task and remove
Groovy AntBuilder will allow us to port more build logic from Groovy to Java but is out of the scope of
This PR.
2020-12-08 13:07:36 +01:00
Fan Jingbo 74141c17e8
[DOCS] Fix typos (#65951) 2020-12-07 11:39:04 -05:00
Armin Braun 06a31a0aca
Add List Append Utility Method (#65576)
(list -> copy -> add one -> wrap immutable) is a pretty common pattern in CS
updates and tests => added a shortcut for it here and used it in easily identifyable
spots.
2020-12-01 02:47:21 +01:00
Julie Tibshirani 060415715d
Return partial failures if search was cancelled. (#64382)
In #63520, we started cancelling searches that encounter shard failures and
don't allow partial results. In this case we return an 'all shards failed'
response, since there are no successful responses.

This PR proposes to return a 'partial shards failure' instead. The reasoning:
it's misleading to claim that 'all shards failed' when we only know at least
one shard failed (and preemptively cancelled the rest).
2020-11-24 12:54:06 -08:00
Ryan Ernst cfe6e68fde
Add codebases hints for testing (#65281)
This commit adds codebases files for the remaining plugin security
policies that use jar specific grants.
2020-11-20 10:01:51 -08:00
Jay Modi 76c22f751e
HttpsServer can use TLSv1.3 on JDK16+ (#64496)
This commit changes code that previously pinned to TLSv1.2 when running
on JDK 12+ to allow the use of TLSv1.3 if on JDK 16 or newer. There was
a bug in the HttpsServer code that has finally been fixed, which
prevented the use of TLSv1.3 as the HttpsServer would endlessly loop.
The JDK bug is JDK-8254967.

Closes #38646
2020-11-19 10:46:19 -07:00
Rene Groeschke 810e7ff6b0
Move tasks in build scripts to task avoidance api (#64046)
- Some trivial cleanup on build scripts
- Change task referencing in build scripts to use task avoidance api
where replacement is trivial.
2020-11-12 12:04:15 +01:00
Nik Everett a08b52f3bd
Add `runtime_mappings` to search request (#64374)
This adds a way to specify the `runtime_mappings` on a search request
which are always "runtime" fields. It looks like:
```
curl -XDELETE -uelastic:password -HContent-Type:application/json localhost:9200/test
curl -XPOST -uelastic:password -HContent-Type:application/json 'localhost:9200/test/_bulk?pretty&refresh' -d'
{"index": {}}
{"animal": "cat", "sound": "meow"}
{"index": {}}
{"animal": "dog", "sound": "woof"}
{"index": {}}
{"animal": "snake", "sound": "hisssssssssssssssss"}
'

curl -XPOST -uelastic:password -HContent-Type:application/json localhost:9200/test/_search?pretty -d'
{
  "runtime_mappings": {
    "animal.upper": {
      "type": "keyword",
      "script": "for (String s : doc[\"animal.keyword\"]) {emit(s.toUpperCase())}"
    }
  },
  "query": {
    "match": {
      "animal.upper": "DOG"
    }
  }
}'
```

NOTE:
If we have to send a search request with runtime mappings to a node that
doesn't support runtime mappings at all then we'll fail the search
request entirely. The alternative would be to not send those runtime
mappings and let the node fail the search request with an "unknown field"
error. I believe this is would be hard to surprising because you defined
the field in the search request.

NOTE:
It isn't obvious but you can also use `runtime_mappings` to override fields
inside objects by naming the runtime fields with `.` in them. Like this:
```
curl -XDELETE -uelastic:password -HContent-Type:application/json localhost:9200/test
curl -uelastic:password -XPOST -HContent-Type:application/json localhost:9200/test/_bulk?refresh -d'
{"index":{}}
{"name": {"first": "Andrew", "last": "Wiggin"}}
{"index":{}}
{"name": {"first": "Julian", "last": "Delphiki", "suffix": "II"}}
'

curl -uelastic:password -XPOST -HContent-Type:application/json localhost:9200/test/_search?pretty -d'{
  "runtime_mappings": {
    "name.first": {
      "type": "keyword",
      "script": "if (\"Wiggin\".equals(doc[\"name.last.keyword\"].value)) {emit(\"Ender\");} else if (\"Delphiki\".equals(doc[\"name.last.keyword\"].value)) {emit(\"Bean\");}"
    }
  },
  "query": {
    "match": {
      "name.first": "Bean"
    }
  }
}'
```

Relates to #59332
2020-11-10 12:38:59 -05:00