Commit Graph

336 Commits

Author SHA1 Message Date
Mark Vieira 67cfa635c9
Publish plugin-classloader library JAR (#68926) 2021-02-11 12:00:39 -08:00
Przemyslaw Gomulka 71d43b598d
Refactor usage of compatible version (#68648)
Compatible API version is at the moment represented by both Version and
byte - representing a major version. This can lead to a confusion which
representation to use, as well as to incorrect assumptions that minor
versions are supported (with the use of Version.V_7_0_0)

Current usage of XContentParser.useCompatible is also not allowing to
define two compatible implementations. This is not about
support N-2 compatibility, but to allow to continue development when a
major release is performed.

This commit is introducing the CompatibleVersion object responsible for
wrapping around a major version of compatible API.

relates #68100
2021-02-10 10:22:34 +01:00
Rory Hunter 780f273067
Replace NOT operator with explicit `false` check - part 8 (#68625)
Part 8.

We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.

We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
2021-02-08 15:20:34 +00:00
Rory Hunter 509a154954
Replace NOT operator with explicit `false` check - part 7 (#68454)
Part 7.

We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.

We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
2021-02-03 15:20:31 +00:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
Przemyslaw Gomulka 392d924304
Extend NamedObjectRegistry with compatible entries (#68274)
when parsing xContent objects named object can be registered and need to
be parsed as per N-1 rules.
named objects are declared on a parser (PARSER.declareNamedObject) and
the corresponding parsing logic is registered in NamedObjectRegistry.
There is a need to use an old N-1 "configuration" of the registry in
order to support the compatible parsing logic.

This commit extends the NamedObjectRegistry with additional set of
compatible entries, which provide the compatible parsing logic. Those
entries are then in turn used in parseNamedObject when xContentParser is
using compatibility.

relates #51816
2021-02-02 09:16:01 +01:00
Nik Everett 419ce10989
Add grok and dissect methods to runtime fields (#68088)
This adds a `grok` and a `dissect` method to runtime fields which
returns a `Matcher` style object you can use to get the matched
patterns. A fairly simple script to extract the "verb" from an apache
log line with `grok` would look like this:
```
String verb = grok('%{COMMONAPACHELOG}').extract(doc["message"].value)?.verb;
if (verb != null) {
  emit(verb);
}
```

And `dissect` would look like:
```
String verb = dissect('%{clientip} %{ident} %{auth} [%{@timestamp}] "%{verb} %{request} HTTP/%{httpversion}" %{status} %{size}').extract(doc["message"].value)?.verb;
if (verb != null) {
  emit(verb);
}
```

We'll work later to get it down to a clean looking one liner, but for
now, this'll do.

The `grok` and `dissect` methods are special in that they only run at
script compile time. You can't pass non-constants to them. They'll
produce compile errors if you send in a bad pattern. This is nice
because they can be expensive to "compile" and there are many other
optimizations we can make when the patterns are available up front.

Closes #67825
2021-02-01 14:16:01 -05:00
Przemyslaw Gomulka 0c648a2459
Make XContentParser aware of compatible API version (#68113)
Making XContentParser aware of the compatible API version. This is a preparatory work for supporting compatible changes to named xcontent object parsing.
XContentParser#useCompatibleApi will not be used for surgical compatible implementations, it will be only used in the infrastructure to select a compatible namedxcontent registry. Therefore information about the exact major version is not needed.

relates #51816
2021-02-01 19:05:48 +01:00
Rory Hunter ad1f876daa
Replace NOT operator with explicit `false` check (#67817)
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.

We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
2021-01-26 14:47:09 +00:00
bellengao f2ac6da90d
Fix parsing empty value with brackets failed in Dissect ingest processor (#65524) 2021-01-21 08:11:10 -06:00
Przemyslaw Gomulka 8ce39ddc2b
Make ParsedMediaType truly immutable (#67552)
By accident ParsedMediaType was not truly immutable and it was possible
to modify other ParsedMediaType's by accident when building.
ParsedMediaType#parseMediaType(XContentType,Map) was modifying
ParsedMediaTypes within XContentType. This should never happen.

This commit is addressing this by making Map of parameters immutable.

closes #67545
2021-01-18 08:54:18 +01:00
Przemyslaw Gomulka 08b6bd1141
Precompute ParsedMediaType for XContentType (#67409)
XContentType instances should have a precomputed ParsedMediaType instances
that will prevent unnecessary parsing of its media type.

The unnecessary parsing was causing a performance drop and this commit addresses
that. The performance was affected because XContentBuilder is newly created per 
each request, which in case of bulk request can be many instances. 
Each XContentBuilder instance is using XContentType.mediaType which was parsed
into ParsedMediaType. That in bulk request is unnecessary as the content-type 
is already known - from XContent#type

By default when Accept or Content-Type headers are not specified, 
application/json; charset=UTF-8 is used on a response content-type. 
The redundant charset parameter is dropped in this commit as well.
2021-01-14 14:01:02 +01:00
Przemyslaw Gomulka 5e74f79e22
Support response content-type with versioned media type (#65500)
This commit allows returning a correct requested response content-type - it did not work for versioned media types.
It is done by adding new vendor specific instances to XContent and TextFormat enums. These instances can then "format" the response content type string when provided with parameters. This is similar to what SQL plugin does with its media types.

#51816
2021-01-05 09:23:22 +01:00
Rene Groeschke eee6e11883
Port all task definitions to task avoidance api (#66738)
This finishes porting all tasks created in gradle build scripts and plugins to use 
the task avoidance api (see #56610)

* Port all task definitions to task avoidance api
* Fix last task created during configuration
* Fix test setup in  :modules:reindex
* Declare proper task inputs
2021-01-04 12:32:19 +01:00
Ioannis Kakavas bd873698bc
Ensure CI is run in FIPS 140 approved only mode (#64024)
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.

This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.

It also addresses a number of tests that would fail in approved only mode
Mainly:

    Tests that use PBKDF2 with a password less than 112 bits (14char). We
    elected to change the passwords used everywhere to be at least 14
    characters long instead of mandating
    the use of pbkdf2_stretch because both pbkdf2 and
    pbkdf2_stretch are supported and allowed in fips mode and it makes sense
    to test with both. We could possibly figure out the password algorithm used
    for each test and adjust password length accordingly only for pbkdf2 but
    there is little value in that. It's good practice to use strong passwords so if
    our docs and tests use longer passwords, then it's for the best. The approach
    is brittle as there is no guarantee that the next test that will be added won't
    use a short password, so we add some testing documentation too.
    This leaves us with a possible coverage gap since we do support passwords
    as short as 6 characters but we only test with > 14 chars but the
    validation itself was not tested even before. Tests can be added in a followup,
    outside of fips related context.

    Tests that use a PKCS12 keystore and were not already muted.

    Tests that depend on running test clusters with a basic license or
    using the OSS distribution as FIPS 140 support is not available in
    neither of these.

Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
2020-12-23 21:00:49 +02:00
Ioannis Kakavas abaf81e37e
Allow ESRestTestClient to trust certificates (#66559)
ESRestTestCase rest clients could only be configured to trust
the certificate authorities that were contained in a truststore. In
certain cases (like in fips mode where JKS/PKCS12 keystores) cannot
be used, it's beneficial to be able to trust specific certificate
authorities (indicated by the CA PEM endoded certificate)
2020-12-23 17:29:43 +02:00
Rene Groeschke defaa93902
Avoid tasks materialized during configuration phase (#65922)
* Avoid tasks materialized during configuration phase
* Fix RestTestFromSnippet testRoot setup
2020-12-12 16:14:17 +01:00
Hendrik Muhs 9b47889153
[Transform] use ISO dates in output instead of epoch millis (#65584)
Transform writes dates as epoch millis, this does not work for historic data in some cases or is
unsupported. Dates should be written as such. With this PR transform starts writing dates in ISO
format, but as existing transform might rely on the format it provides backwards compatibility for
old jobs as well as a setting to write dates as epoch millis.

fixes #63787
2020-12-07 15:34:28 +01:00
Przemyslaw Gomulka 91df1b8edf
Introduce Compatible Version plugin (#64481)
A RestCompatibilityPlugin and its xpack implementation allow to calculate a version requested on REST request. It uses accept, content-type headers to return a version.
It also performs a validation of allowed combinations of versions and values provided on accept/content-type headers

relates #51816
2020-11-23 14:19:23 +01:00
Przemyslaw Gomulka 618d8bcec6
Allow registering compatible handlers (#64423)
Adding an infrastructure to allow for registration of Compatible Handlers.
Compatible handlers are RestHandlers used for handling rest request from old version clients ( CURRENT-1 version). They might be registered under an endpoint that was removed or changed in CURRENT version (different path, method or an endpoint completely removed).
But they also can be registered under the same endpoint (same path, method as the RestHandler in CURRENT)
RestHandler's endpoint is at the moment 2dimensional - a method and a path.

This PR adds a 3rd dimension - a version.

Registration:
RestHandler declares a new compatibleWithVersion method, which will be overridden by Compatible Handlers and returning a Version.CURRENT -1. By default the method returns Version.CURRENT
compatibleWithVersion is used when iterating over handlers within RestController#registerHandler. The returned value is used to set a version on MethodHandlers

Lookup:
An interface CompatibleVersion is introduced in order to abstract a logic to calculate a compatible version requested by a user.
It is not implemented in this PR. A simplified, always returning Version.CURRENT implementation is used.
Within RestController, a version is calculated with the use of CompatibleVersion, then the lookup for MethodHandlers is performed (the logic is the same)
Once it is find, an additional lookup for a RestHandler for requested version is made.

The requested version has to be also passed down to XContentBuilder in order to allow for per version serialisation logic

relates #51816
2020-11-16 09:11:24 +01:00
Rene Groeschke 810e7ff6b0
Move tasks in build scripts to task avoidance api (#64046)
- Some trivial cleanup on build scripts
- Change task referencing in build scripts to use task avoidance api
where replacement is trivial.
2020-11-12 12:04:15 +01:00
Przemyslaw Gomulka 0bb64cbe1e
Handle incorrect header values (#64708)
When Accept or Content-Type header values are incorrect, a request
should be gracefully rejected and an exception message returned to a
client.

relates #64689
relates #51816
2020-11-10 09:07:32 +01:00
Przemyslaw Gomulka dd5bcd4093
Ignore media ranges when parsing (#64721)
Browsers are sending media ranges with quality factors on Accept header.
We should ignore the value and respond with applicaiton/json

closes #64689
relates #51816
2020-11-09 09:03:18 +01:00
Przemyslaw Gomulka 710500a50a
Do not allow spaces within MediaType's parameters (#64650)
Per https://tools.ietf.org/html/rfc7231#section-3.1.1.1 MediaType can
have spaces around parameters pair, but do not allow spaces within
parameter. That means that parameter pair forbids spaces around =

follow up after
#64406 (comment)
relates #51816
2020-11-06 09:25:21 +01:00
Przemyslaw Gomulka c219e176e9
Introduce per REST endpoint media types (#64406)
Per REST endpoint media types declaration allows to make parsing/validation more strict.

If a media type was declared only in one endpoint (for instance CSV in SQL endpoint) it should not be allowed to parse that media type when using other endpoints.
However, the Compatible API need to be able to understand all media types supported by Elasticsearch in order to parse a compatible-with=version parameter.
This implies that endpoints need to declare which media type they support and how to parse them (if introducing new media types - like SQL).

How to parse:
MediaType interface still serves as an abstraction on top of XContentType and TextFormat. It also has a declaration of mappings String-MediaType with parameters. Parameters declares the names of parameters and regex to validate its values.
This instructs how to perform the parsing. For instance - XContentType.JSON has the mapping of application/vnd.elasticsearch+json -> JSON and allows parameters compatible-with=\d and charset=utf-8

MediaTypeParser was simplified into ParsedMediaType class with static factory method for parsing.

How to declare:
RestHandler interface is extended with a validAcceptMediaTypes which returns a MediaTypeRegistry - a class that encapsulates mappings of string (type/subtype) to MediaType, allowed parameters and formatPathParameter values.
We only need to allow of declaration of valid media types for Accept header. Content-Type valid media types are fixed to XContentType instances - json, yaml, smile, cbor.

relates #51816
2020-11-05 09:47:13 +01:00
Tanguy Leroux 34fc3033c8
Add CacheFile#fsync() method to ensure cached data are written on disk (#64201)
This commit adds a new CacheFile.fsync() method to fsync the cache 
file on disk. This method uses the utility method IOUtils.fsync(Path, 
boolean) that executes a FileChannel.force() call under the hood.
2020-11-03 15:08:54 +01:00
Armin Braun eaac6496fc
Fix #invariant Assertion in CacheFile (#64180)
Fix #invariant Assertion in CacheFile

closes #64141
2020-10-27 17:59:17 +01:00
Lee Hinman 7620e9415c
Add assert that raw and readable xcontent field names are different (#63332)
This adds asserts that will catch the case where we accidentally provide the same raw and readable
field name in xcontent.
2020-10-06 10:52:49 -06:00
Przemyslaw Gomulka d04edcdfe7
Allow parsing Content-Type and Accept headers with version (#61427)
Content-Type and Accept headers can be populated with a versioned form of media types
like application/vnd.elasticsearch+json;compatible-with=7
when previously it was simple application/json or (cbor, yaml..) - this is still supported.
Extending MediaTypeParser to validate the parameters.

relates  #51816
2020-10-05 15:12:48 +02:00
Nik Everett 2e346f3fae
Grok: "native" results (#62843)
This adds the ability to fetch java primitives like `long` and `float`
from grok matches rather than their boxed versions. It also allows
customizing the which fields are extracted and how they are extracted.
By default we continue to fetch a `Map<String, Object>` but runtime
fields will be able to catch *just* the fields it is interested
in, and the values will be primitives.
2020-09-24 10:46:38 -04:00
Nik Everett 59eac2262e
Grok: Handle utf-8 natively (#62794)
This adds a method to `Grok` that matches against sections offset from
utf-8 byte arrays:
```
Map<String, Object> captures(byte[] utf8Bytes, int offset, int length)
```

This'll be useful for the grok-flavored runtime fields because they
want to match against utf-8 encoded strings stored in a big array. And
joni already supports this.
2020-09-23 08:42:43 -04:00
Nik Everett d4e014c4c2
Extract capture config from grok patterns up front (#62706)
This extracts the configuration for extracting values from a groked
string when building the grok expression to do two things:
1. Create a method exposing that configuration on `Grok` itself which
   will be used grok `grok` flavored runtime fields.
2. Marginally speed up extracting grok values by skipping a little
   string manipulation.
2020-09-22 13:55:43 -04:00
Nik Everett 152ee3aca4
Raname grok's built-in patterns (#62735)
This reworks the code around grok's built-in patterns to name things
more like the rest of the code. Its not a big deal, but I'm just more
used to having `public static final` constants in SHOUTING_SNAKE_CASE.
2020-09-22 09:31:20 -04:00
Tim Vernum 235521ccaf
Include trusted issuer details in SSL diagnostics (#61702)
This commit changes the SSL Diagnostic warning to include additional
details about trusted certificate issuers when the provide certificate
chain does not match any trust anchors.

- If there are no trusted issuers, this is explicitly called out
- If there is one trusted issuer, it is listed by name (DN) and fingerprint
- If there are between 2 and 10 trusted issuers, then they are listed
  by name (DN)
- If there are more than 10 trusted issuers, the number of issuers is
  included in the message (but no other details).
2020-09-18 15:16:37 +10:00
Przemyslaw Gomulka 86ba7324c8
Media-type parser (#61987)
Splitting method XContentType.fromMediaTypeOrFormat into two separate methods. This will help to validate media type provided in Accept or Content-Type headers.
Extract parsing logic from XContentType (fromMediaType and fromFormat methods) to a separate MediaTypeParser class. This will help reuse the same parsing logic for XContentType and TextFormat (used in sql)

`Media-Types type/subtype; parameters` parsing is in defined https://tools.ietf.org/html/rfc7231#section-3.1.1.1

part of  #61427
2020-09-17 16:47:32 +02:00
Dan Hermann 9dcab76427
Preserve grok pattern ordering and add sort option (#61671) 2020-09-08 07:10:27 -05:00
Armin Braun 8fc9581d35
Speed up XContent Collection Parsing (#61442)
1. Get rid of the capturing lambda on the hot path that inlines very badly
2. Remove as many bounds checks as possible, thereby reducing method size and improving inlining
2020-08-27 09:24:49 +02:00
Armin Braun 305bebfcad
Stop Needlessly Copying Bytes in XContent Parsing (#61447)
Wrapping a `BytesArray` in a `StreamInput` for deserialization is inefficient.
This forces Jackson to internally buffer (i.e. copy) all bytes from the `BytesArray`
before deserializing, adding overhead for copying the bytes and managing the buffers.

This commit fixes a number of spots where `BytesArray` is the most common type of
`BytesReference` to special case this type and parse it more efficiently.
Also improves parsing `String`s to use the more efficient direct `String` parsing APIs.
2020-08-24 15:03:21 +02:00
Armin Braun 82f040716d
Fix Broken Stream Close in writeRawValue (#60625)
Small oversight in #56078 that only showed up during backporting where a stream copy was turned from a non-closing to a closing one. Enhanced part of a test in this PR to make it show up in master also even though we practically never use this method with stream targets that actually close.
2020-08-04 12:46:15 +02:00
Armin Braun e28dbde289
Unify Stream Copy Buffer Usage (#56078)
We have various ways of copying between two streams and handling thread-local
buffers throughout the codebase. This commit unifies a number of them and
removes buffer allocations in many spots.
2020-08-03 18:21:24 +02:00
Nhat Nguyen f238d0a367
Set timeout of master requests on follower to unbounded (#60070)
Today, a follow task will fail if the master node of the follower 
cluster is temporarily overloaded and unable to process master node
requests (such as update mapping, setting, or alias) from a follow-task
within the default timeout. This error is transient, and follow-tasks
should not abort. We can avoid this problem by setting the timeout of
master node requests on the follower cluster to unbounded.

Closes #56891
2020-07-27 09:23:30 -04:00
Yannick Welsch 2078509af5
Set specific keepalive options by default on supported platforms (#59278)
keepalives tell any intermediate devices that the connection remains alive, which helps with overzealous firewalls that are
killing idle connections. keepalives are enabled by default in Elasticsearch, but use system defaults for their
configuration, which often times do not have reasonable defaults (e.g. 7200s for TCP_KEEP_IDLE) in the context of
distributed systems such as Elasticsearch.

This PR sets the socket-level keep_alive options for network.tcp.{keep_idle,keep_interval} to 5 minutes on configurations
that support it (>= Java 11 & (MacOS || Linux)) and where the system defaults are set to something higher than 5
minutes. This helps keep the connections alive while not interfering with system defaults or user-specified settings
unless they are deemed to be set too high by providing better out-of-the-box defaults.
2020-07-27 13:38:24 +02:00
Armin Braun 785a91c1e3
Optimize some Spots around Closing Resources (#60049)
The single element `close` calls go through a very inefficient path that includes creating
a one element list.
`releaseOnce` is only with a single non-null input in production in two spots so no need for
varargs and any complexity here.
`ReleasableBytesStreamOutput` does not require any `releaseOnce` wrapping because we already have
that kind of logic implemented in `org.elasticsearch.common.util.AbstractArray` (which we were
wrapping here) already.
2020-07-22 23:41:48 +02:00
Ryan Ernst 5d6a2dd35b
Add license feature usage api (#59342)
This commit adds a new api to track when gold+ features are used within
x-pack. The tracking is done internally whenever a feature is checked
against the current license. The output of the api is a list of each
used feature, which includes the name, license level, and last time it
was used. In addition to a unit test for the tracking, a rest test is
added which ensures starting up a default configured node does not
result in any features registering as used.

There are a couple features which currently do not work well with the
tracking, as they are checked in a manner that makes them look always
used. Those features will be fixed in followups, and in this PR they are
omitted from the feature usage output.
2020-07-14 12:25:36 -07:00
Yang Wang a28ce1e21c
Improve role cache efficiency for API key roles (#58156)
This PR ensure that same roles are cached only once even when they are from different API keys.
API key role descriptors and limited role descriptors are now saved in Authentication#metadata 
as raw bytes instead of deserialised Map<String, Object>. 
Hashes of these bytes are used as keys for API key roles. Only when the required role is not found 
in the cache, they will be deserialised to build the RoleDescriptors. The deserialisation is directly 
from raw bytes to RoleDescriptors without going through the current detour of 
"bytes -> Map -> bytes -> RoleDescriptors".
2020-07-13 21:23:23 +10:00
Przemysław Witek d39bde9c95
Simplify parser declarations when specialist types are stored in strings (#58996) 2020-07-06 12:15:42 +02:00
Rene Groeschke a1e1b33a47
Detangle JdkJarHellCheck from build tool building (#58601)
* Detangle JdkJarHellCheck from build tool building

- allows building the tool with same runtime as es
- allows building build tools with newer runtime version and keep ThirdPartyAuditTask
running with minimum runtime to ensure we check against correct jre
- add jdkjarhell test jar setup into fixture
2020-06-30 16:20:02 +02:00
Rene Groeschke 9526c7a4b3
Replace compile configuration usage with api (#58451)
- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build
2020-06-30 09:37:09 +02:00
Ryan Ernst 2671ff7572
Migrate plugin packaging tests to java (#58518)
This commit converts the bats tests for the plugin cli into the java
packaging test framework. The new tests only use the example plugin to
test the plugin cli. The tests for each individual plugin's contents
after being installed are handled by a new unit test for the plugin
installer added in #58287.
2020-06-25 13:38:40 -07:00
Ryan Ernst 7c839eefe8
Create utility for custom config setup in packaging tests (#58352)
This commit creates a shared withCustomConfig method that may be used by
any packaging test. The method will copy the config directory and
override the conf path appropriately depending on the distribution type.
2020-06-23 14:49:20 -07:00
Rene Groeschke 5f9d1f1d7c
Unify dependency licenses task configuration (#58116)
- Remove duplicate dependency configuration
- Use task avoidance api accross the build
- Remove redundant licensesCheck config
2020-06-17 18:27:16 +02:00
Rene Groeschke 680ea07f7f
Remove deprecated usage of testCompile configuration (#57921)
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
2020-06-12 13:34:53 +02:00
Jake Landis f5910664b7
Ensure Joni warning are logged at debug (#57302)
When Joni, the regex engine that powers grok emits a warning it
does so by default to System.err. System.err logs are all bucketed
together in the server log at WARN level. When Joni emits a warning,
it can be extremely verbose, logging a message for each execution
again that pattern. For ingest node that means for every document
that is run that through Grok. Fortunately, Joni provides a call
back hook to push these warnings to a custom location.

This commit implements Joni's callback hook to push the Joni warning
to the Elasticsearch server logger (logger.org.elasticsearch.ingest.common.GrokProcessor)
at debug level. Generally these warning indicate a possible issue with
the regular expression and upon creation of the Grok processor will
do a "test run" of the expression and log the result (if any) at WARN 
level. This WARN level log should only occur on pipeline creation which 
is a much lower frequency then every document. 

Additionally, the documentation is updated with instructions for how
to set the logger to debug level.
2020-06-09 13:33:27 -05:00
Mark Vieira 627ef279fd
Include vendored code notices in distribution notice files (#57017) 2020-06-01 15:23:41 -07:00
Christoph Büscher 1211699473
Fix `bool` query behaviour on null value (#56817)
Until 7.7 we used to ignore `null` values for `bool`queries `minimum_should_match`,
parameters and also for the `must`,  `must_not`, `should` and `filter` clauses.
An internal refactoring has changed this so now we get a parsing error. While `null` 
should not a common value here, we should restore the old behaviour for bwc for now.

Closes #56812
2020-05-26 16:23:02 +02:00
Jason Tedor 689244e08e
Fix some licenses in our own code (#56978)
All of these files were written by us, and not sourced from
anywhere. Therefore, the license head should be granting licenses to
Elasticsearch, rathern than to the ASF. This commit address them by
changing the license to our standard Apache 2.0 license header.
2020-05-20 09:23:48 -04:00
Ioannis Kakavas cb4ebfc042
Mutes failing test in FIPS 140 mode (#56807)
SSLReloadDuringStartupIntegTests was recently introduced but it is
failing in FIPS mode because of the use of JKS keystore. This
change mutes it in FIPS mode. It also adjusts
PemUtilsTests#readEcCurveTests to be more robust in general and
also work in FIPS mode.
2020-05-18 09:56:42 +03:00
Ryan Ernst c0ee68b0a0
Move publishing configuration to a separate plugin (#56727)
This is another part of the breakup of the massive BuildPlugin. This PR
moves the code for configuring publications to a separate plugin. Most
of the time these publications are jar files, but this also supports the
zip publication we have for integ tests.
2020-05-14 18:56:59 -07:00
Jason Tedor f45b8dd88b
Upgrade to Jackson 2.10.4 (#56188)
Another Jackson release is available. There are some CVEs addressed,
none of which impact us, but since we can now bump Jackson easily, let
us move along with the train to avoid the false positives from security
scanners.
2020-05-06 17:19:22 -04:00
Tal Levy f26ee1298d
Add geo_shape support for geotile_grid and geohash_grid (#55966)
this commit adds aggregation support for the geo_shape field
type on geo*_grid aggregations.

it introduces a Tiler for both tiles and hashes that enables a new type of
ValuesSource to replace the GeoPoint's CellIdSource. This makes it possible
for the existing Aggregator to be re-used, so no new implementations of
the grid aggregators are added.
2020-05-05 08:00:16 -07:00
Hendrik Muhs 33f134f556
[Transform] add throttling (#56007)
add throttling to transform, throttling will slow down search requests by delaying the execution based on a documents per second metric. 

fixes #54862
2020-05-05 11:34:54 +02:00
Jake Landis 839ac4dd6a
Prevent stack overflow for numerous grok patterns. (#55899)
This was noticed for a pipeline that was defining hundreds of
grok patterns inline with a single grok processor.

The recursive call used to translate a Grok pattern to a regular
expression can overflow the stack. This commit converts that method 
to an iterative method. 

Co-authored-by: Przemko Robakowski <probakowski@users.noreply.github.com>
2020-04-30 19:29:18 -05:00
Ioannis Kakavas fc90eef734
Fix PemKeyConfigTests (#55577)
We were creating PemKeyConfig objects using different private
keys but always using testnode.crt certificate that uses the
RSA public key. The PemKeyConfig was built but we would
then later fail to handle SSL connections during the TLS
handshake eitherway.
This became obvious in FIPS tests where the consistency
checks that FIPS 140 mandates kick in and failed early
becausethe private key was of different type than the
public key
2020-04-30 09:33:08 +03:00
Igor Motov 38bc1f9222
Add InstantiatingObjectParser (#55483)
Introduces InstantiatingObjectParser which is similar to the
ConstructingObjectParser, but instantiates the object using its constructor
instead of a builder function.

Closes #52499
2020-04-22 11:35:17 -04:00
Tim Vernum d68dcd5981
Fix use of password protected PKCS#8 keys for SSL (#55457)
PEMUtils would incorrectly fill the encryption password with zeros
(the '\0' character) after decrypting a PKCS#8 key.

Since PEMUtils did not take ownership of this password it should not
zero it out because it does not know whether the caller will use that
password array again. This is actually what PEMKeyConfig does - it
uses the key encryption password as the password for the ephemeral
keystore that it creates in order to build a KeyManager.
2020-04-22 13:15:02 +10:00
William Brafford 38cd668ad0
Remove deprecated third-party methods from tests (#55255)
I've noticed that a lot of our tests are using deprecated static methods
from the Hamcrest matchers. While this is not a big deal in any
objective sense, it seems like a small good thing to reduce compilation
warnings and be ready for a new release of the matcher library if we
need to upgrade. I've also switched a few other methods in tests that
have drop-in replacements.
2020-04-15 16:31:51 -04:00
Ryan Ernst 842ce32870
Use task avoidance with forbidden apis (#55034)
Currently forbidden apis accounts for 800+ tasks in the build. These
tasks are aggressively created by the plugin. In forbidden apis 3.0, we
will get task avoidance
(https://github.com/policeman-tools/forbidden-apis/pull/162), but we
need to ourselves use the same task avoidance mechanisms to not trigger
these task creations. This commit does that for our foribdden apis
usages, in preparation for upgrading to 3.0 when it is released.
2020-04-15 13:23:55 -07:00
Ioannis Kakavas 1cec91ec86
Add support for more named curves (#55179)
We implicitly only supported the prime256v1 ( aka secp256r1 )
curve for the EC keys we read as PEM files to be used in any
SSL Context. We would not fail when trying to read a key
pair using a different curve but we would silently assume
that it was using `secp256r1` which would lead to strange
TLS handshake issues if the curve was actually another one.

This commit fixes that behavior in that it
supports parsing EC keys that use any of the named curves
defined in rfc5915 and rfc5480 making no assumptions about
whether the security provider in use supports them (JDK8 and
higher support all the curves defined in rfc5480).
2020-04-15 10:39:14 +03:00
Mark Vieira 0e55fdeae9
Re-add origin url information to publish POM files (#55171) 2020-04-14 11:48:36 -07:00
Gordon Brown e9bc3e8234
Disallow negative TimeValues (#53913)
This commit causes negative TimeValues, other than -1 which is sometimes used as
a sentinel value, to be rejected during parsing.

Also introduces a hack to allow ILM to load policies which were written to the
cluster state with a negative min_age, treating those values as 0, which should
match the behavior of prior versions.
2020-03-26 09:22:07 -06:00
Alan Woodward 561222badc
Report parser name and location in XContent deprecation warnings (#53805)
It's simple to deprecate a field used in an ObjectParser just by adding deprecation
markers to the relevant ParseField objects. The warnings themselves don't currently
have any context - they simply say that a deprecated field has been used, but not
where in the input xcontent it appears. This commit adds the parent object parser
name and XContentLocation to these deprecation messages.

Note that the context is automatically stripped from warning messages when they
are asserted on by integration tests and REST tests, because randomization of
xcontent type during these tests means that the XContentLocation is not constant
* Strip xcontentloc in yaml tests

* Handle negative pos values
2020-03-20 11:40:27 +00:00
Alan Woodward c6cdd3a4c2 Revert "Report parser name and location in XContent deprecation warnings (#53752)"
This reverts commit 7636930ceb.

There is some randomization in the YAML test suite which means we can't check
for exact xcontentlocation in the deprecation warning headers.
2020-03-19 12:29:42 +00:00
Alan Woodward 7636930ceb
Report parser name and location in XContent deprecation warnings (#53752)
It's simple to deprecate a field used in an ObjectParser just by adding deprecation
markers to the relevant ParseField objects. However, the warnings themselves don't 
currently have any context; they simply say that a deprecated field has been used, 
but not where in the input it appears. This commit adds the parent object parser
name and XContentLocation to these deprecation messages.
2020-03-19 11:26:04 +00:00
Dominic Page d1cbdfb753
Geo shape query vs geo point (#52382)
Enable geo_shape query to work on geo_point fields for shapes: circle, polygon, multipolygon, rectangle

see: #48928

Co-Authored-By:  @iverase
2020-03-18 17:03:52 +01:00
Alan Woodward e23311ce51
Make it possible to deprecate all variants of a ParseField with no replacement (#53722)
Sometimes we want to deprecate and remove a ParseField entirely, without replacement;
for example, the various places where we specify a _type field in 7x. Currently we can
tell users only that a particular field name should not be used, and that another name should
be used in its place. This commit adds the ability to say that a field should not be used at
all.
2020-03-18 14:15:30 +00:00
Ryan Ernst d63cda1bcb
Upgrade jackson to 2.10.3 and GeoIP to 2.13.1 (#53642)
Re-applies the change from #53523 along with test fixes.

closes #53626
closes #53624
closes #53622
closes #53625

Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Jake Landis <jake.landis@elastic.co>
2020-03-17 10:26:35 -07:00
Mark Vieira 060b4eed59
Revert "Upgrade to Jackson 2.10.3 and GeoIP2 to 2.13.1 (#53523)"
This reverts commit 7bc75f48

Signed-off-by: Mark Vieira <portugee@gmail.com>
2020-03-15 18:10:14 -07:00
Jason Tedor 7bc75f48d8
Upgrade to Jackson 2.10.3 and GeoIP2 to 2.13.1 (#53523)
This commit upgrades our Jackson dependency to 2.10.3 and our GeoIP2
dependency to 2.13.1.
2020-03-14 10:22:29 -04:00
David Kyle 32a032dbad
Fix parseNamedObject leaving end object token (#53400)
Fixes a bug in #53017 where ObjectParser.parseNamedObject would leave the end object 
token unconsumed meaning subsequent fields would not be parsed.
2020-03-12 11:43:12 +00:00
David Kyle 12613fa7b1
Add ObjectParser.declareNamedObject (singular) method (#53017)
Add the convenience method AbstractObjectParser.declareNamedObject (singular) to 
complement the existing declareNamedObjects (plural).
2020-03-11 10:18:46 +00:00
Alan Woodward fbd8d797f3
Allow specifying an exclusive set of fields on ObjectParser (#52893)
ObjectParser allows you to declare a set of required fields, such that at least one
of the set must appear in an xcontent object for it to be valid. This commit adds
the similar concept of a set of exclusive fields, such that at most one of the set
must be present. It also enables required fields on ConstructingObjectParser, and
re-implements PercolateQueryBuilder.fromXContent() to use object parsing as
an example of how this works.
2020-03-03 09:50:32 +00:00
Nik Everett 75d83db8c7
HLRC support for string_stats (#52163)
This adds a builder and parsed results for the `string_stats`
aggregation directly to the high level rest client. Without this the
HLRC can't access the `string_stats` API without the elastic licensed
`analytics` module.

While I'm in there this adds a few of our usual unit tests and
modernizes the parsing.
2020-02-12 18:01:14 -05:00
Zachary Tong 208cbc224c
Allow ObjectParsers to specify required sets of fields (#49661)
ConstructingObjectParser can be used to specify required fields,
but it is still difficult to configure "sets" of fields where only
one of the set is required (requiring hand-rolled logic in each
ConstructingObjectParser, or adding special validation methods
to objects that are called after building the object).

This commit adds a new method on ObjectParser which allows
the parsers to register required sets.  E.g. ["foo", "bar"] can be
registered, which means "foo", "bar" or both must be configured
by the user otherwise an exception is thrown.

This pattern crops up in many places in our parsers; a good example are
the aggregation "field" and "script" fields.  One or both must be 
configured on all aggregations, omitting both should result in an exception.  
This was previously handled far downstream resulting in an aggregation
exception, when it should be a parse exception.
2020-02-11 12:47:03 -05:00
Gautam eb097700cf
Missing suffix for German Month "Juli" in Grok Pattern MONTH (#51579) (#51591) 2020-02-03 14:53:30 -06:00
Ryan Ernst bf317e8c4e
Remove comparison to true for booleans (#51723)
While we use `== false` as a more visible form of boolean negation
(instead of `!`), the true case is implied and the true value does not
need to explicitly checked. This commit converts cases that have slipped
into the code checking for `== true`.
2020-01-31 16:34:27 -08:00
William Brafford c117c0cf0a
Password-protected Keystore Feature Branch PR (#51123)
* Reload secure settings with password (#43197)

If a password is not set, we assume an empty string to be
compatible with previous behavior.
Only allow the reload to be broadcast to other nodes if TLS is
enabled for the transport layer.

* Add passphrase support to elasticsearch-keystore (#38498)

This change adds support for keystore passphrases to all subcommands
of the elasticsearch-keystore cli tool and adds a subcommand for
changing the passphrase of an existing keystore.
The work to read the passphrase in Elasticsearch when
loading, which will be addressed in a different PR.

Subcommands of elasticsearch-keystore can handle (open and create)
passphrase protected keystores

When reading a keystore, a user is only prompted for a passphrase
only if the keystore is passphrase protected.

When creating a keystore, a user is allowed (default behavior) to create one with an
empty passphrase

Passphrase can be set to be empty when changing/setting it for an
existing keystore

Relates to: #32691
Supersedes: #37472

* Restore behavior for force parameter (#44847)

Turns out that the behavior of `-f` for the add and add-file sub
commands where it would also forcibly create the keystore if it
didn't exist, was by design - although undocumented.
This change restores that behavior auto-creating a keystore that
is not password protected if the force flag is used. The force
OptionSpec is moved to the BaseKeyStoreCommand as we will presumably
want to maintain the same behavior in any other command that takes
a force option.

*  Handle pwd protected keystores in all CLI tools  (#45289)

This change ensures that `elasticsearch-setup-passwords` and
`elasticsearch-saml-metadata` can handle a password protected
elasticsearch.keystore.
For setup passwords the user would be prompted to add the
elasticsearch keystore password upon running the tool. There is no
option to pass the password as a parameter as we assume the user is
present in order to enter the desired passwords for the built-in
users.
For saml-metadata, we prompt for the keystore password at all times
even though we'd only need to read something from the keystore when
there is a signing or encryption configuration.

* Modify docs for setup passwords and saml metadata cli (#45797)

Adds a sentence in the documentation of `elasticsearch-setup-passwords`
and `elasticsearch-saml-metadata` to describe that users would be
prompted for the keystore's password when running these CLI tools,
when the keystore is password protected.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

* Elasticsearch keystore passphrase for startup scripts (#44775)

This commit allows a user to provide a keystore password on Elasticsearch
startup, but only prompts when the keystore exists and is encrypted.

The entrypoint in Java code is standard input. When the Bootstrap class is
checking for secure keystore settings, it checks whether or not the keystore
is encrypted. If so, we read one line from standard input and use this as the
password. For simplicity's sake, we allow a maximum passphrase length of 128
characters. (This is an arbitrary limit and could be increased or eliminated.
It is also enforced in the keystore tools, so that a user can't create a
password that's too long to enter at startup.)

In order to provide a password on standard input, we have to account for four
different ways of starting Elasticsearch: the bash startup script, the Windows
batch startup script, systemd startup, and docker startup. We use wrapper
scripts to reduce systemd and docker to the bash case: in both cases, a
wrapper script can read a passphrase from the filesystem and pass it to the
bash script.

In order to simplify testing the need for a passphrase, I have added a
has-passwd command to the keystore tool. This command can run silently, and
exit with status 0 when the keystore has a password. It exits with status 1 if
the keystore doesn't exist or exists and is unencrypted.

A good deal of the code-change in this commit has to do with refactoring
packaging tests to cleanly use the same tests for both the "archive" and the
"package" cases. This required not only moving tests around, but also adding
some convenience methods for an abstraction layer over distribution-specific
commands.

* Adjust docs for password protected keystore (#45054)

This commit adds relevant parts in the elasticsearch-keystore
sub-commands reference docs and in the reload secure settings API
doc.

* Fix failing Keystore Passphrase test for feature branch (#50154)

One problem with the passphrase-from-file tests, as written, is that
they would leave a SystemD environment variable set when they failed,
and this setting would cause elasticsearch startup to fail for other
tests as well. By using a try-finally, I hope that these tests will fail
more gracefully.

It appears that our Fedora and Ubuntu environments may be configured to
store journald information under /var rather than under /run, so that it
will persist between boots. Our destructive tests that read from the
journal need to account for this in order to avoid trying to limit the
output we check in tests.

* Run keystore management tests on docker distros (#50610)

* Add Docker handling to PackagingTestCase

Keystore tests need to be able to run in the Docker case. We can do this
by using a DockerShell instead of a plain Shell when Docker is running.

* Improve ES startup check for docker

Previously we were checking truncated output for the packaged JDK as
an indication that Elasticsearch had started. With new preliminary
password checks, we might get a false positive from ES keystore
commands, so we have to check specifically that the Elasticsearch
class from the Bootstrap package is what's running.

* Test password-protected keystore with Docker (#50803)

This commit adds two tests for the case where we mount a
password-protected keystore into a Docker container and provide a
password via a Docker environment variable.

We also fix a logging bug where we were logging the identifier for an
array of strings rather than the contents of that array.

* Add documentation for keystore startup prompting (#50821)

When a keystore is password-protected, Elasticsearch will prompt at
startup. This commit adds documentation for this prompt for the archive,
systemd, and Docker cases.

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

* Warn when unable to upgrade keystore on debian (#51011)

For Red Hat RPM upgrades, we warn if we can't upgrade the keystore. This
commit brings the same logic to the code for Debian packages. See the
posttrans file for gets executed for RPMs.

* Restore handling of string input

Adds tests that were mistakenly removed. One of these tests proved
we were not handling the the stdin (-x) option correctly when no
input was added. This commit restores the original approach of
reading stdin one char at a time until there is no more (-1, \r, \n)
instead of using readline() that might return null

* Apply spotless reformatting

* Use '--since' flag to get recent journal messages

When we get Elasticsearch logs from journald, we want to fetch only log
messages from the last run. There are two reasons for this. First, if
there are many logs, we might get a string that's too large for our
utility methods. Second, when we're looking for a specific message or
error, we almost certainly want to look only at messages from the last
execution.

Previously, we've been trying to do this by clearing out the physical
files under the journald process. But there seems to be some contention
over these directories: if journald writes a log file in between when
our deletion command deletes the file and when it deletes the log
directory, the deletion will fail.

It seems to me that we might be able to use journald's "--since" flag to
retrieve only log messages from the last run, and that this might be
less likely to fail due to race conditions in file deletion.

Unfortunately, it looks as if the "--since" flag has a granularity of
one-second. I've added a two-second sleep to make sure that there's a
sufficient gap between the test that will read from journald and the
test before it.

* Use new journald wrapper pattern

* Update version added in secure settings request

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
2020-01-27 19:51:39 -05:00
Nik Everett 224640a3ca
"did you mean" for ObjectParser with top named (#51018)
When you declare an ObjectParser with top level named objects like we do
with `significant_terms` we didn't support "did you mean". This fixes
that.

Relates #50938
2020-01-17 10:41:14 -05:00
Nik Everett 5da5f44de4
Add "did you mean" to ObjectParser (#50938)
Check it out:
```
$ curl -u elastic:password -HContent-Type:application/json -XPOST localhost:9200/test/_update/foo?pretty -d'{
  "dac": {}
}'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "x_content_parse_exception",
        "reason" : "[2:3] [UpdateRequest] unknown field [dac] did you mean [doc]?"
      }
    ],
    "type" : "x_content_parse_exception",
    "reason" : "[2:3] [UpdateRequest] unknown field [dac] did you mean [doc]?"
  },
  "status" : 400
}
```

The tricky thing about implementing this is that x-content doesn't
depend on Lucene. So this works by creating an extension point for the
error message using SPI. Elasticsearch's server module provides the
"spell checking" implementation.
2020-01-14 10:42:05 -05:00
Alexander Reelsen c9786592eb
Sync grok patterns with logstash patterns (#50381)
In order to ensure that logstash and Elasticsearch are able to understand
the same patterns, this commit adapts to changes in logstash, adds a few
patterns and changes a few.
2020-01-08 14:50:43 +01:00
Nik Everett 792b5e150b
Teach ObjectParser a happy pattern (#50691)
We *very* commonly have object with ctors like:
```
public Foo(String name)
```

And then declare a bunch of setters on the object. Every aggregation
works like this, for example. This change teaches `ObjectParser` how to
build these aggregations all on its own, without any help. This'll make
it much cleaner to parse aggs, and, probably, a bunch of other things.
It'll let us remove lots of wrapping. I've used this new power for the
`avg` aggregation just to prove that it works outside of a unit test.
2020-01-07 10:55:17 -05:00
Nik Everett 5ea750f2ca
Clean up wire test case a bit (#50627)
* Adds JavaDoc to `AbstractWireTestCase` and
`AbstractWireSerializingTestCase` so it is more obvious you should prefer
the latter if you have a choice
* Moves the `instanceReader` method out of `AbstractWireTestCase` becaue
it is no longer used.
* Marks a bunch of methods final so it is more obvious which classes are
for what.
* Cleans up the side effects of the above.
2020-01-05 14:42:34 -05:00
Nik Everett 4ccbd3e3a1
x-content: Support collapsed named objects (#50564)
This adds support for "collapsed" named object to `ObjectParser`. In
particular, this supports the sort of xcontent that we use to specify
significance heuristics. See #25519 and this example:

```
GET /_search
{
    "query" : {
        "terms" : {"force" : [ "British Transport Police" ]}
    },
    "aggregations" : {
        "significant_crime_types" : {
            "significant_terms" : {
                "field" : "crime_type",
                "mutual_information" : { <<------- This is the name
                    "include_negatives": true
                }
            }
        }
    }
}
```

I believe there are a couple of things that work this way.

I've held off on moving the actual parsing of the significant heuristics
to this code to keep the review more compact. The moving is pretty
mechanical stuff in the aggs framework.
2020-01-03 11:58:29 -05:00
Igor Motov a26e4d1e5e
Geo: Switch generated WKT to upper case (#50285)
Switches generated WKT to upper case to
conform to the standard recommendation.

Relates #49568
2019-12-18 07:28:56 -10:00
Rory Hunter 9f069f795c
Refactor environment variable processing for Docker (#49612)
Closes #45223.

The current Docker entrypoint script picks up environment variables and
translates them into -E command line arguments. However, since any tool
executes via `docker exec` doesn't run the entrypoint, it results in
a poorer user experience.

Therefore, refactor the env var handling so that the -E options are
generated in `elasticsearch-env`. These have to be appended to any
existing command arguments, since some CLI tools have subcommands and
-E arguments must come after the subcommand.

Also extract the support for `_FILE` env vars into a separate script, so
that it can be called from more than once place (the behaviour is
idempotent).

Finally, add noop -E handling to CronEvalTool for parity, and support
`-E` in MultiCommand before subcommands.
2019-12-16 09:57:18 +00:00
Yannick Welsch 678aeb747e
Make elasticsearch-node tools custom metadata-aware (#48390)
The elasticsearch-node tools allow manipulating the on-disk cluster state. The tool is currently
unaware of plugins and will therefore drop custom metadata from the cluster state once the
state is written out again (as it skips over the custom metadata that it can't read). This commit
preserves unknown customs when editing on-disk metadata through the elasticsearch-node
command-line tools.
2019-12-10 09:45:27 +01:00
Orhan Toy 8b3c599bfb Consistent case in CLI option descriptions (#49635)
This commit improves the casing of messages in the CLI help descriptions.
2019-12-05 13:33:36 -08:00
Yannick Welsch d9560e5340
Add SecureSM support for newer IDEA versions (#49747)
IntelliJ IDEA moved their JUnit runner to a different package. While this does not break running
tests in IDEA, it leads to an ugly exception being thrown at the end of the tests:

Exception in thread "main" java.lang.SecurityException: java.lang.System#exit(0) calls are not
allowed
	at org.elasticsearch.secure_sm.SecureSM$2.run(SecureSM.java:248)
	at org.elasticsearch.secure_sm.SecureSM$2.run(SecureSM.java:215)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:310)
	at org.elasticsearch.secure_sm.SecureSM.innerCheckExit(SecureSM.java:215)
	at org.elasticsearch.secure_sm.SecureSM.checkExit(SecureSM.java:206)
	at java.base/java.lang.Runtime.exit(Runtime.java:111)
	at java.base/java.lang.System.exit(System.java:1781)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:59)

This commit adds support for newer IDEA versions in SecureSM.
2019-12-04 13:49:43 +01:00
Tal Levy 782b4f4436
correct licensing and incorporation of FastMath (#49122)
this resolves incorrectly licensed code in #49009.

ESSloppyMath is made as a wrapper around FastMath.java which is 
not meant to be modified with code beyond the original source
2019-11-21 07:10:01 -08:00
Tim Vernum bbaa1f5fd5
Improved diagnostics for TLS trust failures (#48911)
- Improves HTTP client hostname verification failure messages
- Adds "DiagnosticTrustManager" which logs certificate information
  when trust cannot be established (hostname failure, CA path failure,
  etc)

These diagnostic messages are designed so that many common TLS
problems can be diagnosed based solely (or primarily) on the
elasticsearch logs.

These diagnostics can be disabled by setting 

     xpack.security.ssl.diagnose.trust: false
2019-11-20 18:19:31 +11:00
Rory Hunter 090859bea9
Don't close stderr under `--quiet` (#47208)
Closes #46900. When running ES with `--quiet`, if ES then exits abnormally, a
user has to go hunting in the logs for the error. Instead, never close
System.err, and print more information to it if ES encounters a fatal error
e.g. config validation, or some fatal runtime exception. This is useful when
running under e.g. systemd, since the error will go into the journal.

Note that stderr is still closed in daemon (`-d`) mode.
2019-11-15 13:58:28 +00:00