Commit Graph

990 Commits

Author SHA1 Message Date
Nhat Nguyen c893a3e495
Make soft-deletes mandatory in 8.0 (#51122)
Creating indices with soft deletes disabled is no longer supported in 8.0.
2020-01-17 17:34:22 -05:00
Nhat Nguyen 09b46c8646
Goodbye and thank you synced flush! (#50882)
Synced flush was a brilliant idea. It supports instant recoveries with a 
quite small implementation. However, with the presence of sequence
numbers and retention leases, it is no longer needed. This change
removes it from 8.0.

Relates #5077
2020-01-16 09:43:07 -05:00
Christoph Büscher 9a4357ae04
Deprecate and remove camel-case nGram and edgeNGram tokenizers (#50862)
We already deprecated and removed the camel-case versions of the nGram and edgeNGram 
filters a while ago and we should do the same with the nGram and edgeNGram tokenizers.
This PR deprecates the use of these names in favour of ngram and edge_ngram in 7
and disallows usage in new indices starting with 8.

Closes #50561
2020-01-14 17:18:47 +01:00
Julie Tibshirani 54f5907067
Remove the 'template' field in index templates. (#49460)
The `template` field was deprecated in 6.0 in favor of `index_patterns`, and can
now be removed.

Relates to #21009.
2020-01-09 17:07:58 -08:00
David Turner 8c6f821c75
Remove the 'local' parameter of /_cat/nodes (#50594)
The cat nodes API performs a `ClusterStateAction` then a `NodesInfoAction`.
Today it accepts the `?local` parameter and passes this to the
`ClusterStateAction` but this parameter has no effect on the `NodesInfoAction`.
This is surprising, because `GET _cat/nodes?local` looks like it might be a
completely local call but in fact it still depends on every node in the
cluster.

This parameter was deprecated in 7.x in #50499 and this commit removes it.

Relates #50088
2020-01-06 14:12:51 +00:00
Julie Tibshirani 460d545921
Remove support for sparse vectors. (#48781)
Follow up to #48368. This PR removes support for `sparse_vector` on new
indices. On 7.x indices a `sparse_vector` can still be defined, but it is not
possible to index or search on it.
2019-11-14 16:54:48 -05:00
Tim Vernum eb3c57b8eb
Reject misconfigured/ambiguous SSL server config (#45892)
This commit makes it an error to start a node where either of the
server contexts (xpack.security.transport.ssl and
xpack.security.http.ssl) meet either of these conditions:

1. The server lacks a certificate/key pair (i.e. neither
   ssl.keystore.path not ssl.certificate are configured)
2. The server has some ssl configuration, but ssl.enabled is not
   specified. This new validation does not care whether ssl.enabled is
   true or false (though other validation might), it simply makes it
   an error to configure server SSL without being explicit about
   whether to enable that configuration.
2019-11-07 11:51:48 -05:00
David Turner 3ce7a37f1f
Remove index.force_memory_term_dictionary setting (#48873)
This temporary setting was introduced in #39741 but has had no effect since it
was deprecated in #42838. This commit removes this setting.
2019-11-05 20:17:32 -05:00
Julie Tibshirani 939c24223b
Remove deprecated vector functions. (#48725)
Follow up to #48604. This PR removes the deprecated vector function signatures
of the form `cosineSimilarity(query, doc['field'])`.
2019-10-31 10:19:03 -07:00
Julie Tibshirani a12b4094bf
Remove deprecated endpoints containing _xpack. (#48170)
The endpoints with `_xpack` in their path were deprecated in 7.x and can now be
removed. This commit removes deprecated endpoints for the following APIs:
* deprecation
* graph
* license
* monitoring
* rollup
* SQL
* watcher
2019-10-28 18:23:04 -07:00
David Turner 7b652adfbf
Remove include_relocations setting (#47717)
Setting `cluster.routing.allocation.disk.include_relocations` to `false` is a
bad idea since it will lead to the kinds of overshoot that were otherwise fixed
in #46079. This setting was deprecated in #47443. This commit removes it.
2019-10-08 13:33:49 +02:00
Colin Goodheart-Smithe a574a66a21 Removes typed URLs from mapping APIs (#41676)
Relates to #41059
2019-09-23 18:20:07 +01:00
Christoph Büscher 2351aa3efb
Disallow `_field_names` enabled setting (#46681)
After deprecating the `enabled` setting for `_field_names`starting with 7.5, 
this change disallows the setting on new indices in 8.0. The setting continues
to work for indices created in 7.x where we continue emitting a deprecation warning.

Relates to #42854
2019-09-23 14:13:44 +02:00
Alan Woodward 7c90801aff
Remove types from Get/MultiGet (#46587)
This commit removes types from the ShardGetService, and propagates this API change
up through the Transport and Rest actions for Get and MultiGet

Relates to #41059
2019-09-20 14:22:57 +01:00
James Rodewig 97802d8aff
[DOCS] Change // CONSOLE comments to [source,console] (#46441) 2019-09-06 10:55:16 -04:00
James Rodewig 466c59a4a7
[DOCS] Replace "// TESTRESPONSE" magic comments with "[source,console-result] (#46295) 2019-09-05 16:47:18 -04:00
Jim Ferenczi a4ed7b1ca1
Decouple shard allocation awareness from search and get requests (#45735)
With this commit, Elasticsearch will no longer prefer using shards in the same location
(with the same awareness attribute values) to process `_search` and `_get` requests.
Instead, adaptive replica selection (the default since 7.0) should route requests more efficiently
using the service time of prior inter-node communications. Clusters with big latencies between
nodes should switch to cross cluster replication to isolate nodes within the same zone.
Note that this change only targets 8.0 since it is considered as breaking. However a follow up
pr should add an option to activate this behavior in 7.x in order to allow users to opt-in early.

Closes #43453
2019-09-04 21:48:03 +02:00
André Letterer 5ec9d7be4d
Fixed synchronizing REST API inflight breaker names with internal variable (#40878)
The internal configuration settings were like that: network.breaker.inflight_requests
But the exposed REST API had the value names with underscore like that: network.breaker.in_flight_requests
This was now corrected to without underscores like that: network.breaker.inflight_requests
2019-09-03 14:56:36 +02:00
Jason Tedor c9d397fc77
Remove the pidfile setting (#45940)
The pidfile setting was deprecated in version 7.4.0 of Elasticsearch for
removal in Elasticsearch 8.0.0. This commit removes the pidfile setting.
2019-08-24 17:19:35 -04:00
Jason Tedor fe9f0b48fd
Remove processors setting (#45905)
The processors setting was deprecated in version 7.4.0 of Elasticsearch
for removal in Elasticsearch 8.0.0. This commit removes the processors
setting.
2019-08-23 14:30:53 -04:00
David Turner bc31ea752e
Always auto-release the flood-stage block (#45274)
Removes support for using a system property to disable the automatic release of
the write block applied when a node exceeds the flood-stage watermark.

Relates #42559
2019-08-08 11:47:14 +01:00
Gordon Brown b2fa9209e5
Add warning about versions to Deprecation API docs (#36624)
Add a note that the Deprecation API may not be up to date with all
breaking changes until the last minor version in a major version series.
2019-08-07 13:48:45 -06:00
Lee Hinman 9677d045a5
Rename HLRC 'indexlifecycle' components to 'ilm' (#44982)
Related to #44917 and #44725, this commit renames the HLRC components
for indexlifecycle to ilm.
2019-07-30 15:09:06 -06:00
Jason Tedor 5b2d1a5d39
Limit processors by available processors (#44894)
This commit limits the processors setting to be more than the number of
available processors.
2019-07-27 00:07:12 +09:00
Tanguy Leroux 321c2b8627
Force Merge should reject requests with `only_expunge_deletes` and `max_num_segments` set (#44761)
This commit changes the ForceMergeRequest.validate() method so that it does 
not accept the parameters only_expunge_deletes and max_num_segments 
to be set at the same time.

The motivation is that InternalEngine.forceMerge() just ignores the max. number 
of segments parameter when the only expunge parameter is set to true, leaving 
the wrong impression to the user that max. number of segments has been applied. 
It also changes InternalEngine.forceMerge() so that it now throws an exception 
when both parameters are set, and modifies tests where needed.

Because it changes the behavior of the REST API I marked this as >breaking. 

Closes #43102
2019-07-26 09:34:36 +02:00
Lisa Cawley 9413f97c1f
[DOCS] Reformat info and deprecation APIs to use new API format (#44599) 2019-07-18 16:10:22 -07:00
David Turner 0df12a8580
Fail node containing ancient closed index (#44264)
Today we fail the node at startup if it contains an index that is too old to be
compatible with the current version, unless that index is closed. If the index
is closed then the node will start up and this puts us into a bad state: the
index cannot be opened and must be reindexed using an earlier version, but we
offer no way to get that index into a node running an earlier version so that
it can be reindexed. Downgrading the node in-place is decidedly unsupported and
cannot be expected to work since the node already started up and upgraded the
rest of its metadata. Since #41731 we actively reject downgrades to versions ≥
v7.2.0 too.

This commit prevents the node from starting in the presence of any too-old
indices (closed or not). In particular, it does not write any upgraded metadata
in this situation, increasing the chances an in-place downgrade might be
successful. We still actively reject the downgrade using #41731, because we
wrote the node metadata file before checking the index metadata, but at least
there is a way to override this check.

Relates #21830, #44230
2019-07-15 15:19:22 +01:00
Armin Braun 58aae32f71
Fix Incorrect Version in Migration Docs (#44325)
* This was accidentally left at `7.3` but #33164 was merged too late and it should now be `7.4`
2019-07-15 08:40:14 +02:00
Armin Braun fe2a870668
HLRC: Fix '+' Not Correctly Encoded in GET Req. (#33164)
* HLRC: Fix '+' Not Correctly Encoded in GET Req.

* Encode `+` correctly as `%2B` in URL paths
* Keep encoding `+` as space in URL parameters
* Closes #33077
2019-07-15 08:09:49 +02:00
Armin Braun d6e23e9bd5
Provide an Option to Use Path-Style-Access with S3 Repo (#41966)
* Provide an Option to Use Path-Style-Access with S3 Repo

* As discussed, added the option to use path style access back again and
deprecated it.
* Defaulted to `false`
* Added warning to docs

* Closes #41816
2019-07-04 17:37:47 +02:00
Jay Modi 0a41b13cd8 Remove the client transport profile filter (#43236)
Now that the transport client has been removed, the client transport
profile filter can be removed from security. This filter prevented node
actions from being executed using a transport client.
2019-07-03 19:32:24 +10:00
Jim Ferenczi a614415838
Remove deprecated sort options: nested_path and nested_filter (#42809)
This commit removes the nested_path and nested_filter options deprecated in 6x.
This change also checks that the sort field has a [nested] option if it is under a nested
object and throws an exception if it's not the case.

Closes #27098
2019-06-27 17:30:02 +02:00
Henning Andersen ac3e4a6799
Reindex remove outer level size (#43373)
This commit finalizes the work done to rename size to max_docs in
reindex and update/delete by query. size is no longer supported in URL
or outer level body for the 3 APIs (though size in update/delete-by-query
will and has always been interpreted as scroll_size, it is not to be relied
upon).

Continuation of #41894

Closes #24344
2019-06-26 09:36:17 +02:00
Andrey Ershov b4f30cf1ed
Get snapshots response example for the breaking change (#43475)
#42090 PR added support
for requesting snapshots from multiple repositories. And it has changed
the response format in a non-BwC way.
There is a mentioning of a response format change in the breaking
changes docs, however, there is no example of how new format looks
like. Pointed out by @dakrone.
This commit adds the missing example.
2019-06-25 10:37:06 +03:00
Andrey Ershov 680d6edc0b
Get snapshots support for multiple repositories (#42090)
This commit adds multiple repositories support to get snapshots
request.
If some repository throws an exception this method does not fail fast
instead, it returns results for all repositories.
This PR is opened in favour of #41799, because we decided to change
the response format in a non-BwC manner. It makes sense to read a
discussion of the aforementioned PR.
This is the continuation of work done here #15151.
2019-06-19 16:04:13 +03:00
Zachary Tong 64642e4a7d
RollupStart endpoint should return OK if job already started (#41502)
If a job is started or indexing, RollupStart should always return
a success (200 OK) response since the job is, in fact, started
2019-06-03 11:24:16 -04:00
Marios Trivyzas 6dd4d2b7a6
Remove CommonTermsQuery and cutoff_frequency param (#42654)
Remove `common` query and `cutoff_frequency` parameter of
`match` and `multi_match` queries. Both have already been
deprecated for the next 7.x version.

Closes: #37096
2019-05-31 17:06:06 +02:00
Yannick Welsch 6e39433cd5
Remove "nodes/0" folder prefix from data path (#42489)
With the removal of node.max_local_storage_nodes, there is no need anymore to keep the data in
subfolders indexed by a node ordinal. This commit makes it so that ES 8.0 will store data directly in
$DATA_DIR instead of $DATA_DIR/nodes/$nodeOrdinal.

Upon startup, Elasticsearch will check to see if there is data in the old location, and automatically
move it to the new location. This automatic migration only works if $nodeOrdinal is 0, i.e., multiple
node instances have not previously run on the same data path, which required for
node.max_local_storage_nodes to explicitly be configured.
2019-05-28 19:09:09 +02:00
Julie Tibshirani f07b90f3c3
Remove support for chained multi-fields. (#42333)
Follow-up to #41926, where we deprecated support for multi-fields within
multi-fields.

Addresses #41267.
2019-05-28 09:49:40 -07:00
Jason Tedor 1b0c728cfa
Remove deprecated search.remote settings (#42381)
We deprecated these settings awhile ago, in favor of cluster.remote. In
7.x we were gentle and provided automatic upgrade of these settings to
the new settings. Now it is time for them to go. This commit removes the
deprecated search.remote settings.
2019-05-23 08:06:07 -07:00
Yannick Welsch c459ea828f
Remove node.max_local_storage_nodes (#42428)
This setting, which prior to Elasticsearch 5 was enabled by default and caused all kinds of
confusion, has since been disabled by default and is not recommended for production use. The
preferred way going forward is for users to explicitly specify separate data folders for each started
node to ensure that each node is consistently assigned to the same data path.

Relates to #42426
2019-05-23 16:02:12 +02:00
Simon Willnauer cb402220d8
Remove deprecated Repository methods (#42359)
We deprecated `restoreShard` and `snapshotShard` in #42213
This change removes the deprecated methods and their usage and adds
a note in the migration docs.
2019-05-23 12:29:39 +02:00
Ryan Ernst f45a4731bb
Remove the migrate tool (#42174)
This commit removes the deprecated migrate tool which was used to
migrate users from the file realm to native realm when the native realm
was first created.
2019-05-17 14:49:05 -04:00
Colin Goodheart-Smithe 3647d7c94e
Removes typed REST endpoints from search and related APIs (#41640) 2019-05-09 14:41:11 +01:00
James Rodewig adf67053f4
[DOCS] Add anchors for Asciidoctor migration (#41648) 2019-04-30 10:19:09 -04:00
Henning Andersen 8dbdd06886
Reindex from Remote encoding (#41007)
Removed the leniency when encoding remote reindex search requests that
was introduced in 7.x. All index-names are now encoded before being sent
to the remote host.

Follow-up to #40303
2019-04-16 16:46:54 +02:00
Tim Brooks 07e0f0df21
Remove deprecated transport settings (#40918)
This is related to #36652. There are a number of transport settings that
were deprecated and replaced with new versions in 7.x. This commit
removes them from 8.0.
2019-04-12 15:42:05 -06:00
Gordon Brown 454e4967f1
Add ILM poll_interval limit to breaking changes (#41095)
ILM's poll_interval setting now has a lower limit of 1 second, which is
technically a breaking change.
2019-04-12 10:43:10 -06:00
Lisa Cawley f88e56a627
[DOCS] Add notable-breaking-changes tags (#40990) 2019-04-08 18:20:59 -07:00
Jason Tedor 8deadfb2d1
Bump the minimum Java version to Java 11 (#40754)
With the 8.0.0 release of Elasticsearch we will bump the minimum
required Java to Java 11. This commit puts this into effect on the
master branch.
2019-04-07 16:16:23 -04:00
Christoph Büscher 1597f69e44
Make Fuzziness reject illegal values earlier (#33511)
The current java implementation of Fuzziness leaves a lot of room for
initializing it with illegal values that either cause errors later when the
queries reach the shards where they are executed or values that are silently
ignored in favour of defaults. We should instead tighten the java implementation
of the class so that we only accept supported values. Currently those are
numeric values representing the edit distances 0, 1 and 2, optionally also as
float or string, and the "AUTO" fuzziness, which can come in a cusomizable
variant that allows specifying two value that define the positions in a term
where the AUTO option increases the allowed edit distance.

This change removes several redundant ways of object construction and adds input
validation to the remaining ones. Java users should either use one of the
predefined constants or use the static factory methods `fromEdits(int)` or
`fromString(String)` to create instances of the class, while other ctors are
hidden. This allows for instance control, e.g. returning one of the constants
when creating instances from an integer value.

Previously the class would accept any positive integer value and any float
value, while in effect the maximum allowed edit distance was capped at 2 in
practice. These values while throw an error now, as will any other String value
other than "AUTO" that where previously accepted but led to numeric exceptions
when the query was executed.
2019-04-05 10:35:35 +02:00
Deb Adair 75c5a8cd6a [DOCS] Removed redundant (not quite right) information about upgrades. 2019-04-04 16:20:08 -07:00
Tim Vernum 7b6e7146ea
Remove obsolete security settings (#40496)
Removes the deprecated accept_default_password setting.
This setting become redundant when default passwords were removed
from 6.0, but the setting was kept for BWC.

Removes native role store cache settings.
These have been unused since 5.2 but were kept for BWC.
2019-03-29 18:04:49 +11:00
lcawl 7dc6ddf62c [DOCS] Remove broken link 2019-03-22 15:57:27 -07:00
lcawl 05af3a51fb [DOCS] Removes notable breaking changes section 2019-03-22 15:45:13 -07:00
Gordon Brown df11ba8a32
Remove Migration Upgrade and Assistance APIs (#40075)
The Migration Assistance API has been functionally replaced by the
Deprecation Info API, and the Migration Upgrade API is not used for the
transition from ES 6.x to 7.x, and does not need to be kept around to
repair indices that were not properly upgraded before upgrading the
cluster, as was the case in 6.
2019-03-15 15:34:50 -06:00
Henning Andersen 62bb853161
Blob Store compress default to true (#40033)
Changed default of compress setting from false to true for blob store
repositories. This aligns the code with existing documentation and also
seems like the better default.
2019-03-15 12:23:22 +01:00
Lisa Cawley dda8cf82f8
[DOCS] Adds notable changes section (#38238) 2019-03-08 10:51:27 -08:00
Yannick Welsch 3b71a31557
Remove Zen1 (#39466)
Removes all traces of Zen1 from the code base. Some of these commits will also be backported to
7.0/7.x (#39470) as the cluster.coordination package was making use of some things in
discovery.zen and we want to keep 7.x as close as possible to master.
2019-03-04 15:51:12 +01:00
Henning Andersen ac7ec99bea
Unify blob store compress setting (#39346)
Blob store compression was all implemented generally, except reading the
setting for it. Moved the setting to BlobStoreRepository to unify this.
Also removed deprecated env setting 'repositories.fs.compress'.

This is a follow up on #39073
2019-02-28 12:34:13 +01:00
Christoph Büscher 6e3e3b1530
Move some token filter migration notes (#39072)
The removal of the "nGram" and "edgeNGram" token filter was filed in a new
migrate_8_0/mappings.asciidoc document but it should better be in an
analysis.asciidoc section and file.
2019-02-26 15:02:43 +01:00
Tamara Braun c1ab8218f1 Enforce Completion Context Limit (#38675)
This change adds a limit to the number of completion contexts that a completion field can define.

Closes #32741
2019-02-18 12:51:39 +01:00
Christoph Büscher 7bb2da197d
Remove `nGram` and `edgeNGram` token filter names (#38911)
In #30209 we deprecated the camel case `nGram` filter name in favour of `ngram` and
did the same for `edgeNGram` and `edge_ngram`. Using these names has been deprecated
since 6.4 and is issuing deprecation warnings since then.
I think we can remove these filters in 8.0. In a backport of this PR I would change what was a
dreprecation warning from 6.4. to an error starting with new indices created in 7.0.
2019-02-15 20:15:05 +01:00
Lisa Cawley 06674235b2
[DOCS] Fixes broken migration links (#38655) 2019-02-08 16:57:36 -08:00
Lisa Cawley 4775919cac
[DOCS] Adds placeholders for v8 highlights, breaking changes, release notes (#38641) 2019-02-08 15:48:42 -08:00
David Turner 5a3c452480
Align docs etc with new discovery setting names (#38492)
In #38333 and #38350 we moved away from the `discovery.zen` settings namespace
since these settings have an effect even though Zen Discovery itself is being
phased out. This change aligns the documentation and the names of related
classes and methods with the newly-introduced naming conventions.
2019-02-06 11:34:38 +00:00
Luca Cavanna a7046e001c
Remove support for maxRetryTimeout from low-level REST client (#38085)
We have had various reports of problems caused by the maxRetryTimeout
setting in the low-level REST client. Such setting was initially added
in the attempts to not have requests go through retries if the request
already took longer than the provided timeout.

The implementation was problematic though as such timeout would also
expire in the first request attempt (see #31834), would leave the
request executing after expiration causing memory leaks (see #33342),
and would not take into account the http client internal queuing (see #25951).

Given all these issues, it seems that this custom timeout mechanism 
gives little benefits while causing a lot of harm. We should rather rely 
on connect and socket timeout exposed by the underlying http client 
and accept that a request can overall take longer than the configured 
timeout, which is the case even with a single retry anyways.

This commit removes the `maxRetryTimeout` setting and all of its usages.
2019-02-06 08:43:47 +01:00
Boaz Leskes 033ba725af
Remove support for internal versioning for concurrency control (#38254)
Elasticsearch has long [supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning) compare and set (a.k.a optimistic concurrency control) operations using internal document versioning. Sadly that approach is flawed and can sometime do the wrong thing. Here's the relevant excerpt from the resiliency status page:

> When a primary has been partitioned away from the cluster there is a short period of time until it detects this. During that time it will continue indexing writes locally, thereby updating document versions. When it tries to replicate the operation, however, it will discover that it is partitioned away. It won’t acknowledge the write and will wait until the partition is resolved to negotiate with the master on how to proceed. The master will decide to either fail any replicas which failed to index the operations on the primary or tell the primary that it has to step down because a new primary has been chosen in the meantime. Since the old primary has already written documents, clients may already have read from the old primary before it shuts itself down. The version numbers of these reads may not be unique if the new primary has already accepted writes for the same document 

We recently [introduced](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/optimistic-concurrency-control.html) a new sequence number based approach that doesn't suffer from this dirty reads problem. 

This commit removes support for internal versioning as a concurrency control mechanism in favor of the sequence number approach.

Relates to #1078
2019-02-05 20:53:35 +01:00
David Turner f2dd5dd6eb
Remove DiscoveryPlugin#getDiscoveryTypes (#38414)
With this change we no longer support pluggable discovery implementations. No
known implementations of `DiscoveryPlugin` actually override this method, so in
practice this should have no effect on the wider world. However, we were using
this rather extensively in tests to provide the `test-zen` discovery type. We
no longer need a separate discovery type for tests as we no longer need to
customise its behaviour.

Relates #38410
2019-02-05 17:42:24 +00:00
David Turner 3b2a0d7959
Rename no-master-block setting (#38350)
Replaces `discovery.zen.no_master_block` with `cluster.no_master_block`. Any
value set for the old setting is now ignored.
2019-02-05 08:47:56 +00:00
David Turner 2d114a02ff
Rename static Zen1 settings (#38333)
Renames the following settings to remove the mention of `zen` in their names:

- `discovery.zen.hosts_provider` -> `discovery.seed_providers`
- `discovery.zen.ping.unicast.concurrent_connects` -> `discovery.seed_resolver.max_concurrent_resolvers`
- `discovery.zen.ping.unicast.hosts.resolve_timeout` -> `discovery.seed_resolver.timeout`
- `discovery.zen.ping.unicast.hosts` -> `discovery.seed_addresses`
2019-02-05 08:46:52 +00:00
Przemyslaw Gomulka 89feaa0e23
Migration doc for audit json log file (#38165)
The migration documentation for an audit logging changes. Removal of plaintext logs and rename of json log file

relates #32850
2019-02-05 09:09:15 +01:00
Benjamin Trent a70f54fc77
Adding ml_settings entry to HLRC and Docs for deprecation_info (#38118) 2019-02-01 12:45:28 -06:00
Jay Modi 2ca22209cd
Enable TLSv1.3 by default for JDKs with support (#38103)
This commit enables the use of TLSv1.3 with security by enabling us to
properly map `TLSv1.3` in the supported protocols setting to the
algorithm for a SSLContext. Additionally, we also enable TLSv1.3 by
default on JDKs that support it.

An issue was uncovered with the MockWebServer when TLSv1.3 is used that
ultimately winds up in an endless loop when the client does not trust
the server's certificate. Due to this, SSLConfigurationReloaderTests
has been pinned to TLSv1.2.

Closes #32276
2019-02-01 08:34:11 -07:00
Jim Ferenczi 6fa93ca493
Forbid negative field boosts in analyzed queries (#37930)
This change forbids negative field boost in the `query_string`, `simple_query_string`
and `multi_match` queries.
Negative boosts are not allowed in Lucene 8 (scores must be positive).
The backport of this change to 6x will turn the error into a deprecation warning
in order to raise the awareness of this breaking change in 7.0.

Closes #33309
2019-02-01 11:41:40 +01:00
Tim Vernum 6fcbd07420
Remove heuristics that enable security on trial licenses (#38075)
In 6.3 trial licenses were changed to default to security
disabled, and ee added some heuristics to detect when security should
be automatically be enabled if `xpack.security.enabled` was not set.

This change removes those heuristics, and requires that security be
explicitly enabled (via the `xpack.security.enabled` setting) for
trial licenses.

Relates: #38009
2019-02-01 17:59:13 +11:00
Henning Andersen ff7620fd7f
Fail start on obsolete indices documentation (#37786)
Added breaking changes documentation for node start up obsolete indices
detection.

Issue #27073
2019-01-31 15:40:06 +01:00
David Turner 81c443c9de
Deprecate minimum_master_nodes (#37868)
Today we pass `discovery.zen.minimum_master_nodes` to nodes started up in
tests, but for 7.x nodes this setting is not required as it has no effect.
This commit removes this setting so that nodes are started with more realistic
configurations, and deprecates it.
2019-01-30 20:09:15 +00:00
Lee Hinman cac6b8e06f
Add ECS schema for user-agent ingest processor (#37727) (#37984)
* Add ECS schema for user-agent ingest processor (#37727)

This switches the format of the user agent processor to use the schema from [ECS](https://github.com/elastic/ecs).
So rather than something like this:

```
{
  "patch" : "3538",
  "major" : "70",
  "minor" : "0",
  "os" : "Mac OS X 10.14.1",
  "os_minor" : "14",
  "os_major" : "10",
  "name" : "Chrome",
  "os_name" : "Mac OS X",
  "device" : "Other"
}
```

The structure is now like this:

```
{
  "name" : "Chrome",
  "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
  "os" : {
    "name" : "Mac OS X",
    "version" : "10.14.1",
    "full" : "Mac OS X 10.14.1"
  },
  "device" : "Other",
  "version" : "70.0.3538.102"
}
```

This is now the default for 7.0. The deprecated `ecs` setting in 6.x is not
supported.

Resolves #37329

* Remove `ecs` setting from docs
2019-01-30 11:24:18 -07:00
Adrien Grand c8af0f4bfa
Use mappings to format doc-value fields by default. (#30831)
Doc-value fields now return a value that is based on the mappings rather than
the script implementation by default.

This deprecates the special `use_field_mapping` docvalue format which was added
in #29639 only to ease the transition to 7.x and it is not necessary anymore in
7.0.
2019-01-30 10:31:51 +01:00
Albert Zaharovits 697b2fbe52
Remove implicit index monitor privilege (#37774)
Restricted indices (currently only .security-6 and .security) are special
internal indices that require setting the `allow_restricted_indices` flag
on every index permission that covers them. If this flag is `false`
(default) the permission will not cover these and actions against them
will not be authorized.
However, the monitoring APIs were the only exception to this rule.

This exception is herein forfeited and index monitoring privileges have to be
granted explicitly, using the `allow_restricted_indices` flag on the permission,
as is the case for any other index privilege.
2019-01-29 21:10:03 +02:00
Przemyslaw Gomulka 891320f5ac
Elasticsearch support to JSON logging (#36833)
In order to support JSON log format, a custom pattern layout was used and its configuration is enclosed in ESJsonLayout. Users are free to use their own patterns, but if smooth Beats integration is needed, they should use ESJsonLayout. EvilLoggerTests are left intact to make sure user's custom log patterns work fine.

To populate additional fields node.id and cluster.uuid which are not available at start time, 
a cluster state update will have to be received and the values passed to log4j pattern converter.
A ClusterStateObserver.Listener is used to receive only one ClusteStateUpdate. Once update is received the nodeId and clusterUUid are set in a static field in a NodeAndClusterIdConverter. 

Following fields are expected in JSON log lines: type, tiemstamp, level, component, cluster.name, node.name, node.id, cluster.uuid, message, stacktrace
see ESJsonLayout.java for more details and field descriptions

Docker log4j2 configuration is now almost the same as the one use for ES binary. 
The only difference is that docker is using console appenders, whereas ES is using file appenders.

relates: #32850
2019-01-29 07:20:09 +01:00
Jim Ferenczi 787acb14b9
Track total hits up to 10,000 by default (#37466)
This commit changes the default for the `track_total_hits` option of the search request
to `10,000`. This means that by default search requests will accurately track the total hit count
up to `10,000` documents, requests that match more than this value will set the `"total.relation"`
to `"gte"` (e.g. greater than or equals) and the `"total.value"` to `10,000` in the search response.
Scroll queries are not impacted, they will continue to count the total hits accurately.
The default is set back to `true` (accurate hit count) if `rest_total_hits_as_int` is set in the search request.
I choose `10,000` as the default because that's also the number we use to limit pagination. This means that users will be able to know how far they can jump (up to 10,000) even if the total number of hits is not accurate.

Closes #33028
2019-01-25 13:45:39 +01:00
Tim Vernum 03690d12b2
Remove TLS 1.0 as a default SSL protocol (#37512)
The default value for ssl.supported_protocols no longer includes TLSv1
as this is an old protocol with known security issues.
Administrators can enable TLSv1.0 support by configuring the
appropriate `ssl.supported_protocols` setting, for example:

xpack.security.http.ssl.supported_protocols: ["TLSv1.2","TLSv1.1","TLSv1"]

Relates: #36021
2019-01-25 15:46:39 +11:00
Christoph Büscher 967de04257
Uppercasing some docs section title (#37781)
Section titles are mostly uppercase, only a few cases where query DSL parameters
or Java method names are used as the title they should be lowercased.
2019-01-24 22:54:55 +01:00
Albert Zaharovits b6936e3c1e
Remove index audit output type (#37707)
This commit removes the Index Audit Output type, following its deprecation
in 6.7 by 8765a31d4e. It also adds the migration notice (settings notice).

In general, the problem with the index audit output is that event indexing
can be slower than the rate with which audit events are generated,
especially during the daily rollovers or the rolling cluster upgrades.
In this situation audit events will be lost which is a terrible failure situation
for an audit system.
Besides of the settings under the `xpack.security.audit.index` namespace, the
`xpack.security.audit.outputs` setting has also been deprecated and will be
removed in 7. Although explicitly configuring the logfile output does not touch
any deprecation bits, this setting is made redundant in 7 so this PR deprecates
it as well.

Relates #29881
2019-01-24 12:36:10 +02:00
Albert Zaharovits 5308746270
Remove Watcher Account "unsecure" settings (#36736)
Removes all sensitive settings (passwords, auth tokens, urls, etc...) for
watcher notifications accounts. These settings were deprecated (and
herein removed) in favor of their secure sibling that is set inside the
elasticsearch keystore. For example:
`xpack.notification.email.account.<id>.smtp.password`
is no longer a valid setting, and it is replaced by
`xpack.notification.email.account.<id>.smtp.secure_password`
2019-01-20 12:51:24 +02:00
Przemyslaw Gomulka b6e5ccaf8a
Remove the AbstracLifecycleComponent constructor with Settings (#37523)
Adding the migration guide and removing the deprecated in 6.x
constructor

relates #35560
relates #34488
2019-01-17 09:10:09 +01:00
Jay Modi f3edbe2911
Security: remove SSL settings fallback (#36846)
This commit removes the fallback for SSL settings. While this may be
seen as a non user friendly change, the intention behind this change
is to simplify the reasoning needed to understand what is actually
being used for a given SSL configuration. Each configuration now needs
to be explicitly specified as there is no global configuration or
fallback to some other configuration.

Closes #29797
2019-01-14 14:06:22 -07:00
Jun Ohtani 38b698d455
[Analysis] Deprecate Standard Html Strip Analyzer in master (#26719)
* [Analysis] Deprecate Standard Html Strip Analyzer

Deprecate only Standard Html Strip Analyzer
If user create index with the analyzer since 7.0, es throws an exception.
If an index was created before 7.0, es issue deprecation log
We will remove it in 8.0

Related #4704
2019-01-09 12:42:00 +09:00
Daniel Mitterdorfer 75f3443c62
Rename setting to enable mmap
With this commit we rename `node.store.allow_mmapfs` to
`node.store.allow_mmap`. Previously this setting has controlled whether
`mmapfs` could be used as a store type. With the introduction of
`hybridfs` which also relies on memory-mapping,
`node.store.allow_mmapfs` also applies to `hybridfs` and thus we rename
it in order to convey that it is actually used to allow memory-mapping
but not a specific store type.

Relates #36668
Relates #37070
2019-01-03 07:10:34 +01:00
Damien Alexandre 47f0a47f3d [Docs] Fix badly escaped keywords in migrate_7_0/search.asciidoc (#36964) 2018-12-27 09:18:28 +01:00
lcawl 32bed098bb [DOCS] Synchs titles of X-Pack APIs 2018-12-20 10:27:24 -08:00
David Turner 1a23417aeb
[Zen2] Update documentation for Zen2 (#34714)
This commit overhauls the documentation of discovery and cluster coordination,
removing mention of the Zen Discovery module and replacing it with docs for the
new cluster coordination mechanism introduced in 7.0.

Relates #32006
2018-12-20 13:02:44 +00:00
Lisa Cawley 4140b9eede
[DOCS] Update X-Pack terminology in security docs (#36564) 2018-12-19 14:53:37 -08:00
Nicholas Knize 9b1534b79b Update breaking changes doc with changes to geo_shape defaults
This commit updates the breaking changes doc to highlight the
impact of changing geo_shape defaults on timed index creation.
2018-12-19 10:12:31 -06:00
Nick Knize ec0dc2c0e9
[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#36751)
* [Geo] Expose BKDBackedGeoShapes as new VECTOR strategy

This commit exposes lucene's LatLonShape field as a new
strategy in GeoShapeFieldMapper. To use the new indexing
approach, strategy should be set to "vector" in the
geo_shape field mapper. If the tree parameter is set
the mapper will throw an IAE. Note the following:

When using vector strategy:

* geo_shape query does not support querying by POINT,
MULTIPOINT, or GEOMETRYCOLLECTION.
* LINESTRING and MULTILINESTRING queries do not support
WITHIN relation.
* CONTAINS relation is not supported.
* The tree, precision, tree_levels, distance_error_pct,
and points_only parameters will not throw an exception
but they have no effect and will be marked as
deprecated..

All other features are supported.

* revert change to PercolatorFieldMapper

* fix ExistsQuery for geo_shape vector strategy

* add deprecation logging for tree, precision, tree_levels, distance_error_pct, and points_only

* initial update to geoshape docs, including mapping migration updates

* initial support for GeoCollection queries

* fix docs and javadoc errors

* clean up geocollection queries

* set deprecated mapping tests to NOTCONSOLE

* fix geo-shape mapper asciidoc mapping and test warnings

* add support for point queries using LatLonShapeBoundingBoxQuery

* update GeoShapeQueryBuilderTests to include POINT queries for VECTOR strategy. Other comment cleanups

* add lucene geometry build testing to ShapeBuilder tests

* remove deprecated prefix tree mapping from geo-shape.asciidoc

* refactor GeoShapeFieldMapper into LegacyGeoShapeFieldMapper and GeoShapeFieldMapper

Both classes derive from BaseGeoShapeFieldMapper that provides shared parameters:
coerce, ignoreMalformed, ignore_z_value, orientation.

* update docs to remove vector strategy

* fix GeometryCollectionBuilder#buildLucene to return the object created by the shape builder

* fix LineLength failure in GeoJsonShapeParserTests

* ShapeMapper refactor changes from PR feedback

* fix typo in geo-shape.asciidoc

* ignore circle test in docs

* update indexing-approach ref to geoshape-indexing-approach

* add warnings check for LegacyGeoShapeFieldMapper to AbstractBuilderTestCase

* fix deprecatedParameters setup

* update indexing approach

* fixing unexpected warnings failures

* move orientation back to field type

* remove if in LegacyGeoShapeFieldMapper#doXContent. Fix GeoShapeFieldMapper to work with double array as a point

* fix indexing-approach link in circle section of geoshape docs

* add strategy to deprecation warnings check

* fix test failures

* fix typo in QueryStringQueryBuilderTests

* fix total hits to totalHits().value

* fix version number

* add version check to BaseGeoShapeFieldMapper

* fix line length!

* revert version check in BaseGeoShapeFieldMapper

* Fix serialization of mappings of legacy shapes.
2018-12-18 09:54:56 -06:00
Nicholas Knize 96d279ed83 Revert "[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#35320)"
This reverts commit 5bc7822562.
2018-12-17 20:09:46 -06:00
Nick Knize 5bc7822562
[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#35320)
This commit  exposes lucene's LatLonShape field as the
default type in GeoShapeFieldMapper. To use the new 
indexing approach, simply set "type" : "geo_shape" in 
the mappings without setting any of the strategy, precision, 
tree_levels, or distance_error_pct parameters. Note the 
following when using the new indexing approach:

* geo_shape query does not support querying by 
MULTIPOINT.
* LINESTRING and MULTILINESTRING queries do not 
yet support WITHIN relation.
* CONTAINS relation is not yet supported.
The tree, precision, tree_levels, distance_error_pct, 
and points_only parameters are deprecated.
2018-12-17 14:38:14 -06:00
Jeff Hajewski f1f3b28f5c Delete deprecated getValues from ScriptDocValues (#36183)
* Adds deprecation logging to ScriptDocValues#getValues.

First commit addressing issue #22919.

`ScriptDocValues#getValues` was added for backwards compatibility but no
longer needed. Scripts using the syntax `doc['foo'].values` when
`doc['foo']` is a list should be using `doc['foo']` instead.

* Fixes two build errors in #34279

* Removes unused import in ScriptDocValuesDatesTest
* Removes used of `.values` in example in diversified-sampler-aggregation.asciidoc

* Removes use of .values from painless test.

Part of #34279

* Updates tests to use `doc[foo]` syntax rather than `doc[foo].values`.

* Removes use of `getValues()` and replaces use of `doc[foo].values` with `doc[foo]`.

* Indentation fix.

* Remove unnecessary list construction at previous `getValues()` callsite in ScriptDocValues.GeoPoints.

* Update migration doc and add link to `getValue` in ScriptDocValues javadoc.

* Fix compile

* Fix javadoc issue

* Removes ScriptDocValues#getValues usage from painless whitelist.
2018-12-14 07:56:47 -05:00