Commit Graph

821 Commits

Author SHA1 Message Date
Yannick Welsch b2b01dc6de
Properly document keepalive and other tcp options (#60216)
Keepalive options are not well-documented (only in transport section, although also available at http and network level).

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-07-28 10:14:56 +02:00
Julie Tibshirani 8a89d95372
Add search `fields` parameter to support high-level field retrieval. (#60100)
This feature adds a new `fields` parameter to the search request, which
consults both the document `_source` and the mappings to fetch fields in a
consistent way. The PR merges the `field-retrieval` feature branch.

Addresses #49028 and #55363.
2020-07-27 13:25:55 -07:00
James Rodewig 441c3a21b1
[DOCS] Update my-index examples (#60132)
Changes the following example index names to `my-index-000001` for consistency:

* `my-index`
* `my_index`
* `myindex`
2020-07-27 14:46:39 -04:00
Yannick Welsch 2078509af5
Set specific keepalive options by default on supported platforms (#59278)
keepalives tell any intermediate devices that the connection remains alive, which helps with overzealous firewalls that are
killing idle connections. keepalives are enabled by default in Elasticsearch, but use system defaults for their
configuration, which often times do not have reasonable defaults (e.g. 7200s for TCP_KEEP_IDLE) in the context of
distributed systems such as Elasticsearch.

This PR sets the socket-level keep_alive options for network.tcp.{keep_idle,keep_interval} to 5 minutes on configurations
that support it (>= Java 11 & (MacOS || Linux)) and where the system defaults are set to something higher than 5
minutes. This helps keep the connections alive while not interfering with system defaults or user-specified settings
unless they are deemed to be set too high by providing better out-of-the-box defaults.
2020-07-27 13:38:24 +02:00
James Rodewig 2774cd6938
[DOCS] Swap `[float]` for `[discrete]` (#60124)
Changes instances of `[float]` in our docs for `[discrete]`.

Asciidoctor prefers the `[discrete]` tag for floating headings:
https://asciidoctor.org/docs/asciidoc-asciidoctor-diffs/#blocks
2020-07-23 11:48:22 -04:00
Nik Everett f87b31f973
Document supported scenarios for CCS (#58120)
Documents the supported scenarios for CCS.

Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-07-20 10:03:08 -04:00
Rory Hunter 4db094c008
Remove dangling index auto import functionality (#59698)
Closes #48366. Remove all traces of automatically importing dangling indices. This functionality is
deprecated from 7.9.0.
2020-07-17 15:17:58 +01:00
Tim Brooks e05858132d
Update thread pool docs about WRITE queue size (#59643)
This commit updates the thread pool documentation to reflect the change
in the WRITE thread pool default queue size.
2020-07-16 09:32:51 -06:00
debadair 5d7271a910
Update node.asciidoc (#59201) (#59480)
TIP block was missing due to the lack of line break prior to the "TIP"

Co-authored-by: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com>
2020-07-13 16:50:54 -07:00
Nhat Nguyen 961db311f0
Sending operations concurrently in peer recovery (#58018)
Today, we send operations in phase2 of peer recoveries batch by batch 
sequentially. Normally that's okay as we should have a fairly small of
operations in phase 2 due to the file-based threshold. However, if
phase1 takes a lot of time and we are actively indexing, then phase2 can
have a lot of operations to replay.

With this change, we will send multiple batches concurrently (defaults 
to 1) to reduce the recovery time.
2020-07-07 18:00:03 -04:00
Lisa Cawley fca1ece693
[DOCS] Clarifies transform node settings (#59023) 2020-07-07 13:31:04 -07:00
David Turner c661a40083
Add docs for filesystem health checks (#59134)
Documents the feature and settings introduced in #52680.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-07-07 14:14:35 +01:00
Rory Hunter 3f51fedfe6
Dangling indices documentation (#58751)
Part of #48366. Add documentation for the dangling indices
API added in #58176.

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-07-03 16:28:32 +01:00
Adam Locke 3a1258fe97
[DOCS] Add supported ESS settings to ES docs (#57953)
* Adding ESS icons to supported ES settings.

* Adding new file for supported ESS settings.

* Adding supported ESS settings for HTTP and disk-based shard allocation.

* Adding more supported settings for ESS.

* Adding descriptions for each Cloud section, plus additional settings.

* Adding new warehouse file for Cloud, plus additional settings.

* Adding node settings for Cloud.

* Adding audit settings for Cloud.

* Resolving merge conflict.

* Adding SAML settings (part 1).

* Adding SAML realm encryption and signing settings.

* Adding SAML SSL settings.

* Adding Kerberos realm settings.

* Adding OpenID Connect Realm settings.

* Adding OpenID Connect SSL settings.

* Resolving leftover Git merge markers.

* Removing Cloud settings page and link to it.

* Add link to mapping source

* Update docs/reference/docs/reindex.asciidoc

* Incorporate edit of HTTP settings

* Remove "cloud" from tag and ID

* Remove "cloud" from tag and update description

* Remove "cloud" from tag and ID

* Change "whitelists" to "specifies"

* Remove "cloud" from end tag

* Removing cloud from IDs and tags.

* Changing link reference to fix build issue.

* Adding index management page for missing settings.

* Removing warehouse file for Cloud and moving settings elsewhere.

* Clarifying true/false usage of http.detailed_errors.enabled.

* Changing underscore to dash in link to fix ci build.
2020-07-02 14:13:06 -04:00
David Turner acf031cdb5
Forbid read-only-allow-delete block in blocks API (#58727)
* Forbid read-only-allow-delete block in blocks API

The read-only-allow-delete block is not really under the user's control
since Elasticsearch adds/removes it automatically. This commit removes
support for it from the new API for adding blocks to indices that was
introduced in #58094.

* Missing xref

* Reword paragraph on read-only-allow-delete block
2020-07-01 12:57:34 +01:00
David Turner 83d6589b2a
Account for remaining recovery in disk allocator (#58029)
Today the disk-based shard allocator accounts for incoming shards by
subtracting the estimated size of the incoming shard from the free space on the
node. This is an overly conservative estimate if the incoming shard has almost
finished its recovery since in that case it is already consuming most of the
disk space it needs.

This change adds to the shard stats a measure of how much larger each store is
expected to grow, computed from the ongoing recovery, and uses this to account
for the disk usage of incoming shards more accurately.
2020-07-01 08:04:45 +01:00
Yannick Welsch 118521d022
Account for recovery throttling when restoring snapshot (#58658)
Restoring from a snapshot (which is a particular form of recovery) does not currently take recovery throttling into account
(i.e. the `indices.recovery.max_bytes_per_sec` setting). While restores are subject to their own throttling (repository
setting `max_restore_bytes_per_sec`), this repository setting does not allow for values to be configured differently on a
per-node basis. As restores are very similar in nature to peer recoveries (streaming bytes to the node), it makes sense to
configure throttling in a single place.

The `max_restore_bytes_per_sec` setting is also changed to default to unlimited now, whereas previously it was set to
`40mb`, which is the current default of `indices.recovery.max_bytes_per_sec`). This means that no behavioral change
will be observed by clusters where the recovery and restore settings were not adapted.

Relates https://github.com/elastic/elasticsearch/issues/57023

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-06-30 13:08:21 +02:00
Jason Tedor a914d84429
Introduce node.roles setting (#54998)
Today we have individual settings for configuring node roles such as
node.data and node.master. Additionally, roles are pluggable and we have
used this to introduce roles such as node.ml and node.voting_only. As
the number of roles is growing, managing these becomes harder for the
user. For example, to create a master-only node, today a user has to
configure:
 - node.data: false
 - node.ingest: false
 - node.remote_cluster_client: false
 - node.ml: false

at a minimum if they are relying on defaults, but also add:
 - node.master: true
 - node.transform: false
 - node.voting_only: false

If they want to be explicit. This is also challenging in cases where a
user wants to have configure a coordinating-only node which requires
disabling all roles, a list which we are adding to, requiring the user
to keep checking whether a node has acquired any of these roles.

This commit addresses this by adding a list setting node.roles for which
a user has explicit control over the list of roles that a node has. If
the setting is configured, the node has exactly the roles in the list,
and not any additional roles. This means to configure a master-only
node, the setting is merely 'node.roles: [master]', and to configure a
coordinating-only node, the setting is merely: 'node.roles: []'.

With this change we deprecate the existing 'node.*' settings such as
'node.data'.
2020-06-24 14:46:31 -04:00
bellengao 0fb6650146
[DOCS] Fix typo in remote-clusters doc (#57820) 2020-06-09 11:28:02 +02:00
James Rodewig 89e35b4079 [DOCS] Fix heading capitalization in CCS docs 2020-06-02 11:11:28 -04:00
James Rodewig b5e3a998a8 [DOCS] Remove unneeded word from network settings docs 2020-06-01 11:09:17 -04:00
Adam Locke 7dd731b9a2
[DOCS] Explain flood stage watermark. (#57184)
* Changes for issue #36114.

* Adding stronger wording to the new note.

* Removing statement about typically not needting to set the read-only allow delete block.

* Replacing Elasticsearch with {es} variable.
2020-05-28 10:57:40 -04:00
Nick Canzoneri 8d70f0696f
[DOCS] Fix remote cluster client node docs (#57176)
Changes:

* Documents the `r` role in the _cat/nodes API docs

* Fixes a typo in the node docs
2020-05-27 08:41:45 -04:00
James Rodewig 1bfd7efb44
[DOCS] Fix default for `http.compression` setting (#56899)
Elasticsearch enables HTTP compression by default. However, to mitigate
potential security risks like the BREACH attack, compression is disabled by
default if HTTPS is enabled.

This updates the `http.compression` setting definition accordingly and adds
additional context.

Co-authored-by: Leaf-Lin <39002973+Leaf-Lin@users.noreply.github.com>
2020-05-20 12:11:49 -04:00
James Rodewig 70cb519aa7
[DOCS] Relocate discovery module content (#56611)
* Moves `Discovery and cluster formation` content from `Modules` to
`Set up Elasticsearch`.

* Combines `Adding and removing nodes` with `Adding nodes to your
  cluster`. Adds related redirect.

* Removes and redirects the `Modules` page.

* Rewrites parts of `Discovery and cluster formation` to remove `module`
  references and meta references to the section.
2020-05-12 17:39:06 -04:00
James Rodewig 6ef9506beb
[DOCS] Correct setting type for `indices.query.bool.max_clause_count` (#56640)
#56449 incorrectly labelled this as a dynamic setting.

This corrects that error.
2020-05-12 16:25:41 -04:00
James Rodewig 7c449319a1
[DOCS] Relocate `shard allocation` module content (#56535) 2020-05-12 08:55:57 -04:00
James Rodewig af2d13144f
[DOCS] Add reference docs for `search.max_buckets` setting (#56449)
Adds reference-style setting documentation for the `search.max_buckets`
setting.

This setting was previously only documented on the [bucket
aggregations][0] page.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggregations-bucket.html
2020-05-11 08:35:24 -04:00
James Rodewig 4e81fb8833
[DOCS] Relocate `local gateway` setting content (#56448)
Moves `local gateway` setting content to `Configuring Elasticsearch`
from `Modules`.

Relates to #53303.
2020-05-11 08:34:55 -04:00
James Rodewig 12440185c6
[DOCS] Relocate `HTTP` module content (#56386)
Moves `HTTP` content from the [Modules][0] section to the
[Configuring Elasticsearch][1] section.

Supporting changes:
* Replaces `http` with `HTTP` throughout
* Replaces `HTTP module` with `HTTP layer` throughout
* Increments several headings
* Adds explicit anchors to several headings
* Removes several unneeded `[float]` attributes

Closes #53306

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
2020-05-08 13:11:21 -04:00
James Rodewig fabb3e18b9
[DOCS] Relocate `node` module content (#56376)
Moves `node` content from the [Modules][0] section to the
[Configuring Elasticsearch][1] section.

Supporting changes:
* Retitles page `Node settings`
* Increments several headings
* Removes several unneeded `[float]` attributes
* Replaces remaining `[float]` attributes with `[discrete]`
* Updates the `Other node settings` section

Relates to #53307

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
2020-05-08 08:24:00 -04:00
James Rodewig f26f1021e8
[DOCS] Relocate thread pools content (#55814)
Moves [thread pools content][0] from [Modules][1] to
[Configuring Elasticsearch][2].

Supporting changes:
* Changes page title to "Thread pools"
* Increments several headings
* Removes several unneeded `[float]` attributes
* Updates the anchors of several headings

Relates to #53307

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-threadpool.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[2]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html

Co-authored-by: debadair <debadair@elastic.co>

Co-authored-by: debadair <debadair@elastic.co>
2020-05-05 09:48:36 -04:00
James Rodewig a73fef3d62
[DOCS] Create top-level "Search your data" page (#56058)
* [DOCS] Create top-level "Search your data" page

**Goal**

Create a top-level search section. This will let us clean up our search
API reference docs, particularly content from [`Request body search`][0].

**Changes**

* Creates a top-level `Search your data` page. This page is designed to
  house concept and tutorial docs related to search.

* Creates a `Run a search` page under `Search your data`. For now, This
  contains a basic search tutorial. The goal is to add content from
  [`Request body search`][0] to this in the future.

* Relocates `Long-running searches` and `Search across clusters` under
  `Search your data`. Increments several headings in that content.

* Reorders the top-level TOC to move `Search your data` higher. Also
  moves the `Query DSL`, `EQL`, and `SQL access` chapters immediately
  after.

Relates to #48194

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/search-request-body.html
2020-05-05 09:46:29 -04:00
James Rodewig 3dd0aa3229
[DOCS] Relocate network settings content (#55865)
Moves [network settings content][0] from [Modules][1] to the
[Configuring Elasticsearch][2] section.

Supporting changes:
* Changes page title and headings to sentence case
* Increments several headings
* Removes several unneeded `[float]` and `[horizontal]` attributes
* Removes several unneeded blank links in definition lists

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-network.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[2]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
2020-05-05 09:31:45 -04:00
William Brafford f36ab092ed
Deprecated xpack "enable" settings should be no-ops (#55416)
The following settings are now no-ops:

* xpack.flattened.enabled
* xpack.logstash.enabled
* xpack.rollup.enabled
* xpack.slm.enabled
* xpack.sql.enabled
* xpack.transform.enabled
* xpack.vectors.enabled

Since these settings no longer need to be checked, we can remove settings
parameters from a number of constructors and methods, and do so in this
commit.

We also update documentation to remove references to these settings.
2020-05-04 16:49:11 -04:00
Henning Andersen 0bd28aed4e
Disk decider respect watermarks for single data node (#55805)
The disk decider had special handling for the single data node case,
allowing any allocation (skipping watermark checks) for such clusters.
This special handling can now be avoided via a setting.
2020-04-28 11:55:42 +02:00
Stuart Tettemer bd64da0960
Scripting: Deprecate general cache settings (#55038)
* Scripting: Deprecate general cache settings

* Add script.disable_max_compilations_rate setting

* Move construction to ScriptCache

* Use ScriptService to do updates of CacheHolder

* Remove fallbacks

* Add SCRIPT_DISABLE_MAX_COMPILATIONS_RATE_SETTING to ClusterSettings

* Node scope

* Use back compat

* 8.0 for bwc

* script.max_compilations_rate=2048/1m -> script.disable_max_compilations_rate=true in docker compose

* do not guard in esnode

* Doc update

* isSnapshotBuild() -> systemProperty 'es.script.disable_max_compilations_rate', 'true'

* Do not use snapshot in gradle to set max_compilations_rate

* Expose cacheHolder as package private

* monospace 75/5m in cbreaker docs, single space in using

* More detail in general compilation rate error

* Test: don't modify defaultConfig on upgrade
2020-04-22 12:33:33 -06:00
James Rodewig 58b1b33e7b
[DOCS] Relocate `transport` module content (#55472)
Moves `transport` content from the [Modules][0] section to the
[Configuring Elasticsearch][1] section.

Supporting changes:
* Adds explicit anchors to several headings
* Changes several headings to sentence case
* Increments several headings
* Removes several unneeded `[float]` attributes
* Replaces `transport module` with `transport layer`

Relates to #53305

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
2020-04-21 09:43:41 -04:00
David Turner 10ab397d7f
Adjust docs for voting config exclusions API (#55006)
In #50836 we deprecated the existing voting config exclusions API and added a
new one. This commit adjust the docs to match.
2020-04-20 19:47:09 +01:00
William Brafford 92c8a73348
Deprecate disabling basic-license features (#54816)
We believe there's no longer a need to be able to disable basic-license
features completely using the "xpack.*.enabled" settings. If users don't
want to use those features, they simply don't need to use them. Having
such features always available lets us build more complex features that
assume basic-license features are present.

This commit deprecates settings of the form "xpack.*.enabled" for
basic-license features, excluding "security", which is a special case.
It also removes deprecated settings from integration tests and unit
tests where they're not directly relevant; e.g. monitoring and ILM are
no longer disabled in many integration tests.
2020-04-17 09:18:55 -04:00
James Rodewig 44a643c0e5
[DOCS] Relocate "Remote Clusters" page (#55172)
Relocates the "Remote Clusters" documentation from the "Modules" section to the "Set up Elasticsearch" section.

Supporting changes:
* Reorders the "Bootstrap checks for X-Pack" section to immediately follow the "Bootstrap checks"chapter.
* Removes an outdated X-Pack `idef` from the "Remote Clusters" intro.
2020-04-15 10:11:30 -04:00
James Rodewig 4980ea7596
[DOCS] Document `max_concurrent_searches` default (#55116) 2020-04-15 10:02:33 -04:00
James Rodewig 600ccc0a51
[DOCS] Relocate "Plugins" page (#54974) 2020-04-10 15:29:44 -04:00
Jason Tedor cf87ed3754
Passthrough special characters in thread pool docs (#55080)
Some of these characters are special to Asciidoctor and they ruin the
rendering on this page. Instead, we use a macro to passthrough these
characters without Asciidoctor applying any subtitutions to them. This
commit then addresses some rendering issues in the thread pool docs.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-04-10 15:09:27 -04:00
James Rodewig 5cbd05bb10
[DOCS] Relocate `indices` module content (#54903)
Moves `indices` content from the [Modules][0] section to the [Configuring
Elasticsearch][1] section.

Also removes the [Indices][2] landing page and adds a related redirect.

[0]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules.html
[1]: https://www.elastic.co/guide/en/elasticsearch/reference/master/settings.html
[2]: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-indices.html
2020-04-10 12:00:02 -04:00
Jason Tedor a0cb977f23
Clarify available processors (#54907)
The use of available processors, the terminology, and the settings
around it have evolved over time. This commit cleans up some places in
the codes and in the docs to adjust to the current terminology.
2020-04-10 08:38:00 -04:00
Jason Tedor 2bc699679e
Update docs to reflect node.processors (#54855)
We namespaced the previous setting "processors" into
"node.processors". This commit updates some of the documentation to
reflect this.
2020-04-07 13:05:23 -04:00
Lisa Cawley f9cb30963a
[DOCS] Adds tranform node to list of default types (#54850) 2020-04-07 08:35:08 -07:00
lcawl 2dc75ddf76 [DOCS] Fixes typo in node settings 2020-04-02 16:05:39 -07:00
Lisa Cawley 5b85013976
[DOCS] Clarify ML and transform settings on coordinating nodes (#54676) 2020-04-02 15:34:37 -07:00
David Turner ac1b6eb5e9
indices.recovery.max_bytes_per_sec may be per-node (#54633)
The `indices.recovery.max_bytes_per_sec` recovery bandwidth limit can differ
between nodes if it is not set dynamically, but today this is not obvious. This
commit adds a paragraph to its documentation clarifying how to set different
bandwidth limits on each node.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-04-02 18:14:34 +01:00
Jason Tedor 1fc0432b24
Introduce formal role for remote cluster client (#53924)
This commit introduce a formal role for identifying nodes that are
capable of making connections to remote clusters.
2020-03-24 19:21:56 -04:00
Lisa Cawley 35ad63036e
[DOCS] Add transform nodes (#53698) 2020-03-18 15:23:41 -07:00
Tianlun Li fe5092ae24
Deprecate delaying state recovery for master nodes (#53646)
It is useful to be able to delay state recovery until enough data nodes have
joined the cluster, since this gives the shard allocator a decent opportunity
to re-use as much existing data as possible. However we also have the option to
delay state recovery until a certain number of master-eligible nodes have
joined, and this is unnecessary: we require a majority of master-eligible nodes
for state recovery, and there is no advantage in waiting for more.

This commit deprecates the unnecessary settings in preparation for their
removal.

Relates #51806
2020-03-18 10:03:21 +00:00
Tim Brooks abd8a36f9b
Add documentation for remote cluster proxy mode (#52779)
This is related to #49067.
2020-03-09 10:49:41 -06:00
David Turner e2cda1a279
"Adding nodes" instructions only work on localhost (#52677)
The introductory sections of the reference manual contains some simplified
instructions for adding a node to the cluster. Unfortunately they are a little
too simplified and only really work for clusters running on `localhost`. If you
try and follow these instructions for a distributed cluster then the new node
will, confusingly, auto-bootstrap itself into a distinct one-node cluster.

Multiple nodes running on localhost is a valid config, of course, but we should
spell out that these instructions are really only for experimentation and that
it takes a bit more work to add nodes to a distributed cluster. This commit
does so.

Also, the "important config" instructions for discovery say that you MUST set
`discovery.seed_hosts` whereas in fact it is fine to ignore this setting and
use a dynamic discovery mechanism instead. This commit weakens this statement
and links to the docs for dynamic discovery mechanisms.

Finally, this section is also overloaded with some technical details that are
not important for this context and are adequately covered elsewhere, and
completely fails to note that the default discovery port is 9300. This commit
addresses this.
2020-02-27 08:51:17 +00:00
James Rodewig 841d961b58
[DOCS] Document CCS-supported APIs (#52708)
Explicitly notes the Elasticsearch API endpoints that support CCS.

This should deter users from attempting to use CCS with other API
endpoints, such as `GET <index>/_doc/<_id>`.
2020-02-24 09:54:33 -05:00
James Rodewig 67f6840846
[DOCS] Document how CCS handles cluster-level settings (#49941)
Updates the cross-cluster search (CCS) documentation to note how
cluster-level settings are applied.

When `ccs_minimize_roundtrips` is `true`, each cluster applies its own
cluster-level settings to the request.

When `ccs_minimize_roundtrips` is `false`, cluster-level settings for
the local cluster is used. This includes shard limit settings, such as
`action.search.shard_count.limit`, `pre_filter_shard_size`, and
`max_concurrent_shard_requests`. If these limits are set too low, the
request could be rejected.
2020-02-19 09:14:22 -05:00
Yannick Welsch a9afdd7611
Remove fixed_auto_queue_size threadpool type (#52280)
* Remove fixed_auto_queue_size threadpool type

* Remove less

* compilation fix

* weaken assertion to accomodate tests that mock threadpool
2020-02-14 16:20:40 +01:00
David Turner a304d9a656
Ignore timeouts with single-node discovery (#52159)
Today we use `cluster.join.timeout` to prevent nodes from waiting indefinitely
if joining a faulty master that is too slow to respond, and
`cluster.publish.timeout` to allow a faulty master to detect that it is unable
to publish its cluster state updates in a timely fashion. If these timeouts
occur then the node restarts the discovery process in an attempt to find a
healthier master.

In the special case of `discovery.type: single-node` there is no point in
looking for another healthier master since the single node in the cluster is
all we've got. This commit suppresses these timeouts and instead lets the node
wait for joins and publications to succeed no matter how long this might take.
2020-02-11 14:00:06 +00:00
Armin Braun 26b9cf787d
Add Trace Logging of REST Requests (#51684)
Being able to trace log all REST requests to a node would make debugging
a number of issues a lot easier.
2020-02-06 20:05:03 +01:00
István Zoltán Szabó 850278c69a
[DOCS] Adds recommendation on dedicated master-eligible nodes (#51674)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-01-31 12:51:46 +01:00
István Zoltán Szabó 451eb1fa1f
[DOCS] Expands the documentation of Node Query Cache (#51105)
Co-authored-by: debadair <debadair@elastic.co>
2020-01-20 11:11:57 +01:00
Nhat Nguyen 09b46c8646
Goodbye and thank you synced flush! (#50882)
Synced flush was a brilliant idea. It supports instant recoveries with a 
quite small implementation. However, with the presence of sequence
numbers and retention leases, it is no longer needed. This change
removes it from 8.0.

Relates #5077
2020-01-16 09:43:07 -05:00
debadair a3b851e9b9
[DOCS] Move snapshot-restore out of modules. (#49618)
* [DOCS] Move snapshot-restore docs out of modules.

* [DOCS] Incorporates comments from @jrodewig.

* [DOCS] Fix snippet tests
2020-01-09 16:12:02 -08:00
Stuart Tettemer fb6ef69c6b
[DOCS] Deterministic scripted queries are cached (#50408)
Refs: #49321
2019-12-19 16:16:57 -07:00
Lisa Cawley 362ce41eaf
[DOCS] Updates ML links (#50387) 2019-12-19 14:47:28 -08:00
Patryk Krawaczyński de4f701a19 [DOCS] Document `index.queries.cache.enabled` as a static setting (#49886) 2019-12-10 14:23:14 -05:00
James Rodewig 1a574115c1
[DOCS] Document CCR compatibility requirements (#49776)
* Creates a prerequisites section in the cross-cluster replication (CCR)
  overview.
* Adds concise definitions for local and remote cluster in a CCR context.
* Documents that the ES version of the local cluster must be the same
  or a newer compatible version as the remote cluster.
2019-12-02 15:52:13 -05:00
David Turner 69e0b1a0f4
Drop snapshot instructions for autobootstrap fix (#49755)
The "Restore any snapshots as required" step is a trap: it's somewhere between
tricky and impossible to restore multiple clusters into a single one.

Also add a note about configuring discovery during a rolling upgrade to
proscribe any rare cases where you might accidentally autobootstrap during the
upgrade.
2019-12-02 12:43:18 +00:00
István Zoltán Szabó 56888ff194
[DOCS] Removes the default size definition of thread pool types (#49442)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-11-22 11:15:35 +01:00
James Rodewig eca600326f
[DOCS] Document several missing thread pools (#48543)
Adds documentation for the following thread pools:
    - fetch_shard_started
    - fetch_shard_store
    - flush
    - force_merge
    - management

Closes #48524

Co-Authored-By: Jay Modi <jaymode@users.noreply.github.com>
2019-11-21 13:05:53 -05:00
James Rodewig 4db330d9e9
[DOCS] Replace cross-cluster search PNG images with SVGs (#49395) 2019-11-21 09:05:33 -05:00
weizijun 22042cc199 Document all shard allocation filtering attributes (#46992)
This commit adds coverage to the docs for some missing built-in shard
allocation attributes.
2019-11-21 08:29:45 -05:00
SylvainJuge 7072941577
[DOCS] minor fix to documentation: http.host can't default to itself (#48135)
fix minor typos on http.host and transport.host default values.
2019-11-14 15:56:13 +01:00
glerb dd47cf4560 [DOCS] Correct typo in Discovery docs (#48494) 2019-11-05 08:48:20 -05:00
Jason Tedor db015555e1
Fix specification for cluster.remote.connect (#48690)
The docs specify that cluster.remote.connect disables cross-cluster
search. This is correct, but not fully accurate as it disables any
functionality that relies on remote cluster connections: cross-cluster
search, remote data feeds, and cross-cluster replication. This commit
updates the docs to reflect this.
2019-10-30 11:25:27 -04:00
Ian Danforth aa0eb006d2 [Doc] Fix typo in indices module docs (#48598) 2019-10-28 21:40:52 +01:00
James Rodewig f4ac711d17
[DOCS] Add 'Selecting gateway and seed nodes' section to CCS docs (#48297) 2019-10-21 12:13:44 -04:00
François-Clément Brossard 0b107a0a09 Clarify low watermark documentation (#48112)
Today the docs say that the low watermark has no effect on any shards that have
never been allocated, but this is confusing. Here "shard" means "replication
group" not "shard copy" but this conflicts with the "never been allocated"
qualifier since one allocates shard copies and not replication groups.

This commit removes the misleading words. A newly-created replication group
remains newly-created until one of its copies is assigned, which might be quite
some time later, but it seems better to leave this implicit.
2019-10-16 12:27:39 +01:00
David Turner 9e30a57ca5
More bootstrap docs tweaks (#47809)
Clarifies not to set `cluster.initial_master_nodes` on nodes that are joining
an existing cluster.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-10-10 10:53:27 +02:00
David Turner 7b652adfbf
Remove include_relocations setting (#47717)
Setting `cluster.routing.allocation.disk.include_relocations` to `false` is a
bad idea since it will lead to the kinds of overshoot that were otherwise fixed
in #46079. This setting was deprecated in #47443. This commit removes it.
2019-10-08 13:33:49 +02:00
David Turner 9d67a02a56
Deprecate include_relocations setting (#47443)
Setting `cluster.routing.allocation.disk.include_relocations` to `false` is a
bad idea since it will lead to the kinds of overshoot that were otherwise fixed
in #46079. This commit deprecates this setting so it can be removed in the next
major release.
2019-10-08 09:15:13 +02:00
Lisa Cawley 4e4990c6a0
[DOCS] Cleans up links to security content (#47610) 2019-10-04 16:10:26 -07:00
James Rodewig 7583c07fa8
[DOCS] Reorder index APIs alphabetically (#46981) 2019-10-01 15:13:27 -04:00
Alan Woodward c1f99e2d75
Remove `_type` from SearchHit (#46942)
This commit removes the `_type` field from all search hit responses.

Relates to #41059
2019-09-23 19:14:54 +01:00
David Turner c01f58aac9
Remove docs for proxy mode (#46677)
We added docs for proxy mode in #40281 but on reflection we should not be
documenting this setting since it does not play well with all proxies and we
can't recommend its use. This commit removes those docs and expands its Javadoc
instead.
2019-09-13 22:17:03 +01:00
Peter Dyson 43719c6c6a [DOCS] Add missing mention of current version to snapshot docs (#46516) 2019-09-12 08:30:29 -04:00
David Turner a84908cebd Clarify that discovery ignores master-ineligibles (#44835)
The changes in #32006 mean that the discovery process can no longer use
master-ineligible nodes as a stepping-stone between master-eligible nodes.
This was normally an indication of a strange and possibly-fragile configuration
and was not recommended. This commit clarifies that only master-eligible nodes
are now involved with discovery.
2019-09-12 11:12:35 +01:00
James Rodewig 5c78f606c2
[DOCS] Change // CONSOLE comments to [source,console] (#46440) 2019-09-09 10:45:37 -04:00
James Rodewig e43be90e6c
[DOCS] [5 of 5] Change // TESTRESPONSE comments to [source,console-results] (#46449) 2019-09-06 14:05:36 -04:00
James Rodewig 466c59a4a7
[DOCS] Replace "// TESTRESPONSE" magic comments with "[source,console-result] (#46295) 2019-09-05 16:47:18 -04:00
Jim Ferenczi a4ed7b1ca1
Decouple shard allocation awareness from search and get requests (#45735)
With this commit, Elasticsearch will no longer prefer using shards in the same location
(with the same awareness attribute values) to process `_search` and `_get` requests.
Instead, adaptive replica selection (the default since 7.0) should route requests more efficiently
using the service time of prior inter-node communications. Clusters with big latencies between
nodes should switch to cross cluster replication to isolate nodes within the same zone.
Note that this change only targets 8.0 since it is considered as breaking. However a follow up
pr should add an option to activate this behavior in 7.x in order to allow users to opt-in early.

Closes #43453
2019-09-04 21:48:03 +02:00
Armin Braun df01766c15
Repository Cleanup Endpoint (#43900)
* Snapshot cleanup functionality via transport/REST endpoint.
* Added all the infrastructure for this with the HLRC and node client
* Made use of it in tests and resolved relevant TODO
* Added new `Custom` CS element that tracks the cleanup logic.
Kept it similar to the delete and in progress classes and gave it
some (for now) redundant way of handling multiple cleanups but only allow one
* Use the exact same mechanism used by deletes to have the combination
of CS entry and increment in repository state ID provide some
concurrency safety (the initial approach of just an entry in the CS
was not enough, we must increment the repository state ID to be safe
against concurrent modifications, otherwise we run the risk of "cleaning up"
blobs that just got created without noticing)
* Isolated the logic to the transport action class as much as I could.
It's not ideal, but we don't need to keep any state and do the same
for other repository operations
(like getting the detailed snapshot shard status)
2019-08-21 12:02:44 +02:00
James Rodewig 28107b2221
Retitle and relocate cross-cluster search docs (#45608) 2019-08-15 16:11:04 -04:00
James Rodewig f1661ab058
[DOCS] Rewrite cross-cluster seach docs (#45583) 2019-08-15 13:23:25 -04:00
James Rodewig 66b8261e1b
[DOCS] Add diagrams to cross-cluster search documentation (#45569) 2019-08-15 10:59:58 -04:00
Chris Dean 96a234e461
[DOCS] - Updating chunk_size values to fix size value notation. Chunksize41591 (#45552)
* changes to chunk_size #41591

* update to chunk size to include ` `

* Update docs/plugins/repository-azure.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/modules/snapshots.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/plugins/repository-azure.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Update docs/plugins/repository-s3.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* edits to fix passive voice
2019-08-14 13:47:07 -05:00
Chris Dean 7b21ee75a3
[DOCS] Added cross-link to snapshot lifecycle management. Closes #44588. (#45408) 2019-08-09 16:33:40 -05:00
David Turner bc31ea752e
Always auto-release the flood-stage block (#45274)
Removes support for using a system property to disable the automatic release of
the write block applied when a node exceeds the flood-stage watermark.

Relates #42559
2019-08-08 11:47:14 +01:00