Commit Graph

35 Commits

Author SHA1 Message Date
James Rodewig cf4fde33cf
[DOCS] Fix typo in mount searchable snapshots API docs (#75786) (#75984)
Co-authored-by: Luca Wintergerst <luca.wintergerst@elastic.co>
2021-08-02 14:46:43 -04:00
James Rodewig 5729bb8d49
[DOCS] Update alias references (#73427)
Updates several `index aliases` references to `aliases`.
2021-05-27 16:00:57 -04:00
James Rodewig ba66669eb3
[DOCS] Rename mount types for searchable snapshots (#72699)
Changes:

* Renames 'full copy searchable snapshot' to 'fully mounted index.'
* Renames 'shared cache searchable snapshot' to 'partially mounted index.'
* Removes some unneeded cache setup instructions for the frozen tier. We added a default cache size with #71844.
2021-05-05 16:35:33 -04:00
David Kilfoyle c1c5e67f37
[DOCS] Remove frozen tier restriction for ESS (#72624)
* Remove frozen tier restriction for ESS

* Remove section from 'Use ES for time series data'

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2021-05-03 15:39:14 -07:00
Tanguy Leroux f50b35719e
Revive documentation for experimental Searchable Snapshots APIs (#71927)
This commit revives the documentation of the "Clear Cache" and 
"Shard Stats" APIs of Searchable Snapshots that was removed 
in #62217. This is a partial revert of the commit b545c55 with 
some light wording changes.

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2021-04-20 18:34:48 +02:00
Henning Andersen c57fbe8eaf
Frozen default cache size (#71844)
This commit adds a default cache size to frozen tier of the greater of
90% and total disk size minus 100 GB.
2021-04-20 17:37:39 +02:00
Yannick Welsch 8c1938a524
Remove experimental status from frozen tier docs (#71923)
Removes the experimental status for the frozen tier / shared_cache searchable snapshots for the 7.13 release. 
Also adapts docs that URL repositories are now supported in 7.13 for searchable snapshots.
2021-04-20 15:37:56 +02:00
Tanguy Leroux ceaa16eddc
Add node level cache stats for searchable snapshots (#71701)
This commit adds node-level statistics about the searchable 
snapshots shared cache that can be retrieved using the REST 
endpoint `GET /_searchable_snapshots/cache/stats`.

And the returned informations are:
{
  "nodes" : {
    "eerrtBMtQEisohZzxBLUSw" : {
      "shared_cache" : {
        "reads" : 6051,
        "bytes_read" : "5.1mb",
        "bytes_read_in_bytes" : 5448829,
        "writes" : 37,
        "bytes_written" : "1.1mb",
        "bytes_written_in_bytes" : 1208320,
        "evictions" : 5,
        "num_regions" : 32,
        "size" : "1mb",
        "size_in_bytes" : 1048576,
        "region_size" : "32kb",
        "region_size_in_bytes" : 32768
      }
    }
  }
}
2021-04-19 17:02:27 +02:00
James Rodewig c23f001151
[DOCS] Add how-to guide for time series data (#71195) 2021-04-05 13:00:06 -04:00
James Rodewig d3c56e6fca [DOCS] Remove unneeded articles for Elasticsearch Service and Elastic Agent 2021-04-02 16:01:59 -04:00
Jason Tedor d340432622
Remove frozen cache setting leniency (#71013)
We previously allowed but deprecated the ability for the shared cache to
be positively sized on nodes without the frozen role. This is because we
only allocate shared_cache searchable snapshots to nodes with the frozen
role. This commit completes our intention to deprecate/remove this
ability.
2021-04-02 14:26:01 -04:00
James Rodewig 955df4337b
[DOCS] Fix typos for Elasticsearch Service and Elastic Agent (#71076) 2021-03-31 08:36:21 -04:00
James Rodewig 15f863813c [DOCS] Fix wording 2021-03-30 06:50:53 -04:00
David Turner dd69ae95d7
Note recovery settings affect searchable snapshots (#70771)
Adds a short note that `max_restore_bytes_per_sec` and
`indices.recovery.max_bytes_per_sec` also affect the recovery of a
searchable snapshot index.
2021-03-24 09:22:44 +00:00
Yannick Welsch f9aa282b7e
Additional docs for shared_cache searchable snapshots (#70566)
This adds additional documentation for shared_cache searchable snapshots that are targeting the frozen tier:

- it generalizes the introduction section on searchable snapshots, mentioning that they come in two flavors now 
as well as the relation to cold and frozen tiers,
- it expands the shared_cache section and
- it adds Cloud-specific instructions for getting started with the frozen tier

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: debadair <debadair@elastic.co>
Co-authored-by: David Turner <david.turner@elastic.co>
2021-03-22 16:42:58 +01:00
David Turner e88038575d
Document searchable snapshots supported repos (#69508)
Adds a note listing the repository types that can be used with
searchable snapshots.
2021-02-24 09:02:16 +00:00
David Turner 71763ca37f
Add docs on repository reliability and backups (#68740)
This commit spells out how important repository reliability is to
searchable snapshots, and also documents a procedure for taking a backup
of a snapshot repository.

Relates #54944
2021-02-09 14:57:06 +00:00
Yannick Welsch e97f233fbe
Remove beta label from searchable snapshot docs (#68724)
This feature will go GA in 7.11.
2021-02-09 09:28:03 +01:00
Yannick Welsch 50f4a0b77c
Add partial searchable snapshot support for a frozen tier (#68509)
A frozen tier is backed by an external object store (like S3) and caches only a
small portion of data on local disks. In this way, users can reduce hardware
costs substantially for infrequently accessed data. For the frozen tier we only
pull in the parts of the files that are actually needed to run a given search.
Further, we don't require the node to have enough space to host all the files.
We therefore have a cache that manages which file parts are available, and which
ones not. This node-level shared cache is bounded in size (typically in relation
to the disk size), and will evict items based on a LFU policy, as we expect some
parts of the Lucene files to be used more frequently than other parts. The level
of granularity for evictions is at the level of regions of a file, and does not
require evicting full files. The on-disk representation that was chosen for the
cold tier is not a good fit here, as it won't allow evicting parts of a file.
Instead we are using fixed-size pre-allocated files and have implemented our own
memory management logic to map regions of the shard's original Lucene files onto
regions in these node-level shared files that are representing the on-disk
cache.

This PR adds the core functionality to searchable snapshots to power such a
frozen tier:
- It adds the node-level shared cache that evicts file regions based on a LFU
  policy
- It adds the machinery to dynamically download file regions into this cache and
  serve their contents when searches execute.
- It extends the mount API with a new parameter, `storage`, which selects the
  kind of local storage used to accelerate searches of the mounted index. If set
  to `full_copy` (default, used for cold tier), each node holding a shard of the
  searchable snapshot index makes a full copy of the shard to its local storage.
  If set to `shared_cache`, the shard uses the newly introduced shared cache,
  only holding a partial copy of the index on disk (used for frozen tier).

Co-authored-by: Tanguy Leroux <tlrx.dev@gmail.com>
Co-authored-by: Armin Braun <me@obrown.io>
Co-authored-by: David Turner <david.turner@elastic.co>
2021-02-05 09:15:07 +01:00
James Rodewig 87105cb04a
[DOCS] Fix typo (#68446) 2021-02-03 09:02:48 -05:00
David Turner 4c8b75d208
Document taking snapshots of searchable snapshots (#68136)
Taking a snapshot of a cluster containing searchable snapshot indices is
kind of mindbending. This commit adds docs to indicate that this does
work.

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2021-02-01 18:15:34 +00:00
James Rodewig 44943e592a
[DOCS] Fix typo (#68085) (#68158)
Co-authored-by: Mykola Shestopal <nick4eva@users.noreply.github.com>
2021-01-28 16:46:10 -05:00
David Turner 68a733a347 Adjust resiliency docs for searchable snapshots (#67630)
Today we recommend every index to have at least one replica in our
guidelines for designing a resilient cluster. This advice does not apply
to searchable snapshot indices. This commit adjusts the resiliency docs
to account for this. It also slightly adjusts the wording in the
searchable snapshots docs to be more consistent about the distinction
between a "searchable snapshot" and a "searchable snapshot index".
2021-01-19 17:40:49 +00:00
bellengao e198bb233e
[DOCS] Correct the default value of `wait_for_completion` query param (#65800)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-12-04 15:52:35 -05:00
Henning Andersen 8fa1eea6f6
Searchable snapshot terminology (#65549)
We chose to use searchable snapshot index over snapshot-backed index, so
changed terminology towards this in a couple places.
2020-11-30 17:14:47 +01:00
Henning Andersen 302e5761fc
Clarify searchable snapshot cost trade-offs (#65384)
Clarify that searchable snapshots only result in cost savings for less
frequently accessed data and that the savings do not apply to the entire
cluster.
2020-11-26 13:40:08 +01:00
debadair b95d9c4aa4
[DOCS] Add searchable snapshots topic. (#63040)
* [DOCS] Add searchable snapshots topic.

* [DOCS] Add definitions & remove fully-remote storage.

* [DOCS] Fixed duplicate anchor.

* Expand conceptual docs for searchable snapshots

* Rewordings

* Glossary tidy-up

* Beta

* Reword

* More performance idea to a TIP

* use -> manage

* red -> not green

* Missing space?

* Update docs/reference/glossary.asciidoc

* Fix beta label

* Use more attributes, fix link titles

* Apply suggestions from code review

Co-authored-by: debadair <debadair@elastic.co>

* Reformat

* Minor rewordings

* More minor rewordings

* Address Henning's comments

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-22 14:34:32 -07:00
James Rodewig 1b0350bb80
[DOCS] Move searchable snapshots to beta (#63436) 2020-10-08 08:52:44 -04:00
Tanguy Leroux b545c55557
Remove REST APIs documentation for experimental Searchable Snapshot APIs (#62217)
This commit removes the documentation for some specific Searchable Snapshot REST APIs:
- clear cache
- searchable snapshot stats
- repository stats

These APIs are low-level and are useful to investigate the behavior of snapshot 
backed indices but we expect them to be removed in the future or to appear in 
a different form.
2020-09-10 15:46:41 +02:00
Yannick Welsch 64d68240f8
Adjust searchable snapshot license (#60578)
No longer needs Platinum license for testing on staging.
2020-08-03 13:18:06 +02:00
James Rodewig 80b674fb25
[DOCS] Reformat snippets to use two-space indents (#59973) 2020-07-21 12:24:26 -04:00
James Rodewig 9071c8298b
[DOCS] Add data streams to searchable snapshot API docs (#59325) 2020-07-13 08:45:36 -04:00
Lisa Cawley 8b9293b3bf
[DOCS] Replace docdir attribute with es-repo-dir (#57489) 2020-06-01 15:55:05 -07:00
Yannick Welsch 03a60769f3
Provide repository-level stats for searchable snapshots (#55051)
Provides basic repository-level stats that will allow us to get some insight into how many
requests are actually being made by the underlying SDK. Currently only tracks GET and LIST
calls for S3 repositories. Most of the code is unfortunately boiler plate to add a new endpoint
that will help us better understand some of the low-level dynamics of searchable snapshots.
2020-04-14 14:12:48 +02:00
Tanguy Leroux f6feb6c2c8
Merge feature/searchable-snapshots branch into master (#54803)
This commit merges the searchable-snapshots feature branch into master.
See #54803 for the complete list of squashed commits.

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
2020-04-06 15:51:05 +02:00