Commit Graph

60 Commits

Author SHA1 Message Date
Rory Hunter 3bec319e26
Change env var prefix in Docker settings support (#76192)
Closes #76148. Previously, in #74327, we added support for setting
Elasticsearch settings in the Docker image via environment variables.
However there are scenarios e.g in Kubernetes where it is easy to
accidentally define environment variables that the ES startup process
rejects.

Work around this by using the more explicit env var prefix
`ES_SETTING_`.
2021-08-09 09:47:29 +01:00
Rory Hunter d08b851a5b
Accept settings in snake case in Docker image (#74327)
Closes #74036. Since some orchestration platforms forbid periods in
environment variable names, allow Docker users to pass settings to ES
using an alternative name scheme. For example:

    bootstrap.memory_lock

...becomes:

    ES_BOOTSTRAP_MEMORY__LOCK

The setting name is uppercased, prefixed, all underscores are converted
to double underscores, and all periods are converted to underscores.
2021-07-09 19:46:58 +01:00
debadair bef9b8a9b3
[DOCS] Edit JVM settings info. Closes #72259 (#72350)
* [DOCS] Edit JVM settings info. Closes #72259

* Apply suggestions from code review

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

* Incorporated review feedback.
2021-04-28 18:30:32 -07:00
James Rodewig 693807a6d3
[DOCS] Fix double spaces (#71082) 2021-03-31 09:57:47 -04:00
debadair ba45251833
[DOCS] Update package description. (#68565)
* [DOCS] Update package description.

* Fixed xref
2021-02-04 16:47:12 -08:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
James Rodewig cb3e0051e0
[DOCS] Make cat API verbose query param explicit (#67300) 2021-01-11 17:19:23 -05:00
James Rodewig e9c1cded98
[DOCS] Fix outdated heap references (#66646) 2020-12-21 14:01:12 -05:00
Rory Hunter e49fd15e0c Fix docs typo 2020-12-16 20:29:20 +00:00
Rory Hunter 68b5465575
Make it possible to use Stack logging in Docker (#65778)
Closes #62758.

Include the Stack log4j config in the Docker image, in order to
make it possible to write logs in a container environment in the
same way as for an archive or package deployment. This is useful
in situations where the user is bind-mounting the logs directory
and has their own arrangements for log shipping.

To use stack logging, set the environment variable `ES_LOG_STYLE`
to `file`. It can also be set to `console`, which is the same as
not specifying it at all.

The Docker logging config is now auto-generated at image build time,
by running the default config through a transformer program when
preparing the distribution in an image builder step.

Also, in the docker distribution `build.gradle`, I changed a helper
closure into a class with a static method in order to fix an
issue where the Docker image was always being rebuilt, even when
there were no changes.
2020-12-10 12:25:47 +00:00
Rory Hunter a32a0986c3
Make ES files inside Docker container world readable (#64274)
Running the Elasticsearch Docker image with a different GID is
possible but trappy, since at present all the ES files are only
readable by the user and group. This PR documents a Docker CLI flag
that fixes this situation, by ensuring the container user is added
to the default group (which is `root`, GID 0).

I also added a test for this case, and refactored the Docker tests
to use a builder pattern for constructing the `docker run` command.
The existing code was becoming unwieldy and hard to change.
2020-10-30 13:26:44 +00:00
Adam Locke 65ba0d4d32
Fixing typo in file name. (#64226) 2020-10-27 10:48:31 -04:00
Rory Hunter ec350157ea
Use a tiny base image for Docker builds (#52519)
Closes #51670, closes #50838.

Introduce a tiny base image for Docker builds. It aims to create a basic filesystem with as little as possible, which is mostly glibc, busybox and bash. A statically-built curl is also provided.

We still use CentOS 8 as a base. All the fun stuff happens in the Dockerfile.
2020-10-12 14:58:01 +01:00
Rory Hunter a3b7566004 Upgrade Docker image from CentOS 7 to 8 2020-08-06 13:43:33 +01:00
Russ Cam ad496dfa63
Change vm.max_map_count on Docker WSL2 backend (#58153)
This commit adds docs for how to change
vm.max_map_count when running on Docker
Desktop with WSL2 backend on Windows.
2020-08-06 14:26:49 +10:00
Lisa Cawley 751e2d0deb
[DOCS] Fixes license management links (#58213) 2020-06-16 16:43:54 -07:00
DU-ds 4891c457d0 add jvm clarification (#57460)
Emphasise in the Docker documentation that although the default heap size is
1GB, the docker-compose.yml example specifies 512MB.
2020-06-05 11:50:24 +01:00
Jason Tedor 749b623de0
Introduce jvm.options.d for customizing JVM options (#51882)
This commit introduces the ability to override JVM options by adding
custom JVM options files to a jvm.options.d directory. This simplifies
administration of Elasticsearch by not requiring administrators to keep
the root jvm.options file in sync with changes that we make to the root
jvm.options file. Instead, they are not expected to modify this file but
instead supply their own in jvm.options.d. In Docker installations, this
means they can bind mount this directory in. In future versions of
Elasticsearch, we can consider removing the root jvm.options file
(instead, providing all options there as system JVM options).
2020-02-08 16:54:26 -05:00
Rory Hunter 87ee44d8bc
Clarify use of ES_JAVA_OPTS and Docker (#51867)
Closes #51626. Tweak the documentation around configuring the heap size
when using Docker, to state that:

- using `ES_JAVA_OPTS` is the preferred method
- Any `ES_JAVA_OPTS` overrides the defaults in `jvm.options`
- It's possible to bind-mount a custom `jvm.options`
2020-02-06 09:34:53 +00:00
William Brafford c117c0cf0a
Password-protected Keystore Feature Branch PR (#51123)
* Reload secure settings with password (#43197)

If a password is not set, we assume an empty string to be
compatible with previous behavior.
Only allow the reload to be broadcast to other nodes if TLS is
enabled for the transport layer.

* Add passphrase support to elasticsearch-keystore (#38498)

This change adds support for keystore passphrases to all subcommands
of the elasticsearch-keystore cli tool and adds a subcommand for
changing the passphrase of an existing keystore.
The work to read the passphrase in Elasticsearch when
loading, which will be addressed in a different PR.

Subcommands of elasticsearch-keystore can handle (open and create)
passphrase protected keystores

When reading a keystore, a user is only prompted for a passphrase
only if the keystore is passphrase protected.

When creating a keystore, a user is allowed (default behavior) to create one with an
empty passphrase

Passphrase can be set to be empty when changing/setting it for an
existing keystore

Relates to: #32691
Supersedes: #37472

* Restore behavior for force parameter (#44847)

Turns out that the behavior of `-f` for the add and add-file sub
commands where it would also forcibly create the keystore if it
didn't exist, was by design - although undocumented.
This change restores that behavior auto-creating a keystore that
is not password protected if the force flag is used. The force
OptionSpec is moved to the BaseKeyStoreCommand as we will presumably
want to maintain the same behavior in any other command that takes
a force option.

*  Handle pwd protected keystores in all CLI tools  (#45289)

This change ensures that `elasticsearch-setup-passwords` and
`elasticsearch-saml-metadata` can handle a password protected
elasticsearch.keystore.
For setup passwords the user would be prompted to add the
elasticsearch keystore password upon running the tool. There is no
option to pass the password as a parameter as we assume the user is
present in order to enter the desired passwords for the built-in
users.
For saml-metadata, we prompt for the keystore password at all times
even though we'd only need to read something from the keystore when
there is a signing or encryption configuration.

* Modify docs for setup passwords and saml metadata cli (#45797)

Adds a sentence in the documentation of `elasticsearch-setup-passwords`
and `elasticsearch-saml-metadata` to describe that users would be
prompted for the keystore's password when running these CLI tools,
when the keystore is password protected.

Co-Authored-By: Lisa Cawley <lcawley@elastic.co>

* Elasticsearch keystore passphrase for startup scripts (#44775)

This commit allows a user to provide a keystore password on Elasticsearch
startup, but only prompts when the keystore exists and is encrypted.

The entrypoint in Java code is standard input. When the Bootstrap class is
checking for secure keystore settings, it checks whether or not the keystore
is encrypted. If so, we read one line from standard input and use this as the
password. For simplicity's sake, we allow a maximum passphrase length of 128
characters. (This is an arbitrary limit and could be increased or eliminated.
It is also enforced in the keystore tools, so that a user can't create a
password that's too long to enter at startup.)

In order to provide a password on standard input, we have to account for four
different ways of starting Elasticsearch: the bash startup script, the Windows
batch startup script, systemd startup, and docker startup. We use wrapper
scripts to reduce systemd and docker to the bash case: in both cases, a
wrapper script can read a passphrase from the filesystem and pass it to the
bash script.

In order to simplify testing the need for a passphrase, I have added a
has-passwd command to the keystore tool. This command can run silently, and
exit with status 0 when the keystore has a password. It exits with status 1 if
the keystore doesn't exist or exists and is unencrypted.

A good deal of the code-change in this commit has to do with refactoring
packaging tests to cleanly use the same tests for both the "archive" and the
"package" cases. This required not only moving tests around, but also adding
some convenience methods for an abstraction layer over distribution-specific
commands.

* Adjust docs for password protected keystore (#45054)

This commit adds relevant parts in the elasticsearch-keystore
sub-commands reference docs and in the reload secure settings API
doc.

* Fix failing Keystore Passphrase test for feature branch (#50154)

One problem with the passphrase-from-file tests, as written, is that
they would leave a SystemD environment variable set when they failed,
and this setting would cause elasticsearch startup to fail for other
tests as well. By using a try-finally, I hope that these tests will fail
more gracefully.

It appears that our Fedora and Ubuntu environments may be configured to
store journald information under /var rather than under /run, so that it
will persist between boots. Our destructive tests that read from the
journal need to account for this in order to avoid trying to limit the
output we check in tests.

* Run keystore management tests on docker distros (#50610)

* Add Docker handling to PackagingTestCase

Keystore tests need to be able to run in the Docker case. We can do this
by using a DockerShell instead of a plain Shell when Docker is running.

* Improve ES startup check for docker

Previously we were checking truncated output for the packaged JDK as
an indication that Elasticsearch had started. With new preliminary
password checks, we might get a false positive from ES keystore
commands, so we have to check specifically that the Elasticsearch
class from the Bootstrap package is what's running.

* Test password-protected keystore with Docker (#50803)

This commit adds two tests for the case where we mount a
password-protected keystore into a Docker container and provide a
password via a Docker environment variable.

We also fix a logging bug where we were logging the identifier for an
array of strings rather than the contents of that array.

* Add documentation for keystore startup prompting (#50821)

When a keystore is password-protected, Elasticsearch will prompt at
startup. This commit adds documentation for this prompt for the archive,
systemd, and Docker cases.

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

* Warn when unable to upgrade keystore on debian (#51011)

For Red Hat RPM upgrades, we warn if we can't upgrade the keystore. This
commit brings the same logic to the code for Debian packages. See the
posttrans file for gets executed for RPMs.

* Restore handling of string input

Adds tests that were mistakenly removed. One of these tests proved
we were not handling the the stdin (-x) option correctly when no
input was added. This commit restores the original approach of
reading stdin one char at a time until there is no more (-1, \r, \n)
instead of using readline() that might return null

* Apply spotless reformatting

* Use '--since' flag to get recent journal messages

When we get Elasticsearch logs from journald, we want to fetch only log
messages from the last run. There are two reasons for this. First, if
there are many logs, we might get a string that's too large for our
utility methods. Second, when we're looking for a specific message or
error, we almost certainly want to look only at messages from the last
execution.

Previously, we've been trying to do this by clearing out the physical
files under the journald process. But there seems to be some contention
over these directories: if journald writes a log file in between when
our deletion command deletes the file and when it deletes the log
directory, the deletion will fail.

It seems to me that we might be able to use journald's "--since" flag to
retrieve only log messages from the last run, and that this might be
less likely to fail due to race conditions in file deletion.

Unfortunately, it looks as if the "--since" flag has a granularity of
one-second. I've added a two-second sleep to make sure that there's a
sufficient gap between the test that will read from journald and the
test before it.

* Use new journald wrapper pattern

* Update version added in secure settings request

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
2020-01-27 19:51:39 -05:00
Rory Hunter 8a6d68b173
Make the Docker build more re-usable in Cloud (#50277)
Closes #49926 and #46166. Rework the Docker image so that it comes with a tiny
init system, to ensure ML processes are correctly cleaned up, and to run ES
as a regular user instead of root.

Also:

   * Ensure no files in the image have the setuid/setgid flag
   * Also improve dependency tracking in the build
   * Remove TAKE_FILE_OWNERSHIP option and its documentation
2020-01-23 10:58:40 +00:00
Kevin Woblick 77d94caa70 [DOCS] Add warning about Docker port exposure (#50169)
Docker bypasses the Uncomplicated Firewall (UFW) on Linux by editing the `iptables` config directly, which leads to the exposure of port 9200, even if you blocked it via UFW.

This adds a warning along with work-arounds to the docs.

Signed-off-by: Kovah <mail@kovah.de>
2019-12-18 09:03:44 -05:00
Dimitrios Liappis 1c9efba809
Clarify gid used by docker image process and bind-mount method
Fix reference about the uid:gid that Elasticsearch runs as inside
the Docker container and add a packaging test to ensure that bind
mounting a data dir with a random uid and gid:0 works as
expected.

Relates #49529
Closes #47929
2019-11-27 10:36:30 +02:00
Rory Hunter 2a4e101a97
Support `_FILE` suffixed env vars in Docker entrypoint (#47573)
Closes #43603. Allow environment variables to be passed to ES in a Docker
container via a file, by setting an environment variable with the `_FILE`
suffix that points to the file with the intended value of the env var.
2019-11-12 14:20:12 +00:00
Arne Welzel 552cc2ea59 [DOCS] Remove extra "when" (#48926) 2019-11-11 10:11:38 +01:00
debadair d92f362df0
[DOCS] Fix cross-doc link. (#48783)
* [DOCS] Fix cross-doc link.

* Fixed xref
2019-10-31 18:57:02 -07:00
debadair 2f9f5765e3
[DOCS] Edited Docker install & tweaked Docker compose file. (#47715)
* [DOCS] Edited Docker install & tweaked Docker compose file.

* Synced with Docker GS in SO

* Incorporated review comments
2019-10-31 18:04:07 -07:00
harsha-s f9227da5b9 Update docker.asciidoc (#47651)
Update `docker-compose.yml` in the documentation in order to increase the max
file descriptor limit.
2019-10-19 15:41:15 +01:00
Lisa Cawley 4e4990c6a0
[DOCS] Cleans up links to security content (#47610) 2019-10-04 16:10:26 -07:00
Dimitrios Liappis 481b359da3
Clarify heap setting in Docker docs (#42754)
Add note in the Docker docs that even when container memory is limited,
we still require specifying -Xms/-Xmx using one of the supported
methods.
2019-06-03 15:12:29 +03:00
Travis Steel 50ed840e52 Fixed typo in docker.asciidoc (#42455) 2019-05-27 11:56:51 +02:00
Yannick Welsch 2de6448781
Highlight the use of single-node discovery in docker docs (#41241)
Relates to https://discuss.elastic.co/t/es-7-and-docker-single-node-cluster/176585
2019-05-08 09:28:27 +02:00
James Rodewig adf67053f4
[DOCS] Add anchors for Asciidoctor migration (#41648) 2019-04-30 10:19:09 -04:00
Joe Zack b6b98f72ce Fix discovery config in docker-compose docs (#41394)
Today's `docker-compose` docs are missing the `discovery.seed_nodes` config on
one of the nodes. With today's configuration the cluster can still form the
first time it is started, because `cluster.initial_master_nodes` requires both
nodes to bootstrap the cluster which ensures that each discover the other.
However if `es02` is elected master it will remove `es01` from the voting
configuration and then when restarted it will form a cluster on its own without
needing to do any discovery. Meanwhile `es01` doesn't know how to find `es02`
after a restart so will be unable to join this cluster.

This commit fixes this by adding the missing configuration.
2019-04-20 09:33:04 +01:00
James Rodewig 2141af6b84
[DOCS] Fix code block length for Asciidoctor migration (#41151) 2019-04-12 11:19:21 -04:00
James Rodewig 0362fe13a6
[DOCS] Fix broken link to Elasticsearh Docker source code (#40979)
* [DOCS] Fix broken link to Elasticsearch Docker source code

* [DOCS] Link to Dockerfile in elastic/elasticsearch repo

* [DOCS] Link to Docker source files in elastic/elasticsearch repo
2019-04-08 16:11:32 -04:00
David Turner 2d114a02ff
Rename static Zen1 settings (#38333)
Renames the following settings to remove the mention of `zen` in their names:

- `discovery.zen.hosts_provider` -> `discovery.seed_providers`
- `discovery.zen.ping.unicast.concurrent_connects` -> `discovery.seed_resolver.max_concurrent_resolvers`
- `discovery.zen.ping.unicast.hosts.resolve_timeout` -> `discovery.seed_resolver.timeout`
- `discovery.zen.ping.unicast.hosts` -> `discovery.seed_addresses`
2019-02-05 08:46:52 +00:00
Jason Tedor 18a3e48a4a
Change file descriptor limit to 65535 (#37537)
Some systems default to a nofile ulimit of 65535. To reduce the pain of
deploying Elasticsearch to such systems, this commit lowers the required
limit from 65536 to 65535.
2019-01-16 17:19:12 -05:00
David Turner 1a23417aeb
[Zen2] Update documentation for Zen2 (#34714)
This commit overhauls the documentation of discovery and cluster coordination,
removing mention of the Zen Discovery module and replacing it with docs for the
new cluster coordination mechanism introduced in 7.0.

Relates #32006
2018-12-20 13:02:44 +00:00
Dimitrios Liappis 1cb578b435
[DOCS] Update sysctl instructions for Docker on Mac (#35755)
Recent Docker for Mac releases[1] have a different path to the tty for
accessing the console of the xhyve vm, required for altering the
`vm.max_map_count` sysctl.

Update instructions on how to enter the xhyve vm for altering the
`vm.max_map_count` sysctl setting on Docker for Mac.

Closes #34817 

[1]
https://forums.docker.com/t/is-it-possible-to-ssh-to-the-xhyve-machine/17426/13
2018-11-21 11:32:29 +02:00
DeDe Morton e01e4393a8
[Docs] Light edit to info about docker images (#32376) 2018-08-06 12:00:07 -07:00
Lisa Cawley efcfd0d827
[DOCS] Removes alternative docker pull example (#31934) 2018-07-11 09:08:32 -07:00
David Pilato 8c6ee7db54
Describe how to add a plugin in Dockerfile (#31340)
When installing a plugin, people need to add the `--batch` option.
It's better to document it as it could be a common use case.
2018-06-15 07:32:49 +02:00
Lisa Cawley 7f0c2e89c2
[DOCS] Moves X-Pack setup to docs (#31145) 2018-06-06 14:46:20 -07:00
Lisa Cawley 9f43d7329b
[DOCS] Move X-Pack-specific Docker content (#27333) 2017-11-10 09:38:32 -08:00
Toby McLaughlin b71f7d3559
Update Docker docs for 6.0.0-rc2 (#27166)
* Update Docker docs for 6.0.0-rc2

* Update the docs to match the new Docker "image flavours" of "basic",
"platinum", and "oss".

* Clarifications for Openshift and bind-mounts

* Bump docker-compose 2.x format to 2.2

* Combine Docker Toolbox instructions for setting vm.max_map_count for
  both macOS + Windows

* devicemapper is not the default storage driver any more on RHEL
2017-11-01 14:24:30 +11:00
Amine Daï 3cb99aeec1 Fix references to vm.max_map_count in Docker docs
This commit fixes some references to vm.max_map_count in the Docker
docs.

Relates #26798
2017-09-29 15:56:18 -04:00
Dimitrios Liappis b789ce737b Docs: Use single-node discovery.type for dev example
For the single node, dev example, the `discovery.type=single-node`[1],[2] 
is a perfect fit and makes the example shorter and more self explanatory.

Also expose the transport port, to help with dev use-cases using the 
transport client.

[1] https://github.com/elastic/elasticsearch/pull/23595
[2] https://github.com/elastic/elasticsearch/pull/23598

Relates #26289
2017-09-15 16:14:47 +03:00
michaelbaamonde c0dbd236c3 Fix typo re: bootstrap.memory_lock in Docker docs. (#26265)
`bootstrap_memory_lock` should be `bootstrap.memory_lock`.
2017-08-18 11:55:56 -04:00
Deb Adair 2765ea41ca [DOCS] Fixed broken cross doc links to security settings. 2017-06-26 17:48:56 -07:00