Commit Graph

1035 Commits

Author SHA1 Message Date
Rory Hunter 71546b3a3e
Refactor plugin CLI commands (#75259)
Part of #70219.

Split the install and remove plugin CLI commands into `Command` and
`Action` classes, so that the main implementations can be reused
in a later PR to create a `Sync` command.
2021-07-28 12:43:54 +01:00
Mark Vieira c6ec78b058
Fix dependency report link to JDK sources (#75742)
OpenJDK sources have migrated to GitHub, so we need to update the link
we generate in our dependencies report appropriately.
2021-07-27 09:45:54 -07:00
Rene Groeschke b5016d9179
Update rpm build to add sha256 payload and file digest (#75731)
This is a follow up on https://github.com/elastic/elasticsearch/pull/75569

and should fix installation problems in FIPS enabled environments.
2021-07-27 17:37:15 +02:00
Rene Groeschke 2c5e406b12
Add Sha256 header in elasticsearch RPMs (#75569)
This adds support for Sha256 header signature in our RPMs by
updating the dependency to the readline library to a version
we have patched until the provided PR (https://github.com/craigwblake/redline/pull/157)
got merged and released by the redline folks.

This work is related to #58257
2021-07-22 08:32:53 +02:00
Jake Landis 4553bf03d5
Remove MavenFilteringHack (#73637) 2021-07-20 17:05:16 -04:00
Rory Hunter 7d4eb5d0bc
Simplify building the default log4j2.properties (#75535)
The process for building `log4j2.properties` in the default distribution
is complicated, expensive and requires all modules and plugins to be
built in order to generate the file. Improve this by directly grabbing
the files and concatenating them.

Also remove code for building the OSS distro config.
2021-07-20 20:33:37 +01:00
Rory Hunter bff790011c Increment Iron Bank base image to 8.4
Acting on a request from the Iron Bank folks.
2021-07-15 21:21:33 +01:00
Rory Hunter d08b851a5b
Accept settings in snake case in Docker image (#74327)
Closes #74036. Since some orchestration platforms forbid periods in
environment variable names, allow Docker users to pass settings to ES
using an alternative name scheme. For example:

    bootstrap.memory_lock

...becomes:

    ES_BOOTSTRAP_MEMORY__LOCK

The setting name is uppercased, prefixed, all underscores are converted
to double underscores, and all periods are converted to underscores.
2021-07-09 19:46:58 +01:00
Ioannis Kakavas 0ad794833f
Determine upgrades correctly for deb packages (#75010)
We are using the `-n` bash operator to determine whether a second
argument is passed to the `postinst` script during installation of
our DEB packages. The reasoning is that we want to differentiate
between installations and upgrades and a second argument to
postinst is only passed when this is an upgrade ( the identifier
of the version the package is upgraded to ).
The problem is that we use `-n` without quoting the $2 and this is
unsafe practice as it might yield unexpected results for `-n` and
`-z` operators used within test brackets. Testing with bash
5.0-6ubuntu1.1 `[ -n $2 ]` returns false both for upgrades and
installations, while `[ -n "$2" ]` behaves as expected.

References:
https://tldp.org/LDP/abs/html/comparison-ops.html
https://wiki.debian.org/MaintainerScripts
2021-07-07 18:00:51 +03:00
Nhat Nguyen 60c2838906
Replace internal usages of SimpleFS with NIOFS (#74996)
SimpleFS is deprecated and removed in Lucene 9. This commit replaces its 
internal usages with NIOFS. Two other usages (store type and SMB) need
to be deprecated before switching to niofs.
2021-07-07 08:57:20 -04:00
Julien Mailleret 8a72639863
Fix UBI source URL (#74357)
This commit fix the source URL for UBI image to ensure that it stays
 consistent with the one generated in
 https://artifacts.elastic.co/reports/dependencies/dependencies-current.html
2021-06-21 15:41:32 +02:00
Rene Groeschke c2e86258d8
Update wrapper to 7.1 (#73941)
- Fix new introduced deprecated usages
- Update to newer ospackage snapshot to include provided PR for fixing deprecated usage

This gradle release comes with improvements on incremental compilation which we should benefit from
2021-06-17 10:59:22 +02:00
Rory Hunter 47029736cf
Pin Alpine Linux version in Docker builds (#74169)
Alpine Linux 3.14.0 is incompatible with older versions of Docker, so pin the
version that we use to 3.13. At some point in the future, it will
be possible to upgrade Alpine.

Also when compiling curl, if the configure step fails and a config.log
file exists, then dump it out before exiting to assist diagnosis.
2021-06-16 14:29:00 +01:00
Rory Hunter a5d2251064
Order imports when reformatting (#74059)
Change the formatter config to sort / order imports, and reformat the
codebase. We already had a config file for Eclipse users, so Spotless now
uses that.

The "Eclipse Code Formatter" plugin ought to be able to use this file as
well for import ordering, but in my experiments the results were poor.
Instead, use IntelliJ's `.editorconfig` support to configure import
ordering.

I've also added a config file for the formatter plugin.

Other changes:
   * I've quietly enabled the `toggleOnOff` option for Spotless. It was
     already possible to disable formatting for sections using the markers
     for docs snippets, so enabling this option just accepts this reality
     and makes it possible via `formatter:off` and `formatter:on` without
     the restrictions around line length. It should still only be used as
     a very last resort and with good reason.
   * I've removed mention of the `paddedCell` option from the contributing
     guide, since I haven't had to use that option for a very long time. I
     moved the docs to the spotless config.
2021-06-16 09:22:22 +01:00
Rory Hunter ee98e8593b
Tighten up write permissions in Docker image (#73942)
Recursively remove write access from the bin, jdk, lib and
modules directories, since this access is not required, and removing
it makes it harder to exploit other issues in an ES distribution.
2021-06-11 13:31:49 +01:00
Ryan Ernst ab1a2e4a84
Add precommit task for detecting split packages (#73784)
Modularization of the JDK has been ongoing for several years. Recently
in Java 16 the JDK began enforcing module boundaries by default. While
Elasticsearch does not yet use the module system directly, there are
some side effects even for those projects not modularized (eg #73517).
Before we can even begin to think about how to modularize, we must
Prepare The Way by enforcing packages only exist in a single jar file,
since the module system does not allow packages to coexist in multiple
modules.

This commit adds a precommit check to the build which detects split
packages. The expectation is that we will add the existing split
packages to the ignore list so that any new classes will not exacerbate
the problem, and the work to cleanup these split packages can be
parallelized.

relates #73525
2021-06-08 15:04:23 -07:00
Ryan Ernst 68817d7ca2
Rename o.e.common in libs/core to o.e.core (#73909)
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.

relates #73784
2021-06-08 09:53:28 -07:00
Ryan Ernst 64054de1ac
Rename bootstrap package in core jar (#73788)
The org.elasticsearch.bootstrap package exists in server with classes
for starting up Elasticsearch. The elasticsearch-core jar has a handful
of classes that were split out from there, namely java version parsing
and jarhell. This commit moves those classes to a new
org.elasticsearch.jdk package so as to not split the server owned
bootstrap package.

relates #73784
2021-06-07 08:14:44 -07:00
Rory Hunter 577010740e
Allow container restarts with file logging (#73101)
Closes #72702. It wasn't possible to restart an Elasticsearch Docker
container when using `ES_LOG_STYLE=file`, and now it is.
2021-05-14 20:40:27 +01:00
Rene Groeschke e609e07cfe
Remove internal build logic from public build tool plugins (#72470)
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic

This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.

It also introduces a set of internal versions of public plugins.

As part of this we also generate the plugin descriptors now.

As a follow up on this we can actually move these public used classes into 
an extra project (declared as included build)

We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
2021-05-06 14:02:35 +02:00
Ryan Ernst 2a39b55dcd
Add snakyaml forbidden apis (#72353)
This commit adds a new signature file for snakeyaml, to avoid certain
constructors. It is not added as part of server signatures so that it
can be used in launchers where there is also a use of snakeyaml.
2021-04-28 08:59:03 -07:00
Ryan Ernst be31c46704
Tweak constructor for yaml parser (#72352)
This commit switches to using a better constructor
when parsing yaml in the machine dependent sizing code.
2021-04-27 22:44:14 -07:00
Rene Groeschke 5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Przemko Robakowski 308aee283d
Update GeoIP processor documentation (#71211)
This PR adds documentation for GeoIPv2 auto-update feature.
It also changes related settings names from geoip.downloader.* to ingest.geoip.downloader to have the same convention as current setting.

Relates to #68920

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2021-04-15 13:47:09 +02:00
Przemko Robakowski 39eb12a972
Enable GeoIP downloader by default (#71505)
This change enables GeoIP downloader by default.
It removes feature flag but adds flag that is used by tests to disable it again (as we don't want to hammer GeoIP database service with every test cluster we spin up).

Relates to #68920
2021-04-15 12:28:37 +02:00
Mark Vieira 6f995e65bc Disable fail on deprecation for packer cache script 2021-04-14 13:07:40 -07:00
Lyudmila Fokina 3b0b7941ae
Warn users if security is implicitly disabled (#70114)
* Warn users if security is implicitly disabled

Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
 clusters.
 This change introduces clear warnings when security features are
 implicitly disabled.
 - a warning header in each REST response if security is implicitly
 disabled;
 - a log message during cluster boot.
2021-04-13 18:33:41 +02:00
Przemko Robakowski 46efa6ad04
Fix problems in GeoIPv2 code (#71598)
This change fixes number of problems in GeoIPv2 code:

- closes streams from Files.list in GeoIpCli, which should fix tests on Windows
- makes sure that total download time in GeoIP stats is non-negative (we serialize it as vInt which can cause problems with negative numbers and it can happen when clock was changed during operation)
- fixes handling of failed/simultaneous downloads, #69951 was meant as a way to prevent 2 persistent tasks to index chunks but it would prevent any update if single download failed mid indexing, this change uses timestamp (lastUpdate) as sort of UUID. This should still prevent 2 tasks to step on each other toes (overwriting chunks) but in the end still only single task should be able to update task state (this is handled by persistent tasks framework)
Closes #71145
2021-04-13 17:10:45 +02:00
Rory Hunter fb1921c9dc
Change deprecation logs data stream name (#68737)
More fixes to deprecation log indexing so that the data stream name and document
contents are more ECS-compatible.
2021-04-13 15:11:57 +01:00
Rene Groeschke 0f40889879
Update build to Gradle 7.0 (#68506)
- Update gradle wrapper to gradle 7.0
- Remove deprecated usages to make build 7.0 compatible
- Fix excludes in docs snippet tasks (See https://github.com/gradle/gradle/issues/16160 for details)
- Fix deprecation warnings in 7.0
- Add explicit dependencies that have been missed
- Make extract native licenses tasks output dir more explicit
- Use a snapshot of the ospackage plugin that includes a fix for 7.0 already
- fix test runtime classpath setup in repository-hdfs
- Make task dependency explicit to fix further deprecation warnings
- Remove manual check for http repo usages that has been deprecated in gradle 7.0
- Update spock to latest 2.0 milestone required for groovy 3
2021-04-13 09:15:08 +02:00
Rory Hunter 167ed57a6d
Transform docker log4j properties at build time (#71346)
For the Docker distribution, we transform the archive distribution's
log4j2 config so that all messages are logged to the console by default.
However this transformation step happens when the Docker image is built,
which means that the source for the transformation must be included in
the Docker context.

Improve this by making the archive distribution's log4j config available
as an artifact in the build, then simply copying it into the Docker context.
The transform logic has been reimplemented as a simple copy filter.
Consequently, the `transform-log4j-config` project has been removed.
2021-04-08 09:53:10 +01:00
Mark Vieira 5477626442
Use latest busybox musl binaries for Docker distribution (#71199) 2021-04-06 09:08:11 -07:00
Mark Vieira 07a5ed2bf7 Remute GeoIpCliTests 2021-04-01 12:55:41 -07:00
Przemko Robakowski ac0fc017ab
Fix setup code in GeoIpCliTests (#71156)
Instead of creating directories manually under temp dir we should simply use createTempDir() twice. This should make Windows be able to delete directories correctly.

This PR also removes unused method.

Fixes #71145
2021-04-01 09:05:15 +02:00
Yash Jipkate 60f4d22722
Change default value of `action.destructive_requires_name` to True. (#66908)
This PR sets the default value of `action.destructive_requires_name`
to `true.` Fixes #61074. Additionally, we set this value explicitly in
test classes that rely on wildcard deletions to clear test state.
2021-03-31 15:59:57 -04:00
Mark Vieira b07851261c Mute GeoIpCliTests 2021-03-31 12:48:23 -07:00
Przemko Robakowski 61fe14565a
Add tool for preparing local GeoIp database service (#71018)
Air-gapped environments can't simply use GeoIp database service provided by Infra, so they have to either use proxy or recreate similar service themselves.
This PR adds tool to make this process easier. Basic workflow is:

download databases from MaxMind site to single directory (either .mmdb files or gzipped tarballs with .tgz suffix)
run the tool with $ES_PATH/bin/elasticsearch-geoip -s directory/to/use [-t target/directory]
serve static files from that directory (for example with docker run -v directory/to/use:/usr/share/nginx/html:ro nginx
use server above as endpoint for GeoIpDownloader (geoip.downloader.endpoint setting)
to update new databases simply put new files in directory and run the tool again
This change also adds support for relative paths in overview json because the cli tool doesn't know about the address it would be served under.

Relates to #68920
2021-03-31 12:30:21 +02:00
Rory Hunter 1c5b7653f0
Build Docker images from exported contexts and test them (#70088)
Closes #69930. Closes #69928.

The ES build currently has 2 types of Docker output - Docker images,
and Docker build contexts. At the moment, only the images are tested,
meaning that bugs in the build contexts can go unnoticed.

This PR changes how we create Docker images so that we first create
the build contexts, and then build the images using them. This does
require some sleight-of-hand - the build contexts expect to download
an Elasticsearch archive directorly from the `Dockerfile`, which
will only ever work for non-snapshot version builds. In order to
get around this, the `Dockerfile` is modified to `COPY` in a local
archive file. Any other dependency files must exist in the build
context archive.

This PR also builds and tests the Iron Bank context. We do not
currently build a Docker image for this at all, and to build an
image requires us to set some build arguments to useful values. We
also need to provide all artifacts to the build, as the `Dockerfile`
cannot download anything. As a result, the `:distribution:docker`
project now defines a GitHub repository so that Gradle will download
a `tini` binary.

Note that there will need to be corresponding changes to
`release-manager`.
2021-03-26 13:58:03 +00:00
Rene Groeschke a329677d4a
Remove oss handling in docker distribution project (#70134)
- Remove docker oss log4j properties
- Remove oss images from docker compose

Follow up on #69775
Related to #68797
2021-03-17 11:20:04 +01:00
Jason Tedor b6eb6e3093
Suppress illegal reflective access in shared cache (#70344)
This commit temporarily supressess an illegal reflective access warning
by opening the java.io module to all unnamed modules. This is needed
when using the searchable snapshots shared cache due to some reflective
access that occurs there. This is temporary while we explore
alternatives.
2021-03-15 16:15:29 -07:00
Ryan Ernst 37c55efbc7
Restrict createClassLoader to modules (#67127)
Creating classloaders is not something plugins really need to do. It is
only necessary for cases of loading dynamic code, which we do within
painless. This commit restricts allowing this permission to modules.
2021-03-09 20:26:52 -08:00
Rene Groeschke 0db220f5a1
Cleanup distribution packaging logic after oss projects have been removed (#70015)
* Cleanup distribution/build.gradle script
2021-03-09 09:21:24 +01:00
Rory Hunter 477ded9769
Include log4j transform jar in docker build context tasks (#69808)
Ensure that the log4j transform jar is included in the generated Docker build context.
Also makes some small fixes to the Iron Bank Dockerfile.
2021-03-03 20:24:43 +00:00
Mark Vieira 3edc532b0e Update ironbank docker build script for changes to dockerBuildContext 2021-03-03 11:07:05 -08:00
Rene Groeschke 74938516ab
Fix dockerBuildContext usage in build scripts (#69896) 2021-03-03 10:55:38 -08:00
Rene Groeschke 24adbbca41
Remove OSS specific distribution packaging (#69775)
* Remove OSS projects from distribution packaging
* Remove OSS flavor from ElasticsearchDistribution
* Remove OSS handling from docker build setup
2021-03-03 13:39:29 +01:00
Jake Landis 3dc9dc01c7
avoid costly regex expression (#69768)
This commit changes part of a regular expression for some tests to
be more performant. While it is difficult articulate why this is change is much
faster, testing has shown for some inputs this match to be less then 1s,
where prior could take over 30s.

related: #69757
2021-03-02 07:35:27 -06:00
Rene Groeschke 487f37fe8f
Remove jcenter repository from our build (#69517)
- JCenter has been deprecated in Gradle
- JCenter was announced to be sunset by JFrog
- Use Gradle plugin portal as maven repository for build dependencies
- Use mavenCentral as general replacement for jcenter

Fixes #68476
2021-02-25 08:36:08 +01:00
Mark Vieira 755615781a
Ensure docker images for current architecture are built for testing (#69554) 2021-02-24 16:09:31 -08:00
Rory Hunter 0f6ad19f15
Support removing multiple plugins at once in the CLI (#69063)
Closes #66476. Add support for removing multiple plugins at the
same time to `elasticsearch-plugin`. Also change references from
"plugin name" to "plugin id", to align better with the installer
class.
2021-02-24 14:10:06 +00:00