elasticsearch/plugins
Armin Braun 6b7f4c6bec
Formalize and Streamline Buffer Sizes used by Repositories (#59771)
Due to complicated access checks (reads and writes execute in their own access context) on some repositories (GCS, Azure, HDFS), using a hard coded buffer size of 4k for restores was needlessly inefficient.
By the same token, the use of stream copying with the default 8k buffer size  for blob writes was inefficient as well.

We also had dedicated, undocumented buffer size settings for HDFS and FS repositories. For these two we would use a 100k buffer by default. We did not have such a setting for e.g. GCS though, which would only use an 8k read buffer which is needlessly small for reading from a raw `URLConnection`.

This commit adds an undocumented setting that sets the default buffer size to `128k` for all repositories. It removes wasteful allocation of such a large buffer for small writes and reads in case of HDFS and FS repositories (i.e. still using the smaller buffer to write metadata) but uses a large buffer for doing restores and uploading segment blobs.

This should speed up Azure and GCS restores and snapshots in a non-trivial way as well as save some memory when reading small blobs on FS and HFDS repositories.
2020-07-22 16:46:37 +02:00
..
analysis-icu Drop some params from IndexFieldData.Builder (#59934) 2020-07-21 08:29:58 -04:00
analysis-kuromoji upgrade to lucene-8.6.0 release (#59596) 2020-07-15 11:58:28 +02:00
analysis-nori upgrade to lucene-8.6.0 release (#59596) 2020-07-15 11:58:28 +02:00
analysis-phonetic upgrade to lucene-8.6.0 release (#59596) 2020-07-15 11:58:28 +02:00
analysis-smartcn upgrade to lucene-8.6.0 release (#59596) 2020-07-15 11:58:28 +02:00
analysis-stempel upgrade to lucene-8.6.0 release (#59596) 2020-07-15 11:58:28 +02:00
analysis-ukrainian upgrade to lucene-8.6.0 release (#59596) 2020-07-15 11:58:28 +02:00
discovery-azure-classic Create plugin for yamlTest task (#56841) 2020-07-06 12:13:01 -05:00
discovery-ec2 Create plugin for yamlTest task (#56841) 2020-07-06 12:13:01 -05:00
discovery-gce Create plugin for yamlTest task (#56841) 2020-07-06 12:13:01 -05:00
examples Create plugin for yamlTest task (#56841) 2020-07-06 12:13:01 -05:00
ingest-attachment Create plugin for yamlTest task (#56841) 2020-07-06 12:13:01 -05:00
mapper-annotated-text MappedFieldType no longer requires equals/hashCode/clone (#59212) 2020-07-09 21:01:29 +01:00
mapper-murmur3 Drop some params from IndexFieldData.Builder (#59934) 2020-07-21 08:29:58 -04:00
mapper-size Create plugin for yamlTest task (#56841) 2020-07-06 12:13:01 -05:00
repository-azure Remove Artificially Low Chunk Size Limits from GCS + Azure Blob Stores (#59279) 2020-07-14 21:34:53 +02:00
repository-gcs Formalize and Streamline Buffer Sizes used by Repositories (#59771) 2020-07-22 16:46:37 +02:00
repository-hdfs Formalize and Streamline Buffer Sizes used by Repositories (#59771) 2020-07-22 16:46:37 +02:00
repository-s3 Enable Fully Concurrent Snapshot Operations (#56911) 2020-07-10 15:19:08 +02:00
store-smb Create plugin for yamlTest task (#56841) 2020-07-06 12:13:01 -05:00
transport-nio Use `getPortRange` in http server tests (#58794) 2020-07-02 13:08:04 -06:00
build.gradle