elasticsearch/x-pack/plugin/data-streams
Mayya Sharipova f18b9d5ac8
Add segment sorter for data streams (#75195)
It is beneficial to sort segments within a datastream's index
by desc order of their max timestamp field, so
that the most recent (in terms of timestamp) segments
will be first.

This allows to speed up sort query on @timestamp desc field,
which is the most common type of query for datastreams,
as we are mostly concerned with the recent data.
This patch addressed this for writable indices.

Segments' sorter is different from index sorting.
An index sorter by itself is  concerned about the order of docs
within an individual segment (and not how the segments are organized),
while the segment sorter is only used during search and allows
to start docs collection with the "right" segment,
so we can terminate the collection faster.

This PR adds a property to IndexShard `isDataStreamIndex` that
shows if a shard is a part of datastream.
2021-09-03 09:42:48 -04:00
..
qa Introduce simple public yaml-rest-test plugin (#76554) 2021-08-31 08:45:52 +02:00
src Add segment sorter for data streams (#75195) 2021-09-03 09:42:48 -04:00
build.gradle Set netty available processors system property for tests globally (#75699) 2021-07-27 11:21:42 -07:00