Today, we allow ESQL to execute against an unlimited number of shards
concurrently on each node. This can lead to cases where we open and hold
too many shards, equivalent to opening too many file descriptors or
using too much memory for FieldInfos in ValuesSourceReaderOperator.
This change limits the number of concurrent shards to 10 per node. This
number was chosen based on the _search API, which limits it to 5.
Besides the primary reason stated above, this change has other
implications:
We might execute fewer shards for queries with LIMIT only, leading to
scenarios where we execute only some high-priority shards then stop.
For now, we don't have a partial reduce at the node level, but if we
introduce one in the future, it might not be as efficient as executing
all shards at the same time. There are pauses between batches because
batches are executed sequentially one by one. However, I believe the
performance of queries executing against many shards (after can_match)
is less important than resiliency.
Closes#103666
Backport of #104832
6bf2c7159b