The weighted tokens query requires a list of token-weight pairs that are sent in with a query rather than calculated using a {nlp} model.
These token pairs are then used in a query against a <<sparse-vector,sparse vector>> or <<rank-features,rank features>> field.
Weighted tokens queries are useful when you want to use an external query expansion model, or quickly prototype changes without reindexing a new model.
(Optional, integer) Tokens whose frequency is more than `tokens_freq_ratio_threshold` times the average frequency of all tokens in the specified field are considered outliers and pruned.
It is strongly recommended to set this to `false` for the main query, but this can be set to `true` for a rescore query to get more relevant results.
Default: `false`.
NOTE: The default values for `tokens_freq_ratio_threshold` and `tokens_weight_threshold` were chosen based on tests using ELSER that provided the most optimal results.
==== Example weighted tokens query with pruning configuration and rescore
The following example adds a pruning configuration to the `text_expansion` query.
The pruning configuration identifies non-significant tokens to prune from the query in order to improve query performance.
Token pruning happens at the shard level.
While this should result in the same tokens being labeled as insignificant across shards, this is not guaranteed based on the composition of each shard.
Therefore, if you are running `text_expansion` with a `pruning_config` on a multi-shard index, we strongly recommend adding a <<rescore>> function with the tokens that were originally pruned from the query.
This will help mitigate any shard-level inconsistency with pruned tokens and provide better relevance overall.
