Fixes the text field mapper and the analyzers class that also retained parameter references that go really heavy.
Makes `TextFieldMapper` take hundreds of bytes compared to multiple kb per instance.
closes#73845
This commit updates two task names:
```
yamlRestCompatTest -> yamlRestTestV7CompatTest
transformV7RestTests -> yamlRestTestV7CompatTransform
```
`7` is the N-1 version and calculated, such that when `8` is
N-1 version the task names will be `yamlRestTestV8CompatTest` and
`yamlRestTestV8CompatTransform`
The motivation for `yamlRestCompatTest -> yamlRestTestV7CompatTest` is that
many projects have configured `yamlRestCompatTest`
but that configuration is specific to the N-1 version. For example,
if we blacklist tests when running compatibility with v7, we don't also
want to blacklist those tests when running compatibility with v8.
By introducing a version-specific identifier in the name, the task will not
even exist when bumping the version creating the need to (correctly) remove
the version-specific condition.
The motivation for `transformV7RestTests -> yamlRestTestV7CompatTransform`
is to provide more consistent naming.
The idea behind the naming is the main task people
are likely familiar with is :
`yamlRestTest` so we will use that as a base.
`yamlRestTestV7CompatTest` to run the version-specific compat tests
`yamlRestTestV7CompatTransform` to run the version-specific transformations for the compat tests
CI should be un-effected since since we introduced a lifecycle task
name `checkRestCompat` which is what CI should be configured to use.
Just like #77131 but for the `MatchOnlyTextFieldMapper`. Also, cleaned up a few
other minor things in it to make the constructor code for this class easier to follow.
Today `AbstractRefCounted` has a `name` field which is only used to
construct the exception message when calling `incRef()` after it's been
closed. This isn't really necessary, the stack trace will identify the
reference in question and give loads more useful detail besides. It's
also slightly irksome to have to name every single implementation.
This commit drops the name and the constructor parameter, and also
introduces a handy factory method for use when there's no extra state
needed and you just want to run a method or lambda when all references
are released.
This commit allows the compatible REST API tests to execute on Windows.
They were previously excluded from Windows due to a command line limit
when defining a very large exclusion list. That exclusion list is much
smaller now and they will now execute properly on Windows.
Also, an empty exclusion list has been removed from the build config.
This adds two utility methods for to validate the parameters to the
`docValueFormat` method and replaces a pile of copy and pasted code with
calls to them. They just emit a standard error message if the any
unsupported parameters are provided.
This PR implements support for multiple validators to a FieldMapper.Parameter.
The Parameter#setValidator method was replaced by Parameter#addValidator that can be called multipled times
to add validation to a parameter.
All validators of a parameter will be executed in the same order as they have been added and if any of them fails all validation will failed.
This change adds a method to the PainlessLookup used to find methods of all allow listed sub classes.
A method is specified for a specific super class, and then a list is built with all matching methods from
the all the allow listed sub classes. If no matches are found, null is returned which is consistent with
the behavior of the other look up methods. It is up to the caller to check.
This introduces a basic public yaml rest test plugin that is supposed to be used by external
elasticsearch plugin authors. This is driven by #76215
- Rename yaml-rest-test to intern-yaml-rest-test
- Use public yaml plugin in example plugins
Co-authored-by: Mark Vieira <portugee@gmail.com>
This change has two main components.
The first is to have method/field resolution for compile-time and run-time use the same code path for
now. This removes copying of member methods between super and sub classes and instead does a
resolution through the class hierarchy. This allows us to correctly implement the next change.
The second is a data structure that allows for the lookup of direct sub classes for all allow listed
classes/interfaces within Painless.
This change allows user to disable GeoIP downloader using elasticsearch.yml and it deletes .geoip_databases index if downloader is disabled.
Closes#76586
Composite runtime fields do not have a mapped type - add null check, test and Nullable annotation to SearchExecutionContext.getObjectMapper(name)
Closes#76716
An attempt to apply the spotless plugin everywhere and then disable
where it wasn't appropriate didn't work, and instead everything was
formatted. Revert how we apply the pluing, and use a different approach
to applying extra configuration in build files.
v7compatibilityNotSupportedTests was introduced to make it easier to
track tests that have been identified as not needing compatible changes
and those that still need to be checked.
We have checked all tests now and the separate list is no longer needed.
relates #51816
relates #73912
This change adds a a new "augmented" annotation to the Painless allowlist parser. The first use of the
annotation supports adding static final fields to a specified allowlist class from another class. This
supports the fields api as we can add additional fields types from other classes and augment the Field
class with the new types.
* Reformatting to keep Checkstyle after formatting
* Configure spotless everywhere, and disable the tasks if necessary
* Add XContentBuilder helpers, fix test
* Tweaks
* Add a TODO
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Script: ulong via fields API
Exposes unsigned long via the fields API.
Unsigned longs default to java signed longs. That means the upper range
appears negative. Consumers should use `Long.compareUnsigned(long, long)`
`Long.divideUnsigned(long, long)` and `Long.remainderUnsigned(long, long)`
to correctly work with values known to be unsigned long.
Alternatively, users may treat the unsigned long type as `BigInteger` using
the field API, `field('ul').as(Field.BigInteger).getValue(BigInteger.ZERO)`.
```
field('ul').as(Field.BigInteger).getValue(BigInteger.valueOf(1000))
field('ul').getValue(1000L)
```
This change also implements the beginning of the converters for the fields
API. The following conversions have been added:
```
ulong <-> BigInteger
long <-> BigInteger
double -> BigInteger
String (parsed as long or double) -> BigInteger
double -> long
String (parsed as long or double) -> long
Date (epoch milliseconds) -> long
Nano Date (epoch nanoseconds) -> long
boolean (1L for true, 0L for false) -> long
```
This fixes the error message in Painless for when an instance-style method isn't found. It now prints the
number of arguments instead of the memory location of the nodes data structure.
Painless extensions now compile against it's `spi` subproject. Gradle
and IntelliJ rig the dependencies correctly but Eclipse doesn't seem to
manage it. This temporarily declares the `spi` as a `compileOnly`
dependency when in Eclipse land so Eclipse can see the classes.
* Name `BulkItemResponse` ctors
`BulkItemResponse` can contain either a success or failure. This
replaces the two constructors used to build either case with named
static methods. So instead of
```
return new BulkItemResponse(0, OpType.CREATE, createResponse);
return new BulkItemResponse(0, OpType.CREATE, failure);
```
you now use
```
return BulkItemResponse.success(0, OpType.CREATE, createResponse);
return BulkItemResponse.failure(0, OpType.CREATE, failure);
```
This makes it marginally easier to read code building these things - you
don't have to know the type of the parameter to know if its a failure
or success.
* Consistent
* Mock response
This change creates a standard for loading additional allow listed elements for the core contexts
within the ScriptPlugin. For each core context we look for org.elasticsearch.script.<context-name>.txt
and if it exists we load it automatically for that context.
The javadoc generation for painless api classes was broken by a recent
commit. This commit fixes it by correct the classpath used to run the
javadoc task. Additionally, the task is added to `check` to ensure it
runs with CI.
closes#76373
This change updates the aggregation script, map script for aggregations, and field scripts to extend
DocBasedScript to give them access to the new fields api.
* Add support in plugins for a separate spi classloader
SPI is how plugins provide extension points for other plugins to
customize their behavior. Plugins may have a separate SPI jar, so as not
to expose the internals of the plugin implementation. In essense, this
system was built as a stopgap for Java modules, where implementation can
be protected in the same jar. However, currently the plugins service
loads both spi and plugin implementations into the same classloader.
This means that although another plugin may only compile against spi,
they could still get jar hell from a duplicate dependency.
This commit adds support for plugins to contain an "spi" subdirectory
which is loaded into a separate classloader. This spi classloader is a
parent to the plugin implementation, as well as any other plugins which
have extended it. Additionally as a demonstration of how this works in
practice, lang-painless is setup as the first plugin to provide an spi
jar, and sql's dependence on painless is changed to only spi, so that
it now has an independent version of antlr.
closes#74448
* Change eql to use painless spi
* checkstyle
* add licens files to eql for antlr
* Address review
We have recently introduced support for grok and dissect to the runtime fields
Painless context that allows to split a field into multiple fields. However, each runtime
field can only emit values for a single field. This commit introduces support for emitting
multiple fields from the same script.
The API call to define a runtime field that emits multiple fields is the following:
```
PUT localhost:9200/logs/_mappings
{
"runtime" : {
"log" : {
"type" : "composite",
"script" : "emit(grok(\"%{COMMONAPACHELOG}\").extract(doc[\"message.keyword\"].value))",
"fields" : {
"clientip" : {
"type" : "ip"
},
"response" : {
"type" : "long"
}
}
}
}
}
```
The script context for this new field type accepts two emit signatures:
* `emit(String, Object)`
* `emit(Map)`
Sub-fields need to be declared under fields in order to be discoverable through
the field_caps API and accessible through the search API.
The way that it emits multiple fields is by returning multiple MappedFieldTypes
from RuntimeField#asMappedFieldTypes. The sub-fields are instances of the
runtime fields that are already supported, with a little tweak to adapt the script
defined by their parent to an artificial script factory for each of the sub-fields
that makes its corresponding sub-field accessible. This approach allows to reuse
all of the existing runtime fields code for the sub-fields.
The runtime section has been flat so far as it has not supported objects until now.
That stays the same, meaning that runtime fields can have dots in their names.
Because there are though two ways to create the same field with the introduction
of the ability to emit multiple fields, we have to make sure that a runtime field with
a certain name cannot be defined twice, which is why the following mappings are
rejected with the error `Found two runtime fields with same name [log.response]`:
```
PUT localhost:9200/logs/_mappings
{
"runtime" : {
"log.response" : {
"type" : "keyword"
},
"log" : {
"type" : "composite",
"script" : "emit(\"response\", grok(\"%{COMMONAPACHELOG}\").extract(doc[\"message.keyword\"].value)?.response)",
"fields" : {
"response" : {
"type" : "long"
}
}
}
}
}
```
Closes#68203
It's in the title, we were not accounting for relative paths at all
here and only saved by the fact that we mostly short-circuit to
non-streaming writes.
Extended testing to catch this case for S3 and would do a follow-up
to extend it for the other implementations as well.
The base whitelist is shared by all script contexts. However, the text
files exist within painless itself, not the limited spi jar which is
meant for extension. This commit moves the base definition into painless
and reworks context loading in painless to add the base whitelist so
that all contexts get it, regardless of whether they have added an
explicit whitelist.
Previously removed in #42654. The query and the parameter won't work under rest api compatibility and an exception with a message is returned advising that just use of match/multi_match is enough
relates #51816
A new trace.id header is added by #74210. It is handled almost the same
way as x-opaque-id. Specifically, it gets passed into default thread
context. This means the existing assertion should expect it in addition
to x-opaque-id.
Adds minimal fields API support to sort and score scripts.
Example: `field('myfield').getValue(123)` where `123` is the default if the field has no values.
Refs: #61388
This change adds support to allow list instance methods on the script class in Painless and super
classes of the script class. It works the same way as allow listing works now for other classes.
Armin Braun
Jake Landis
Seth Michael Larson
David Turner
Ignacio Vera
Nik Everett
Dan Hermann
Christos Soulios
Jack Conradson
Rene Groeschke
Mark Tozzi
Stuart Tettemer
Przemko Robakowski
markharwood
Rory Hunter
Przemyslaw Gomulka
Mark Vieira
Ryan Ernst
Luca Cavanna
Yang Wang
Henning Andersen