adds a health section to the transform stats endpoint and implements reporting assignment, indexing/search and persistence problems, together with a overall health state.
When creating and updating transforms, it is possible for clients to provide secondary headers.
When PUT, _preview, _update is called with secondary authorization headers, those are then used or stored with the transform.
closes: https://github.com/elastic/elasticsearch/issues/86731
It was previously required that the _start API caller required the same roles as the create API caller.
This does not make sense as when the transform is actually running (after _start) we rely solely on the roles of the caller who created the transform.
Consequently, this commit does the permission validations and various checks with the roles of user who created the transform, not the one calling _start
When a transform has a `retention_policy` it needs to be able to delete documents in the destination index.
`create_index` does not necessitate that we can delete documents from it. So, even if we create the index, we need to verify that we can delete documents given the `retention_policy` definition.
This is not a crucial bug as the transform will simply fail later. Its nicer to fail sooner.
closes https://github.com/elastic/elasticsearch/issues/85409
Removes `testenv` annotations and related code. These annotations originally let you skip x-pack snippet tests in the docs. However, that's no longer possible.
Relates to #79309, #31619
Add an _upgrade endpoint to bulk upgrade transforms. _upgrade rewrites all transforms and its
artifacts into the latest format to the latest storage(index). If all transforms are upgraded old
indices and outdated documents get deleted. Using the dry_run option it is possible to check if
upgrades are necessary without applying changes.
Refactoring of cat transform to show more relevant information. The current cat transform shows a
lot of configuration details, however cat should show operationally useful information. This PR
changes the defaults and also adds when transform did a search last.
Transform writes dates as epoch millis, this does not work for historic data in some cases or is
unsupported. Dates should be written as such. With this PR transform starts writing dates in ISO
format, but as existing transform might rely on the format it provides backwards compatibility for
old jobs as well as a setting to write dates as epoch millis.
fixes#63787
Przemysław Witek
István Zoltán Szabó
Hendrik Muhs
Lisa Cawley
Benjamin Trent
James Rodewig