Commit Graph

157 Commits

Author SHA1 Message Date
Kostas Krikellas 137bb45662
Support runtime fields in synthetic source (#99796)
* Support runtime fields in synthetic source

* Update docs/changelog/99796.yaml

* Introduce SyntheticSourceProvider

* Address comments

* More fixes

* Fix checkstyle violation

* More unittest updates

* Use SourceProvider in MapperServiceTestCase

* Remove runtime field from unittest

* Update synthetic source doc
2023-09-26 14:29:56 +03:00
Abdon Pijpelink 5947f3b455
[DOCS] Clarify TSDS/synthetic source/runtime field restrictions (#97980) 2023-08-03 18:28:08 +02:00
Salvatore Campagna 0eeef45ea2
Synthetic source support for flattened fields (#94842)
Here we add synthetic source support for fields whose type is flattened.
Note that flattened fields and synthetic source have the following limitations,
all arising from the fact that in synthetic source we just see key/value pairs
when reconstructing the original object and have no type information in mappings:

* flattened fields use sorted set doc values of keywords, which means two things: 
   first we do not allow duplicate values, second we treat all values as keywords
* reconstructing array of objects results in nested objects (no array)
* reconstructing arrays with just one element results in a single-value field since we
   have no way to distinguish single-valued from multi-values fields other then looking
   at the count of values
2023-04-11 10:54:28 +02:00
Nik Everett b9bb7252be
Docs: synthetic _source can't params._source (#91630)
This documents that `params._source` isn't available for synthetic
`_source` indices and suggests to instead use `doc['foo']` or
`field('foo')`.
2022-11-22 15:23:30 -05:00
Nik Everett 7249e4bd93
Revert "Revert "Update tech preview notice for synthetic source (#91474)" (#91589)" (#91669)
This reverts commit ddef28bd2f.
2022-11-21 09:54:34 -05:00
Nik Everett 02138dc70a
Docs: synthetic _source can remove some arrays (#91632)
Synthetic _source's array flattening activities can remove some arrays
entirely. Specifically:
```
{
  "foo": [
    {
      "bar": 1
    },
    {
      "baz": 2
    }
  ]
}
```

Turns into:
```
{
  "foo": {
    "bar": 1,
    "baz": 2
  }
}
```

See, no more array! It's because the values are flattend to the leaf
fields and didn't have multiple values. This is implied by the docs we
had, but sure wasn't obvious. So now it's documented specifically.
2022-11-16 15:19:42 -05:00
David Kilfoyle ddef28bd2f
Revert "Update tech preview notice for synthetic source (#91474)" (#91589)
This reverts commit c9b13f5f53.
2022-11-15 09:33:40 -05:00
David Kilfoyle c9b13f5f53
Update tech preview notice for synthetic source (#91474) 2022-11-10 12:59:28 -05:00
Nik Everett 82aeb478db
Synthetic `_source`: support `wildcard` field (#90196)
This adds synthetic `_source` support for the `wildcard` field type.
2022-10-12 15:55:13 -04:00
Nik Everett eec7ba4737
Put synthetic source back in tech preview (#90371)
I got some new this morning that we're going to have to rework how we
handle ignore-above in synthetic _source which makes me a bit weary of
removing tech-preview in 8.5. I asked a few folks and they felt more
comfortable giving it a little longer in tech preview. I expect until
ignore-above is in.
2022-09-27 02:15:04 +09:30
Nik Everett 17967a98d3
Remove synthetic _source from tech preview (#90042)
I've been hacking on synthetic source for a while now and not seen any
need to break backwards compatibility or any major bugs. I think it's
time to remove the `preview` marker from it so folks can use it without
fear.
2022-09-13 16:33:10 -04:00
Alan Woodward 224f48e637
[DOCS] document that date and date_nanos fields support synthetic source (#89968) 2022-09-09 17:21:43 +01:00
Nik Everett c4a77d572d
Synthetic _source: support dense_vector (#89840)
This adds support for synthetic _source to `dense_vector` fields.

![image](https://user-images.githubusercontent.com/215970/188734496-0f0772c7-4c7a-46b6-b978-0c220e73474d.png)
2022-09-09 00:54:59 +09:30
Nik Everett b667aa33f0
Synthetic _source: support histogram field (#89833)
Adds support for the `histogram` field type to synthetic _source.

![image](https://user-images.githubusercontent.com/215970/188691249-9d23d1dc-64ab-49a4-8b24-f60fc966c0ac.png)
2022-09-08 01:55:38 +09:30
Nik Everett 104f4e9fb5
Synthetic _source: support version field type (#89706)
This adds support for synthetic _source to the `version` field type. It
works very similarly to `keyword` but with an extra decode step.

I modified the decoder to return a `BytesRef` instead of a `String`
because many of the callers seemed to be converting that string directly
into bytes again. Synthetic source would have wanted to do that. As was
the query infrastructure.
2022-08-30 09:39:50 -04:00
Nik Everett 914e216ebd
Prepare synthetic source docs for tech-preview (#89358)
Now that we're releasing synthetic _source as a tech preview feature, we
no longer want to remove the docs from the non-release builds. And we
want to mark all of the headings describing synthetic `_source` as a
preview.
2022-08-16 10:05:45 -04:00
Christos Soulios ad2dc834a7
Add `synthetic_source` support to `aggregate_metric_double` fields (#88909)
This PR implements synthetic_source support to the aggregate_metric_double
field type

Relates to #86603
2022-08-01 20:42:25 +03:00
Gilad Gal c35cfc9fca
Update synthetic-source.asciidoc (#88880)
* Update synthetic-source.asciidoc

* Update docs/reference/mapping/fields/synthetic-source.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
2022-07-28 10:35:10 +03:00
Alan Woodward 5c11a81913
Add 'mode' option to `_source` field mapper (#88211)
Currently we have two parameters that control how the source of a document
is stored, `enabled` and `synthetic`, both booleans. However, there are only
three possible combinations of these, with `enabled:false` and `synthetic:true`
being disallowed. To make this easier to reason about, this commit replaces
the `enabled` parameter with a new `mode` parameter, which can take the values
`stored`, `synthetic` and `disabled`. The `mode` parameter cannot be set
in combination with `enabled`, and we will subsequently move towards
deprecating `enabled` entirely.
2022-07-18 12:50:10 +01:00
Nik Everett b18bafb207
Docs for synthetic source (#87416)
This adds some basic docs for synthetic source both to get us started
documenting it and to show how I'd like to get it documented - with a
central section in the docs for `_source` and "satellite" sections in
each of the supported field types that link back to the central section.

[Preview](https://elasticsearch_87416.docs-preview.app.elstc.co/guide/en/elasticsearch/reference/master/mapping-source-field.html#synthetic-source)
2022-06-09 09:42:06 -04:00
Dan Hermann 0d21b032b6
[DOCS] Custom routing for data streams 2021-11-10 07:11:50 -06:00
Christoph Büscher f522de6b56
[Docs] Clarify ignore_above behaviour (#79705)
Clarify that `keyword` fields that exceed the optional `ignore_above` setting
are inlcuded in the `_ignored` fields since 7.14.

Closes #79605
2021-10-25 20:27:02 +02:00
James Rodewig 32a516807a
[DOCS] Update routing formulas (#76203)
The `_routing` metadata field docs currently include formulas for how
Elasticsearch routes documents to shards. However, these formulas were not
updated for #18699.  This updates the routing formulas and adds xrefs for
related settings.

Closes #76072
2021-08-09 11:42:33 -04:00
markharwood 3aee4c1f1f
New queryable "_tier" metadata field (#69288)
New _tier metadata field that supports term, terms, exists and wildcard queries on the first data tier preference stated for an index.

Closes #68135
2021-03-31 15:37:37 +01:00
James Rodewig 693807a6d3
[DOCS] Fix double spaces (#71082) 2021-03-31 09:57:47 -04:00
James Rodewig 5c75d004fa
[DOCS] Replace `put` with `create or update` in API names (#70330)
Co-authored-by: debadair <debadair@elastic.co>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-03-15 14:49:44 -04:00
James Rodewig 9af74ec561
[DOCS] Remove added admons (#69452) 2021-02-23 10:35:21 -05:00
James Rodewig 6f11752ffe
[DOCS] Fix `_doc_count` field title (#65704) 2020-12-02 08:13:43 -05:00
Gil Raphaelli b4233f0cd4
[DOCS] Fix _doc_count example typo (#65686) 2020-12-01 13:53:33 -05:00
Wylie Conlon 10ee0f2878
Clarify field data cache behavior in docs (#64375)
* Clarify that field data cache includes global ordinals
* Describe that the cache should be cleared once the limit is reached
* Clarify that the `_id` field does not supported aggregations anymore
* Fold the `fielddata` mapping parameter page into the `text field docs
* Improve cross-linking
2020-11-20 13:53:23 -08:00
Christos Soulios 4dc833fa44
Add doc_count field mapper (#64503)
Bucket aggregations compute bucket doc_count values by incrementing the doc_count by 1 for every document collected in the bucket.

When using summary fields (such as aggregate_metric_double) one field may represent more than one document. To provide this functionality we have implemented a new field mapper (named doc_count field mapper). This field is a positive integer representing the number of documents aggregated in a single summary field.

Bucket aggregations will check if a field of type doc_count exists in a document and will take this value into consideration when computing doc counts.
2020-11-03 17:47:17 +02:00
Alan Woodward 981258b02b
Remove TypeFieldMapper (#62838)
We don't need a special TypeFieldMapper for anything in particular; all access
to the type field can be done via a TypeFieldType that issues appropriate
deprecation warnings.

Relates to #41059
2020-09-30 15:47:29 +01:00
James Rodewig 29e957ecf8
[DOCS] Remove metrics sidebar in `_source` docs (#60777) 2020-08-05 15:57:29 -04:00
James Rodewig ae01606785
[DOCS] Replace `twitter` dataset in docs (#60604) 2020-08-03 12:49:56 -04:00
James Rodewig 441c3a21b1
[DOCS] Update my-index examples (#60132)
Changes the following example index names to `my-index-000001` for consistency:

* `my-index`
* `my_index`
* `myindex`
2020-07-27 14:46:39 -04:00
James Rodewig aa3ddfeefb
[DOCS] Move highlighting docs to separate page (#59768)
Moves the highlighting docs from the deprecated 'Request Body Search'
chapter to the new subpage of the 'Run a search chapter' section.

No substantive changes were made to the content.
2020-07-17 10:15:20 -04:00
James Rodewig cef242db20
[DOCS] Document custom routing support for data streams (#59323) 2020-07-09 16:35:44 -04:00
James Rodewig 7826bbee87
[DOCS] Move search API's `docvalue_fields` examples (#57760)
Changes:

* Condenses and relocates the `docvalue_fields` example to the 'Run a search' 
   page.
* Adds docs for the `docvalue_fields` request body parameter.
* Updates several related xrefs.

Co-authored-by: debadair <debadair@elastic.co>
2020-06-11 10:57:15 -04:00
James Rodewig 51e3d5ab63
[DOCS] Fix source filtering xrefs (#57720) 2020-06-05 08:46:26 -04:00
Julie Tibshirani 7161bd44cb
Remove the top-level 'mapping type' section. (#53374)
It seemed confusing for users that our top-level mapping page still had a
prominent section named 'Mapping Type'. This PR reworks the docs to remove this
reference and adds a note about types removal (similar to the note we added to
other APIs like put mapping).
2020-03-23 12:26:22 -07:00
C.J. Jameson 34feb3cde1
[DOCS] Clarify routing enforcement in docs (#53945)
Removes a mention of the `_doc` mapping type that's
no longer applicable now that mapping types are
removed/deprecated.
2020-03-23 12:34:49 -04:00
Alan Woodward 795a92707f
Remove deprecation warning when doc scripts refer to '_type' field (#53605)
We currently emit a warning in 8x when a script refers to the _type field for
a document. However, in 8x this field no longer exists, so the deprecation
warning is not required.

Relates to #41059
2020-03-18 11:50:36 +00:00
Julie Tibshirani 548fcd09fb
Stop ignoring types warnings in REST tests. (#49333)
In 7.x we added logic to the REST test harness to ignore warnings related to
types removal. This allowed us to continue to run mixed-cluster tests that
included 6.x nodes.

Now that master is on 8.x, we've no longer need to include 6.x nodes in testing
and have removed almost all typed calls. The logic to ignore warnings can
therefore be removed.
2019-11-20 08:23:56 -08:00
lgypro 95c7e6ebff [Docs] Fix syntax error leading to wrong doc ID (#48554)
In order to index a document with id 2, the "&" should be replaced by "?"
2019-10-29 10:26:33 +01:00
Julie Tibshirani 7e6199e7f2
Correct outdated information in _index docs. (#48436)
This PR makes the following updates:
* Update the supported query types to include `prefix` and `wildcard`.
* Specify that queries accept index aliases.
* Clarify that when querying on a remote index name, the separator `:` must be
  present.
2019-10-24 11:00:02 -07:00
Christoph Büscher 2351aa3efb
Disallow `_field_names` enabled setting (#46681)
After deprecating the `enabled` setting for `_field_names`starting with 7.5, 
this change disallows the setting on new indices in 8.0. The setting continues
to work for indices created in 7.x where we continue emitting a deprecation warning.

Relates to #42854
2019-09-23 14:13:44 +02:00
James Rodewig e355759086
[DOCS] Replace "// CONSOLE" comments with [source,console] (#46679) 2019-09-13 11:23:53 -04:00
Christoph Büscher d0a7bbcb69
Deprecate `_field_names` disabling (#42854)
Currently we allow `_field_names` fields to be disabled explicitely, but since
the overhead is negligible now we decided to keep it turned on by default and
deprecate the `enable` option on the field type. This change adds a deprecation
warning whenever this setting is used, going forward we want to ignore and finally
remove it.

Closes #27239
2019-09-11 14:55:30 +02:00
Anton 8167000951 [Docs] Fix typo in field-names-field.asciidoc (#46430) 2019-09-06 18:04:57 +02:00
James Rodewig 97802d8aff
[DOCS] Change // CONSOLE comments to [source,console] (#46441) 2019-09-06 10:55:16 -04:00