elasticsearch/docs/reference/analysis/tokenizers
Robert Muir b9a09c2b06 Analysis: Add additional Analyzers, Tokenizers, and TokenFilters from Lucene
Add `irish` analyzer
Add `sorani` analyzer (Kurdish)

Add `classic` tokenizer: specific to english text and tries to recognize hostnames, companies, acronyms, etc.
Add `thai` tokenizer: segments thai text into words.

Add `classic` tokenfilter: cleans up acronyms and possessives from classic tokenizer
Add `apostrophe` tokenfilter: removes text after apostrophe and the apostrophe itself
Add `german_normalization` tokenfilter: umlaut/sharp S normalization
Add `hindi_normalization` tokenfilter: accounts for hindi spelling differences
Add `indic_normalization` tokenfilter: accounts for different unicode representations in Indian languages
Add `sorani_normalization` tokenfilter: normalizes kurdish text
Add `scandinavian_normalization` tokenfilter: normalizes Norwegian, Danish, Swedish text
Add `scandinavian_folding` tokenfilter: much more aggressive form of `scandinavian_normalization`
Add additional languages to stemmer tokenfilter: `galician`, `minimal_galician`, `irish`, `sorani`, `light_nynorsk`, `minimal_nynorsk`

Add support access to default Thai stopword set "_thai_"

Fix some bugs and broken links in documentation.

Closes #5935
2014-07-03 05:47:49 -04:00
..
classic-tokenizer.asciidoc Analysis: Add additional Analyzers, Tokenizers, and TokenFilters from Lucene 2014-07-03 05:47:49 -04:00
edgengram-tokenizer.asciidoc [DOCS] Fixed some bad UTF8 2014-03-19 12:46:06 +01:00
keyword-tokenizer.asciidoc Update keyword-tokenizer.asciidoc 2014-05-07 15:04:07 +02:00
letter-tokenizer.asciidoc Migrated documentation into the main repo 2013-08-29 01:24:34 +02:00
lowercase-tokenizer.asciidoc Wrong link to Token Filter 2013-12-03 10:39:13 +01:00
ngram-tokenizer.asciidoc [DOCS] Fixed some bad UTF8 2014-03-19 12:46:06 +01:00
pathhierarchy-tokenizer.asciidoc Migrated documentation into the main repo 2013-08-29 01:24:34 +02:00
pattern-tokenizer.asciidoc Migrated documentation into the main repo 2013-08-29 01:24:34 +02:00
standard-tokenizer.asciidoc Migrated documentation into the main repo 2013-08-29 01:24:34 +02:00
thai-tokenizer.asciidoc Analysis: Add additional Analyzers, Tokenizers, and TokenFilters from Lucene 2014-07-03 05:47:49 -04:00
uaxurlemail-tokenizer.asciidoc Migrated documentation into the main repo 2013-08-29 01:24:34 +02:00
whitespace-tokenizer.asciidoc Migrated documentation into the main repo 2013-08-29 01:24:34 +02:00