Adds a setting to disable detailed error messages and full exception stack traces
in HTTP responses. When set to false, the error_trace request parameter will result
in a HTTP 400 response. When the error_trace parameter is not present, the message
of the first ElasticsearchException will be output and no nested exception messages
will be output.
This commit adds the current total number of translog operations to the recovery reporting API. We also expose the recovered / total percentage:
```
"translog": {
"recovered": 536,
"total": 986,
"percent": "54.3%",
"total_time": "2ms",
"total_time_in_millis": 2
},
```
Closes#9368Closes#10042
The behaviour is better in the case someone has multiple levels of nested object fields defined in the mapping and like to define a single inner_hits definition that is two or more levels deep.
If someone wants inner hits on a nested field that is 2 levels deep the following would need to be defined:
```
{
...
"inner_hits" : {
"path" : {
"level1" : {
"inner_hits" : {
"path" : {
"level2" : {
"query" : { .... }
}
}
}
}
}
}
}
```
With this change the above can be defined as:
```
{
...
"inner_hits" : {
"path" : {
"level1.level2" : {
"query" : { .... }
}
}
}
}
```
Closes#9251
The analysis chain should be used instead of relying on this, as it is
confusing when dealing with different per-field analysers.
The `locale` option was only used for `lowercase_expanded_terms`, which,
once removed, is no longer needed, so it was removed as well.
Fixes#9978
Relates to #9973
This commit adds scripting capability to significant_terms.
Custom heuristics can be implemented with a script that provides
parameters subset_freq, superset_freq,subset_size, superset_size.
closes#7850
Changed search_type docs to reflect that the `(dfs_)query_and_fetch` modes are an internal optimization and should not be specified explicitly by the user.
Relates to #9606
As explained in elasticsearch/elasticsearch-mapper-attachments#101, we should have consistent documentation.
The best option is to link the documentation in elasticsearch guide to the most recent README in the plugin repo.
Closes#9756
The request tracer logs in TRACE level under the `transport.tracer` log and is dynamically configurable with include and exclude arrays to filter out unneeded info. By default all requests are logged with the exception of fault detection pings (fired every second).
add the notion of tracers in the MockTransportService for testing purposes
Closes#9286
To support the `_recovery` API, the recovery process keeps track of current progress in a class called RecoveryState. This class currently have some issues, mostly around concurrency (see #6644 ). This PR cleans it up as well as other issues around it:
- Make the Index subsection API cleaner:
- remove redundant information - all calculation is done based on the underlying file map
- clearer definition of what is what: total files, vs reused files (local files that match the source) vs recovered files (copied over). % based progress is reported based on recovered files only.
- cleaned up json response to match other API (sadly this breaks the structure). We now properly report human values for dates and other units.
- Add more robust unit testing
- Detail flag was passed along as state (it's now a ToXContent param)
- State lookup during reporting is now always done via the IndexShard , no more fall backs to many other classes.
- Cleanup APIs around time and move the little computations to the state class as opposed to doing them out of the API
I also improved error messages out of the REST testing infra for things I run into.
Closes#6644Closes#9811
Together with #8782 it should help in the situations simliar to #8887 by adding an ability to get information about currently running snapshot without accessing the repository itself.
Closes#8887
The number of current pending tasks is useful to detect and overloaded master. This commit adds it to the cluster health API. The complete list can be retrieved from the dedicated pending tasks API.
It also adds rest tests for the cluster health variants.
Closes#9877
There are two implications to this change.
First, percolator now uses _uid internally, extracting the id portion
when needed. Second, sorting on _id is no longer possible, since you
can no longer index _id. However, _uid can still be used to sort, and
is better anyways as indexing _id just to make it available to
fielddata for sorting is wasteful.
see #8143closes#9842
Double negatives are confusing, but a triple negative (1 no, 2 non, 3 null)? It takes five minutes to understand this little sentence. Cleaned that up a bit.
Closes#9789
This change removes the deprecated script parameter names ('file', 'id', and 'scriptField').
It also removes the ability to load file scripts using the 'script' parameter. File scripts should be loaded using the 'script_file' parameter only.
This was previously attempted in #8854. I revived that branch and did
some performance testing as was suggested in the comments there.
I fixed all the errors, mostly just the rest tests, which
needed to have http enabled on the node settings (the global cluster
previously had this always enabled). I also addressed the comments from
that issue.
My performance tests involved running the entire test suite on my
desktop which has 6 cores, 16GB of ram, and nothing else was being
run on the box at the time. I ran each set of settings 3 times and
took the average time.
| mode | master | patch | diff |
| ------- | ------ | ----- | ---- |
| local | 409s | 417s | +2% |
| network | 368s | 380s | +3% |
This increase in average time is clearly worthwhile to pay to achieve
isolation of tests. One caveat is the way I fixed the rest tests
is still to have one cluster for the entire suite, so all the rest
tests can still potentially affect each other, but this is an
issue for another day.
There were some oddities that I noticed while running these tests
that I would like to point out, as they probably deserve some
investigation (but orthogonal to this PR):
* The total test run times are highly variable (more than a minute between the min and max)
* Running in network mode is on average actually *faster* than local mode. How is this possible!?
This commit makes the `postings_format` and `doc_values_format` options of
mappings illegal on 2.0 and ignored on 1.x (meaning that the default postings
and doc values formats from the codec will be used in such a case).
This removes a fair amount of code.
Close#8746#9741
Squashed commit of the following:
commit 20835037c9
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 15:27:17 2015 -0700
Use Enum for "_primary" preference
commit 325acbe458
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 14:32:41 2015 -0700
Use ?preference=_primary automatically for realtime GET operations
commit edd49434af
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 14:32:06 2015 -0700
Move engine creation into protected createNewEngine method
commit 67a797a923
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 13:14:01 2015 -0700
Factor out AssertingSearcher so it can be used by mock Engines
commit 62b0c28df8
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 11:43:17 2015 -0700
Use IndexMetaData.isIndexUsingShadowReplicas helper
commit 1a0d456294
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 09:59:31 2015 -0700
Rename usesSharedFilesystem -> isOnSharedFilesystem
commit 73c62df4fc
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 09:58:02 2015 -0700
Add MockShadowEngine and hook it up to be used
commit c8e8db4738
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 09:45:50 2015 -0700
Clarify comment about pre-defined mappings
commit 60a4d5374a
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 09:18:22 2015 -0700
Add a test for shadow replicas that uses field data
commit 7346f9f382
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 08:37:14 2015 -0700
Revert changes to RecoveryTarget.java
commit d90d6980c9
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 08:35:44 2015 -0700
Rename `ownsShard` to `canDeleteShardContent`
commit 23001af834
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 08:35:25 2015 -0700
Remove ShadowEngineFactory, add .newReadOnlyEngine method in EngineFactory
commit b64fef1d2c
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 08:25:19 2015 -0700
Add warning that predefined mappings should be used
commit a1b8b8cf0d
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 17 14:31:50 2015 -0700
Remove unused import and fix index creation example in docs
commit 0b1b852365
Merge: b9d1feda22bd49
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 17 10:56:02 2015 -0700
Merge remote-tracking branch 'refs/remotes/origin/master' into shadow-replicas
commit b9d1fed25a
Merge: 4473e6341fd4d8
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 17 09:02:27 2015 -0700
Merge remote-tracking branch 'refs/remotes/origin/master' into shadow-replicas
commit 4473e63046
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 17 09:00:39 2015 -0700
Add asciidoc documentation for shadow replicas
commit eb699c19f0
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 16:15:39 2015 +0100
remove last nocommit
commit c5ece6d16d
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 16:13:12 2015 +0100
simplify shadow engine
commit 45cd34a12a
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 11:32:57 2015 +0100
fix tests
commit 744f228c19
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 11:28:12 2015 +0100
revert changes to IndexShardGateway - these are leftovers from previous iterations
commit 11886b7653
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 11:26:48 2015 +0100
Back out non-shared FS code. this will go in in a second iteration
commit 77fba571f1
Merge: e8ad6142e3c6a9
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 11:16:46 2015 +0100
Merge branch 'master' into shadow-replicas
Conflicts:
src/main/java/org/elasticsearch/index/engine/Engine.java
commit e8ad614673
Merge: 48a700d1b8d8da
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 10:54:20 2015 +0100
Merge branch 'master' into shadow-replicas
commit 48a700d23c
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 10:50:59 2015 +0100
add test for failing shadow engine / remove nocommit
commit d77414c5e7
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 10:27:56 2015 +0100
remove nocommits in IndexMetaData
commit abb696563a
Author: Simon Willnauer <simonw@apache.org>
Date: Mon Feb 16 17:05:02 2015 +0100
remove nocommit and simplify delete logic
commit 82b9f04491
Author: Simon Willnauer <simonw@apache.org>
Date: Mon Feb 16 16:45:27 2015 +0100
reduce the changes compared to master
commit 28f069b6d9
Author: Simon Willnauer <simonw@apache.org>
Date: Mon Feb 16 16:43:46 2015 +0100
fix primary relocation
commit c4c999dd61
Merge: 2ae80f9455a85d
Author: Simon Willnauer <simonw@apache.org>
Date: Mon Feb 16 15:04:26 2015 +0100
Merge branch 'master' into shadow-replicas
commit 2ae80f9689
Author: Lee Hinman <lee@writequit.org>
Date: Fri Feb 13 16:25:34 2015 -0700
throw UnsupportedOperationException on write operations in ShadowEngine
commit 740c28dd9e
Merge: e5bc047305ba33
Author: Lee Hinman <lee@writequit.org>
Date: Fri Feb 13 15:38:39 2015 -0700
Merge branch 'master' into shadow-replicas
commit e5bc047d7c
Author: Lee Hinman <lee@writequit.org>
Date: Fri Feb 13 11:38:09 2015 -0700
Don't replicate document request when using shadow replicas
commit 213292e067
Author: Simon Willnauer <simonw@apache.org>
Date: Fri Feb 13 13:58:05 2015 +0100
add one more nocommit
commit 83d171cf63
Merge: aea969209eb8d1
Author: Simon Willnauer <simonw@apache.org>
Date: Fri Feb 13 13:52:29 2015 +0100
Merge branch 'master' into shadow-replicas
commit aea96920d9
Author: Simon Willnauer <simonw@apache.org>
Date: Fri Feb 13 09:56:41 2015 +0100
revert unneeded changes on Store
commit ea4e3e58dc
Author: Lee Hinman <lee@writequit.org>
Date: Thu Feb 12 14:26:30 2015 -0700
Add documentation to ShadowIndexShard, remove nocommit
commit 4f71c8d9f7
Author: Lee Hinman <lee@writequit.org>
Date: Thu Feb 12 14:17:22 2015 -0700
Add documentation to ShadowEngine
commit 28a9d18427
Author: Lee Hinman <lee@writequit.org>
Date: Thu Feb 12 14:08:25 2015 -0700
Remove nocommit, document canDeleteIndexContents
commit d8d59dbf6d
Author: Lee Hinman <lee@writequit.org>
Date: Thu Feb 12 10:34:32 2015 -0700
Refactor more shared methods into the abstract Engine
commit a7eb53c1e8
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 17:38:59 2015 +0100
Simplify shared filesystem recovery by using a dedicated recovery handler that skip
most phases and enforces shard closing on the soruce before the target opens it's engine
commit a62b9a70ad
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 15:59:54 2015 +0100
fix compile error after upstream changes
commit abda7807bc
Merge: f22971935f6496
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 15:57:28 2015 +0100
Merge branch 'master' into shadow-replicas
Conflicts:
src/main/java/org/elasticsearch/index/engine/Engine.java
commit f2297199b7
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 12:41:32 2015 +0100
first cut at catchup from primary
make flush to a refresh
factor our ShadowIndexShard to have IndexShard be idential to the master and least intrusive
cleanup abstractions
commit 4a367c0750
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 09:50:36 2015 +0100
fix primary promotion
commit cf2fb807e7
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 16:45:41 2015 -0700
Make assertPathHasBeenCleared recursive
commit 5689b7d2f8
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 15:58:19 2015 -0700
Add testShadowReplicaNaturalRelocation
commit fdbe413353
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 15:28:57 2015 -0700
Use check for shared filesystem in primary -> primary relocation
Also adds a nocommit
commit 06e2eb4496
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 15:21:32 2015 -0700
Add a test checking that indices with shadow replicas clean up after themselves
commit e4dbfb09a6
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 15:08:18 2015 -0700
Fix segment info for ShadowEngine, remove test nocommit
commit 80cf0e884c
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 14:30:13 2015 -0700
Remove nocommit in ShadowEngineTests#testFailStart()
commit 5e33eeaca9
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 14:22:59 2015 -0700
Remove overly-complex test
commit 2378fbb917
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 13:45:44 2015 -0700
Fix missing import
commit 52e9cd1b83
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 13:45:05 2015 -0700
Add a test for replica -> primary promotion
commit a95adbeded
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 12:54:14 2015 -0700
Remove tests that don't apply to ShadowEngine
commit 1896feda9d
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 10:29:12 2015 -0700
Add testShadowEngineIgnoresWriteOperations and testSearchResultRelease
commit 67d7df41ea
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 10:06:05 2015 -0700
Add start of ShadowEngine unit tests
commit ca9beb2d93
Merge: 2d4273657a4646
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 18:03:53 2015 +0100
Merge branch 'master' into shadow-replicas
commit 2d42736fed
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 17:51:22 2015 +0100
shortcut recovery if we are on a shared FS - no need to compare files etc.
commit 24d36c92dd
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 17:08:08 2015 +0100
utilize the new delete code
commit 2a2eed10f5
Merge: 343dc0b173cfc1
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 16:07:41 2015 +0100
Merge branch 'master' into shadow-replicas
Conflicts:
src/main/java/org/elasticsearch/gateway/GatewayMetaState.java
commit 343dc0b527
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 16:05:28 2015 +0100
long adder is not available in java7
commit be02cabfee
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 10 22:04:24 2015 -0700
Add test that restarts nodes to ensure shadow replicas recover
commit 7fcb373f06
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 10 23:19:21 2015 +0100
make test more evil
commit 38135af0c1
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 10 22:25:11 2015 +0100
make tests pass
commit 05975af69e
Author: Lee Hinman <lee@writequit.org>
Date: Mon Jan 12 18:44:29 2015 +0100
Add ShadowEngine
javanna
jaymode
Joshua Rich
Boaz Leskes
Martijn van Groningen
Michael McCandless
Lee Hinman
Petr Bela
olivier bourgain
David Pilato
Britta Weber
Matias Tealdi
Clinton Gormley
Pius
Geoff Bourne
cgp
IsaacHaze
Ryan Ernst
Igor Motov
Colin Goodheart-Smithe
Michael Sander
Robert Muir
markharwood
Adrien Grand