root
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Actions 1 Packages Projects Releases Wiki Activity
minio/cmd/storage-rest-common.go

83 lines
3.2 KiB
Go
Raw Normal View History

Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-05 08:44:06 +08:00
/*
Add etcd part of config support, add noColor/json support (#8439) - Add color/json mode support for get/help commands - Support ENV help for all sub-systems - Add support for etcd as part of config
2019-10-30 15:04:39 +08:00
* MinIO Cloud Storage, (C) 2018-2019 MinIO, Inc.
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-05 08:44:06 +08:00
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package cmd
admin/heal: Fix deep healing to heal objects under more conditions (#8321) - Heal if the part.1 is truncated from its original size - Heal if the part.1 fails while being verified in between - Heal if the part.1 fails while being at a certain offset Other cleanups include make sure to flush the HTTP responses properly from storage-rest-server, avoid using 'defer' to improve call latency. 'defer' incurs latency avoid them in our hot-paths such as storage-rest handlers. Fixes #8319
2019-10-02 04:12:15 +08:00
const (
xl: Make it clear when to create delete marker for a non existant object (#11423)
2021-02-04 02:33:43 +08:00
storageRESTVersion = "v27" // Add force-delete-marker to DeleteVersion
Add data usage collect with its new admin API (#8553) Admin data usage info API returns the following (Only FS & XL, for now) - Number of buckets - Number of objects - The total size of objects - Objects histogram - Bucket sizes
2019-12-12 22:02:37 +08:00
storageRESTVersionPrefix = SlashSeparator + storageRESTVersion
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-05 01:30:59 +08:00
storageRESTPrefix = minioReservedBucketPath + "/storage"
admin/heal: Fix deep healing to heal objects under more conditions (#8321) - Heal if the part.1 is truncated from its original size - Heal if the part.1 fails while being verified in between - Heal if the part.1 fails while being at a certain offset Other cleanups include make sure to flush the HTTP responses properly from storage-rest-server, avoid using 'defer' to improve call latency. 'defer' incurs latency avoid them in our hot-paths such as storage-rest handlers. Fixes #8319
2019-10-02 04:12:15 +08:00
)
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-05 08:44:06 +08:00
const (
Add a custom healthcheck function for online status (#9858) - Add changes to ensure remote disks are not incorrectly taken online if their order has changed or are incorrect disks. - Bring changes to peer to detect disconnection with separate Health handler, to avoid a rather expensive call GetLocakDiskIDs() - Follow up on the same changes for Lockers as well
2020-06-18 05:49:26 +08:00
storageRESTMethodHealth = "/health"
Add data usage collect with its new admin API (#8553) Admin data usage info API returns the following (Only FS & XL, for now) - Number of buckets - Number of objects - The total size of objects - Objects histogram - Bucket sizes
2019-12-12 22:02:37 +08:00
storageRESTMethodDiskInfo = "/diskinfo"
storageRESTMethodCrawlAndGetDataUsage = "/crawlandgetdatausage"
storageRESTMethodMakeVol = "/makevol"
Improve disk formatting stage for large disk sets (#8690)
2019-12-24 08:31:03 +08:00
storageRESTMethodMakeVolBulk = "/makevolbulk"
Add data usage collect with its new admin API (#8553) Admin data usage info API returns the following (Only FS & XL, for now) - Number of buckets - Number of objects - The total size of objects - Objects histogram - Bucket sizes
2019-12-12 22:02:37 +08:00
storageRESTMethodStatVol = "/statvol"
storageRESTMethodDeleteVol = "/deletevol"
storageRESTMethodListVols = "/listvols"
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-05 08:44:06 +08:00
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-05 01:30:59 +08:00
storageRESTMethodAppendFile = "/appendfile"
storageRESTMethodCreateFile = "/createfile"
storageRESTMethodWriteAll = "/writeall"
Support bucket versioning (#9377) - Implement a new xl.json 2.0.0 format to support, this moves the entire marshaling logic to POSIX layer, top layer always consumes a common FileInfo construct which simplifies the metadata reads. - Implement list object versions - Migrate to siphash from crchash for new deployments for object placements. Fixes #2111
2020-06-13 11:04:01 +08:00
storageRESTMethodWriteMetadata = "/writemetadata"
storageRESTMethodDeleteVersion = "/deleteversion"
storageRESTMethodReadVersion = "/readversion"
storageRESTMethodRenameData = "/renamedata"
storageRESTMethodCheckParts = "/checkparts"
storageRESTMethodCheckFile = "/checkfile"
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-05 01:30:59 +08:00
storageRESTMethodReadAll = "/readall"
storageRESTMethodReadFile = "/readfile"
storageRESTMethodReadFileStream = "/readfilestream"
storageRESTMethodListDir = "/listdir"
Support bucket versioning (#9377) - Implement a new xl.json 2.0.0 format to support, this moves the entire marshaling logic to POSIX layer, top layer always consumes a common FileInfo construct which simplifies the metadata reads. - Implement list object versions - Migrate to siphash from crchash for new deployments for object placements. Fixes #2111
2020-06-13 11:04:01 +08:00
storageRESTMethodWalkVersions = "/walkversions"
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-05 01:30:59 +08:00
storageRESTMethodDeleteFile = "/deletefile"
Support bucket versioning (#9377) - Implement a new xl.json 2.0.0 format to support, this moves the entire marshaling logic to POSIX layer, top layer always consumes a common FileInfo construct which simplifies the metadata reads. - Implement list object versions - Migrate to siphash from crchash for new deployments for object placements. Fixes #2111
2020-06-13 11:04:01 +08:00
storageRESTMethodDeleteVersions = "/deleteverions"
Return appropriate errors API versions changes across REST APIs (#8480) This PR adds code to appropriately handle versioning issues that come up quite constantly across our API changes. Currently we were also routing our requests wrong which sort of made it harder to write a consistent error handling code to appropriately reject or honor requests. This PR potentially fixes issues - old mc is used against new minio release which is incompatible returns an appropriate for client action. - any older servers talking to each other, report appropriate error - incompatible peer servers should report error and reject the calls with appropriate error
2019-11-05 01:30:59 +08:00
storageRESTMethodRenameFile = "/renamefile"
storageRESTMethodVerifyFile = "/verifyfile"
ListObjects Metadata Caching (#10648) Design: https://gist.github.com/klauspost/025c09b48ed4a1293c917cecfabdf21c Gist of improvements: * Cross-server caching and listing will use the same data across servers and requests. * Lists can be arbitrarily resumed at a constant speed. * Metadata for all files scanned is stored for streaming retrieval. * The existing bloom filters controlled by the crawler is used for validating caches. * Concurrent requests for the same data (or parts of it) will not spawn additional walkers. * Listing a subdirectory of an existing recursive cache will use the cache. * All listing operations are fully streamable so the number of objects in a bucket no longer dictates the amount of memory. * Listings can be handled by any server within the cluster. * Caches are cleaned up when out of date or superseded by a more recent one.
2020-10-29 00:18:35 +08:00
storageRESTMethodWalkDir = "/walkdir"
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-05 08:44:06 +08:00
)
const (
fix: heal optimizations in crawler to avoid multiple healing attempts (#11173) Fixes two problems - Double healing when bitrot is enabled, instead heal attempt once in applyActions() before lifecycle is applied. - If applyActions() is successful and getSize() returns proper value, then object is accounted for and should be removed from the oldCache namespace map to avoid double heal attempts.
2020-12-29 02:31:00 +08:00
storageRESTVolume = "volume"
storageRESTVolumes = "volumes"
storageRESTDirPath = "dir-path"
storageRESTFilePath = "file-path"
xl: Make it clear when to create delete marker for a non existant object (#11423)
2021-02-04 02:33:43 +08:00
storageRESTForceDelMarker = "force-delete-marker"
fix: heal optimizations in crawler to avoid multiple healing attempts (#11173) Fixes two problems - Double healing when bitrot is enabled, instead heal attempt once in applyActions() before lifecycle is applied. - If applyActions() is successful and getSize() returns proper value, then object is accounted for and should be removed from the oldCache namespace map to avoid double heal attempts.
2020-12-29 02:31:00 +08:00
storageRESTVersionID = "version-id"
fix: readData in bulk call using messagepack byte wrappers (#11228) This PR refactors the way we use buffers for O_DIRECT and to re-use those buffers for messagepack reader writer. After some extensive benchmarking found that not all objects have this benefit, and only objects smaller than 64KiB see this benefit overall. Benefits are seen from almost all objects from 1KiB - 32KiB Beyond this no objects see benefit with bulk call approach as the latency of bytes sent over the wire v/s streaming content directly from disk negate each other with no remarkable benefits. All other optimizations include reuse of msgp.Reader, msgp.Writer using sync.Pool's for all internode calls.
2021-01-08 11:27:31 +08:00
storageRESTReadData = "read-data"
fix: heal optimizations in crawler to avoid multiple healing attempts (#11173) Fixes two problems - Double healing when bitrot is enabled, instead heal attempt once in applyActions() before lifecycle is applied. - If applyActions() is successful and getSize() returns proper value, then object is accounted for and should be removed from the oldCache namespace map to avoid double heal attempts.
2020-12-29 02:31:00 +08:00
storageRESTTotalVersions = "total-versions"
storageRESTSrcVolume = "source-volume"
storageRESTSrcPath = "source-path"
storageRESTDataDir = "data-dir"
storageRESTDstVolume = "destination-volume"
storageRESTDstPath = "destination-path"
storageRESTOffset = "offset"
storageRESTLength = "length"
storageRESTCount = "count"
storageRESTMarkerPath = "marker"
storageRESTPrefixFilter = "prefix"
fix: faster healing when disk is replaced. (#11520)
2021-02-19 03:06:54 +08:00
storageRESTForwardFilter = "forward"
fix: heal optimizations in crawler to avoid multiple healing attempts (#11173) Fixes two problems - Double healing when bitrot is enabled, instead heal attempt once in applyActions() before lifecycle is applied. - If applyActions() is successful and getSize() returns proper value, then object is accounted for and should be removed from the oldCache namespace map to avoid double heal attempts.
2020-12-29 02:31:00 +08:00
storageRESTRecursive = "recursive"
storageRESTReportNotFound = "report-notfound"
storageRESTBitrotAlgo = "bitrot-algo"
storageRESTBitrotHash = "bitrot-hash"
storageRESTDiskID = "disk-id"
storageRESTForceDelete = "force-delete"
Move remote disk StorageAPI abstraction from RPC to REST (#6464)
2018-10-05 08:44:06 +08:00
)