root
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Actions 1 Packages Projects Releases Wiki Activity
minio/cmd/object-api-listobjects_test.go

644 lines
23 KiB
Go
Raw Normal View History

Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
/*
Add constants for commonly used values. (#3588) This is a consolidation effort, avoiding usage of naked strings in codebase. Whenever possible use constants which can be repurposed elsewhere. This also fixes `goconst ./...` reported issues.
2017-01-19 04:24:34 +08:00
* Minio Cloud Storage, (C) 2015-2016, 2017 Minio, Inc.
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
server: Move all the top level files into cmd folder. (#2490) This change brings a change which was done for the 'mc' package to allow for clean repo and have a cleaner github drop in experience.
2016-08-19 07:23:42 +08:00
package cmd
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
import (
"bytes"
Use context.Background() instead of nil Rename Context[Get|Set] -> [Get|Set]Context
2018-03-16 04:27:16 +08:00
"context"
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 13:30:34 +08:00
"crypto/md5"
"encoding/hex"
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
"fmt"
"io/ioutil"
posix: Deprecate custom removeAll/mkdirAll implementations. (#4808) Since go1.8 os.RemoveAll and os.MkdirAll both support long path names i.e UNC path on windows. The code we are carrying was directly borrowed from `pkg/os` package and doesn't need to be in our repo anymore. As a side affect this also addresses our codecoverage issue. Refer #4658
2017-08-13 10:25:43 +08:00
"os"
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
"strconv"
"strings"
"testing"
)
Cleanup: mispell fixes
2016-05-07 03:32:44 +08:00
// Wrapper for calling ListObjects tests for both XL multiple disks and single node setup.
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
func TestListObjects(t *testing.T) {
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
ExecObjectLayerTest(t, testListObjects)
}
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
// Unit test for ListObjects in general.
Object layer tests revamp for individual execution (#2134)
2016-07-08 06:05:51 +08:00
func testListObjects(obj ObjectLayer, instanceType string, t TestErrHandler) {
refactor: refactor code to separate fs into object-layer and fs layer. (#1305)
2016-04-09 01:37:38 +08:00
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
testBuckets := []string{
// This bucket is used for testing ListObject operations.
"test-bucket-list-object",
// Will not store any objects in this bucket,
// Its to test ListObjects on an empty bucket.
"empty-bucket",
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
for _, bucket := range testBuckets {
Use context.Background() instead of nil Rename Context[Get|Set] -> [Get|Set]Context
2018-03-16 04:27:16 +08:00
err := obj.MakeBucketWithLocation(context.Background(), bucket, "")
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
if err != nil {
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
t.Fatalf("%s : %s", instanceType, err.Error())
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
}
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
var err error
testObjects := []struct {
name string
content string
Use getObjectInfo() in both FS and XL ListObjects() to simplify and to return complete object information (#3247)
2016-11-14 03:48:02 +08:00
meta map[string]string
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
}{
Use getObjectInfo() in both FS and XL ListObjects() to simplify and to return complete object information (#3247)
2016-11-14 03:48:02 +08:00
{"Asia-maps.png", "asis-maps", map[string]string{"content-type": "image/png"}},
{"Asia/India/India-summer-photos-1", "contentstring", nil},
{"Asia/India/Karnataka/Bangalore/Koramangala/pics", "contentstring", nil},
{"newPrefix0", "newPrefix0", nil},
{"newPrefix1", "newPrefix1", nil},
{"newzen/zen/recurse/again/again/again/pics", "recurse", nil},
{"obj0", "obj0", nil},
{"obj1", "obj1", nil},
{"obj2", "obj2", nil},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{"z-empty-dir/", "", nil},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
}
for _, object := range testObjects {
Simplify data verification with HashReader. (#5071) Verify() was being called by caller after the data has been successfully read after io.EOF. This disconnection opens a race under concurrent access to such an object. Verification is not necessary outside of Read() call, we can simply just do checksum verification right inside Read() call at io.EOF. This approach simplifies the usage.
2017-10-22 13:30:34 +08:00
md5Bytes := md5.Sum([]byte(object.content))
Fix: Preserve MD5Sum for SSE encrypted objects (#6680) To conform with AWS S3 Spec on ETag for SSE-S3 encrypted objects, encrypt client sent MD5Sum and store it on backend as ETag.Extend this behavior to SSE-C encrypted objects.
2018-11-15 09:36:41 +08:00
_, err = obj.PutObject(context.Background(), testBuckets[0], object.name, mustGetPutObjReader(t, bytes.NewBufferString(object.content),
Move metadata into ObjectOptions for NewMultipart and PutObject (#7060)
2019-02-09 13:31:06 +08:00
int64(len(object.content)), hex.EncodeToString(md5Bytes[:]), ""), ObjectOptions{UserDefined: object.meta})
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
if err != nil {
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
t.Fatalf("%s : %s", instanceType, err.Error())
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
// Formualting the result data set to be expected from ListObjects call inside the tests,
// This will be used in testCases and used for asserting the correctness of ListObjects output in the tests.
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
resultCases := []ListObjectsInfo{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// ListObjectsResult-0.
// Testing for listing all objects in the bucket, (testCase 20,21,22).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
fs: Enable returning ETag along with ListObjects() (#4042) This is to comply with S3 behavior, we previously removed reading `fs.json` for optimization reasons but we have a reason to believe that providing ETag and using gjson provides needed benefit of not having to deal with unmarshalling overhead of golang stdlib. Fixes #4028
2017-04-05 00:14:03 +08:00
{Name: "Asia-maps.png"},
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "Asia/India/India-summer-photos-1"},
{Name: "Asia/India/Karnataka/Bangalore/Koramangala/pics"},
{Name: "newPrefix0"},
{Name: "newPrefix1"},
{Name: "newzen/zen/recurse/again/again/again/pics"},
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{Name: "z-empty-dir/"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-1.
// Used for asserting the truncated case, (testCase 23).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: true,
Objects: []ObjectInfo{
fs: Enable returning ETag along with ListObjects() (#4042) This is to comply with S3 behavior, we previously removed reading `fs.json` for optimization reasons but we have a reason to believe that providing ETag and using gjson provides needed benefit of not having to deal with unmarshalling overhead of golang stdlib. Fixes #4028
2017-04-05 00:14:03 +08:00
{Name: "Asia-maps.png"},
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "Asia/India/India-summer-photos-1"},
{Name: "Asia/India/Karnataka/Bangalore/Koramangala/pics"},
{Name: "newPrefix0"},
{Name: "newPrefix1"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-2.
// (TestCase 24).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: true,
Objects: []ObjectInfo{
fs: Enable returning ETag along with ListObjects() (#4042) This is to comply with S3 behavior, we previously removed reading `fs.json` for optimization reasons but we have a reason to believe that providing ETag and using gjson provides needed benefit of not having to deal with unmarshalling overhead of golang stdlib. Fixes #4028
2017-04-05 00:14:03 +08:00
{Name: "Asia-maps.png"},
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "Asia/India/India-summer-photos-1"},
{Name: "Asia/India/Karnataka/Bangalore/Koramangala/pics"},
{Name: "newPrefix0"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-3.
// (TestCase 25).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: true,
Objects: []ObjectInfo{
fs: Enable returning ETag along with ListObjects() (#4042) This is to comply with S3 behavior, we previously removed reading `fs.json` for optimization reasons but we have a reason to believe that providing ETag and using gjson provides needed benefit of not having to deal with unmarshalling overhead of golang stdlib. Fixes #4028
2017-04-05 00:14:03 +08:00
{Name: "Asia-maps.png"},
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "Asia/India/India-summer-photos-1"},
{Name: "Asia/India/Karnataka/Bangalore/Koramangala/pics"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-4.
// Again used for truncated case.
// (TestCase 26).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: true,
Objects: []ObjectInfo{
fs: Enable returning ETag along with ListObjects() (#4042) This is to comply with S3 behavior, we previously removed reading `fs.json` for optimization reasons but we have a reason to believe that providing ETag and using gjson provides needed benefit of not having to deal with unmarshalling overhead of golang stdlib. Fixes #4028
2017-04-05 00:14:03 +08:00
{Name: "Asia-maps.png"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-5.
// Used for Asserting prefixes.
// Used for test case with prefix "new", (testCase 27-29).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix0"},
{Name: "newPrefix1"},
{Name: "newzen/zen/recurse/again/again/again/pics"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-6.
// Used for Asserting prefixes.
// Used for test case with prefix = "obj", (testCase 30).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-7.
// Used for Asserting prefixes and truncation.
// Used for test case with prefix = "new" and maxKeys = 1, (testCase 31).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: true,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix0"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-8.
// Used for Asserting prefixes.
// Used for test case with prefix = "obj" and maxKeys = 2, (testCase 32).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: true,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "obj0"},
{Name: "obj1"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-9.
// Used for asserting the case with marker, but without prefix.
//marker is set to "newPrefix0" in the testCase, (testCase 33).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix1"},
{Name: "newzen/zen/recurse/again/again/again/pics"},
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{Name: "z-empty-dir/"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-10.
//marker is set to "newPrefix1" in the testCase, (testCase 34).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newzen/zen/recurse/again/again/again/pics"},
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{Name: "z-empty-dir/"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-11.
//marker is set to "obj0" in the testCase, (testCase 35).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "obj1"},
{Name: "obj2"},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{Name: "z-empty-dir/"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-12.
// Marker is set to "obj1" in the testCase, (testCase 36).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "obj2"},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{Name: "z-empty-dir/"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-13.
// Marker is set to "man" in the testCase, (testCase37).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix0"},
{Name: "newPrefix1"},
{Name: "newzen/zen/recurse/again/again/again/pics"},
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{Name: "z-empty-dir/"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-14.
// Marker is set to "Abc" in the testCase, (testCase 39).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
fs: Enable returning ETag along with ListObjects() (#4042) This is to comply with S3 behavior, we previously removed reading `fs.json` for optimization reasons but we have a reason to believe that providing ETag and using gjson provides needed benefit of not having to deal with unmarshalling overhead of golang stdlib. Fixes #4028
2017-04-05 00:14:03 +08:00
{Name: "Asia-maps.png"},
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "Asia/India/India-summer-photos-1"},
{Name: "Asia/India/Karnataka/Bangalore/Koramangala/pics"},
{Name: "newPrefix0"},
{Name: "newPrefix1"},
{Name: "newzen/zen/recurse/again/again/again/pics"},
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{Name: "z-empty-dir/"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-15.
// Marker is set to "Asia/India/India-summer-photos-1" in the testCase, (testCase 40).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "Asia/India/Karnataka/Bangalore/Koramangala/pics"},
{Name: "newPrefix0"},
{Name: "newPrefix1"},
{Name: "newzen/zen/recurse/again/again/again/pics"},
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{Name: "z-empty-dir/"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-16.
// Marker is set to "Asia/India/Karnataka/Bangalore/Koramangala/pics" in the testCase, (testCase 41).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix0"},
{Name: "newPrefix1"},
{Name: "newzen/zen/recurse/again/again/again/pics"},
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{Name: "z-empty-dir/"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-17.
// Used for asserting the case with marker, without prefix but with truncation.
// Marker = "newPrefix0" & maxKeys = 3 in the testCase, (testCase42).
// Output truncated to 3 values.
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: true,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix1"},
{Name: "newzen/zen/recurse/again/again/again/pics"},
{Name: "obj0"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-18.
// Marker = "newPrefix1" & maxkeys = 1 in the testCase, (testCase43).
// Output truncated to 1 value.
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: true,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newzen/zen/recurse/again/again/again/pics"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-19.
// Marker = "obj0" & maxKeys = 1 in the testCase, (testCase44).
// Output truncated to 1 value.
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: true,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "obj1"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-20.
// Marker = "obj0" & prefix = "obj" in the testCase, (testCase 45).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "obj1"},
{Name: "obj2"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-21.
// Marker = "obj1" & prefix = "obj" in the testCase, (testCase 46).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "obj2"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-22.
// Marker = "newPrefix0" & prefix = "new" in the testCase,, (testCase 47).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix1"},
{Name: "newzen/zen/recurse/again/again/again/pics"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-23.
// Prefix is set to "Asia/India/" in the testCase, and delimiter is not set (testCase 55).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "Asia/India/India-summer-photos-1"},
{Name: "Asia/India/Karnataka/Bangalore/Koramangala/pics"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-24.
// Prefix is set to "Asia" in the testCase, and delimiter is not set (testCase 56).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
fs: Enable returning ETag along with ListObjects() (#4042) This is to comply with S3 behavior, we previously removed reading `fs.json` for optimization reasons but we have a reason to believe that providing ETag and using gjson provides needed benefit of not having to deal with unmarshalling overhead of golang stdlib. Fixes #4028
2017-04-05 00:14:03 +08:00
{Name: "Asia-maps.png"},
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "Asia/India/India-summer-photos-1"},
{Name: "Asia/India/Karnataka/Bangalore/Koramangala/pics"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-25.
// Prefix is set to "Asia" in the testCase, and delimiter is set (testCase 57).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
fs: Enable returning ETag along with ListObjects() (#4042) This is to comply with S3 behavior, we previously removed reading `fs.json` for optimization reasons but we have a reason to believe that providing ETag and using gjson provides needed benefit of not having to deal with unmarshalling overhead of golang stdlib. Fixes #4028
2017-04-05 00:14:03 +08:00
{Name: "Asia-maps.png"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-26.
// prefix = "new" and delimiter is set in the testCase.(testCase 58).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix0"},
{Name: "newPrefix1"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-27.
// Prefix is set to "Asia/India/" in the testCase, and delimiter is set to forward slash '/' (testCase 59).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "Asia/India/India-summer-photos-1"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-28.
// Marker is set to "Asia/India/India-summer-photos-1" and delimiter set in the testCase, (testCase 60).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix0"},
{Name: "newPrefix1"},
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
// ListObjectsResult-29.
// Marker is set to "Asia/India/Karnataka/Bangalore/Koramangala/pics" in the testCase and delimeter set, (testCase 61).
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
IsTruncated: false,
Objects: []ObjectInfo{
Formatting issues fixed.
2016-03-22 18:18:22 +08:00
{Name: "newPrefix0"},
{Name: "newPrefix1"},
{Name: "obj0"},
{Name: "obj1"},
{Name: "obj2"},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
},
},
XL: ListObjects should not list when delimiter and prefix are '/'. (#1777)
2016-05-28 06:43:51 +08:00
// ListObjectsResult-30.
// Prefix and Delimiter is set to '/', (testCase 62).
{
IsTruncated: false,
Objects: []ObjectInfo{},
},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
testCases := []struct {
// Inputs to ListObjects.
bucketName string
prefix string
marker string
delimeter string
Fix testcase to not overflow int type (#4739) The int type is only 32-bits wide on 32-bit CPUs. Set the type in the tests to int32 to avoid setting problematic maxKeys values. Fixes #4738
2017-08-05 17:36:47 +08:00
maxKeys int32
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// Expected output of ListObjects.
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
result ListObjectsInfo
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
err error
// Flag indicating whether the test is expected to pass or not.
shouldPass bool
}{
// Test cases with invalid bucket names ( Test number 1-4 ).
Fix s3 compatibility fixes for getBucketLocation,headBucket,deleteBucket (#5842) - getBucketLocation - headBucket - deleteBucket Should return 404 or NoSuchBucket even for invalid bucket names, invalid bucket names are only validated during MakeBucket operation
2018-04-24 11:27:33 +08:00
{".test", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: ".test"}, false},
{"Test", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "Test"}, false},
{"---", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "---"}, false},
{"ad", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "ad"}, false},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// Using an existing file for bucket name, but its not a directory (5).
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
{"simple-file.txt", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "simple-file.txt"}, false},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// Valid bucket names, but they donot exist (6-8).
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
{"volatile-bucket-1", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "volatile-bucket-1"}, false},
{"volatile-bucket-2", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "volatile-bucket-2"}, false},
{"volatile-bucket-3", "", "", "", 0, ListObjectsInfo{}, BucketNotFound{Bucket: "volatile-bucket-3"}, false},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// Valid, existing bucket, but sending invalid delimeter values (9-10).
// Empty string < "" > and forward slash < / > are the ony two valid arguments for delimeter.
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
{"test-bucket-list-object", "", "", "*", 0, ListObjectsInfo{}, fmt.Errorf("delimiter '%s' is not supported", "*"), false},
{"test-bucket-list-object", "", "", "-", 0, ListObjectsInfo{}, fmt.Errorf("delimiter '%s' is not supported", "-"), false},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing for failure cases with both perfix and marker (11).
xl: Add stat calls to keep track of ignored errors. (#4117) Such that in a situation where all errors were ignored we need to reduce the errors using readQuorum to get a consistent error value. Without this change errors generated will never be consistent with for an expected scenario. For example in a 6 disk setup 1 disk is missing and 5 do not have the volume (testbucket) Without this change Stat() would result in different errors depending on which disk died. Can cause confusion to S3 client application. This change addresses need to track type of errors we ignored and bring readQuorum to choose the maximally occuring as the value of truth.
2017-04-14 16:46:16 +08:00
// The prefix and marker combination to be valid it should satisfy strings.HasPrefix(marker, prefix).
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
{"test-bucket-list-object", "asia", "europe-object", "", 0, ListObjectsInfo{}, fmt.Errorf("Invalid combination of marker '%s' and prefix '%s'", "europe-object", "asia"), false},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Setting a non-existing directory to be prefix (12-13).
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
{"empty-bucket", "europe/france/", "", "", 1, ListObjectsInfo{}, nil, true},
tests: Implemented more tests for fs-v1*.go (#2686)
2016-09-17 04:06:49 +08:00
{"empty-bucket", "africa/tunisia/", "", "", 1, ListObjectsInfo{}, nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing on empty bucket, that is, bucket without any objects in it (14).
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
{"empty-bucket", "", "", "", 0, ListObjectsInfo{}, nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Setting maxKeys to negative value (15-16).
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
{"empty-bucket", "", "", "", -1, ListObjectsInfo{}, nil, true},
{"empty-bucket", "", "", "", 1, ListObjectsInfo{}, nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Setting maxKeys to a very large value (17).
Fix testcase to not overflow int type (#4739) The int type is only 32-bits wide on 32-bit CPUs. Set the type in the tests to int32 to avoid setting problematic maxKeys values. Fixes #4738
2017-08-05 17:36:47 +08:00
{"empty-bucket", "", "", "", 111100000, ListObjectsInfo{}, nil, true},
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
// Testing for all 10 objects in the bucket (18).
{"test-bucket-list-object", "", "", "", 10, resultCases[0], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
//Testing for negative value of maxKey, this should set maxKeys to listObjectsLimit (19).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
{"test-bucket-list-object", "", "", "", -1, resultCases[0], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing for very large value of maxKey, this should set maxKeys to listObjectsLimit (20).
Fix testcase to not overflow int type (#4739) The int type is only 32-bits wide on 32-bit CPUs. Set the type in the tests to int32 to avoid setting problematic maxKeys values. Fixes #4738
2017-08-05 17:36:47 +08:00
{"test-bucket-list-object", "", "", "", 1234567890, resultCases[0], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing for trancated value (21-24).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
{"test-bucket-list-object", "", "", "", 5, resultCases[1], nil, true},
{"test-bucket-list-object", "", "", "", 4, resultCases[2], nil, true},
{"test-bucket-list-object", "", "", "", 3, resultCases[3], nil, true},
{"test-bucket-list-object", "", "", "", 1, resultCases[4], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing with prefix (25-28).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
{"test-bucket-list-object", "new", "", "", 3, resultCases[5], nil, true},
{"test-bucket-list-object", "new", "", "", 4, resultCases[5], nil, true},
{"test-bucket-list-object", "new", "", "", 5, resultCases[5], nil, true},
{"test-bucket-list-object", "obj", "", "", 3, resultCases[6], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing with prefix and truncation (29-30).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
{"test-bucket-list-object", "new", "", "", 1, resultCases[7], nil, true},
{"test-bucket-list-object", "obj", "", "", 2, resultCases[8], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing with marker, but without prefix and truncation (31-35).
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
{"test-bucket-list-object", "", "newPrefix0", "", 6, resultCases[9], nil, true},
{"test-bucket-list-object", "", "newPrefix1", "", 5, resultCases[10], nil, true},
{"test-bucket-list-object", "", "obj0", "", 4, resultCases[11], nil, true},
{"test-bucket-list-object", "", "obj1", "", 2, resultCases[12], nil, true},
{"test-bucket-list-object", "", "man", "", 11, resultCases[13], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Marker being set to a value which is greater than and all object names when sorted (36).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// Expected to send an empty response in this case.
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
{"test-bucket-list-object", "", "zen", "", 10, ListObjectsInfo{}, nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Marker being set to a value which is lesser than and all object names when sorted (37).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// Expected to send all the objects in the bucket in this case.
{"test-bucket-list-object", "", "Abc", "", 10, resultCases[14], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Marker is to a hierarhical value (38-39).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
{"test-bucket-list-object", "", "Asia/India/India-summer-photos-1", "", 10, resultCases[15], nil, true},
{"test-bucket-list-object", "", "Asia/India/Karnataka/Bangalore/Koramangala/pics", "", 10, resultCases[16], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing with marker and truncation, but no prefix (40-42).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
{"test-bucket-list-object", "", "newPrefix0", "", 3, resultCases[17], nil, true},
{"test-bucket-list-object", "", "newPrefix1", "", 1, resultCases[18], nil, true},
{"test-bucket-list-object", "", "obj0", "", 1, resultCases[19], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing with both marker and prefix, but without truncation (43-45).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// The valid combination of marker and prefix should satisfy strings.HasPrefix(marker, prefix).
{"test-bucket-list-object", "obj", "obj0", "", 2, resultCases[20], nil, true},
{"test-bucket-list-object", "obj", "obj1", "", 1, resultCases[21], nil, true},
{"test-bucket-list-object", "new", "newPrefix0", "", 2, resultCases[22], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Testing with maxKeys set to 0 (46-52).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// The parameters have to valid.
objectAPI: Fix object API interface, remove unnecessary structs. ObjectAPI changes. ``` ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error) ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error) ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error) CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error) ```
2016-04-03 16:34:20 +08:00
{"test-bucket-list-object", "", "obj1", "", 0, ListObjectsInfo{}, nil, true},
{"test-bucket-list-object", "", "obj0", "", 0, ListObjectsInfo{}, nil, true},
{"test-bucket-list-object", "new", "", "", 0, ListObjectsInfo{}, nil, true},
{"test-bucket-list-object", "obj", "", "", 0, ListObjectsInfo{}, nil, true},
{"test-bucket-list-object", "obj", "obj0", "", 0, ListObjectsInfo{}, nil, true},
{"test-bucket-list-object", "obj", "obj1", "", 0, ListObjectsInfo{}, nil, true},
{"test-bucket-list-object", "new", "newPrefix0", "", 0, ListObjectsInfo{}, nil, true},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// Tests on hierarchical key names as prefix.
// Without delimteter the code should recurse into the prefix Dir.
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Tests with prefix, but without delimiter (53-54).
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
{"test-bucket-list-object", "Asia/India/", "", "", 10, resultCases[23], nil, true},
{"test-bucket-list-object", "Asia", "", "", 10, resultCases[24], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Tests with prefix and delimiter (55-57).
typo: Fix typos across the codebase. (#2442)
2016-08-15 17:44:48 +08:00
// With delimeter the code should not recurse into the sub-directories of prefix Dir.
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
{"test-bucket-list-object", "Asia", "", "/", 10, resultCases[25], nil, true},
{"test-bucket-list-object", "new", "", "/", 10, resultCases[26], nil, true},
{"test-bucket-list-object", "Asia/India/", "", "/", 10, resultCases[27], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Test with marker set as hierarhical value and with delimiter. (58-59)
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
{"test-bucket-list-object", "", "Asia/India/India-summer-photos-1", "/", 10, resultCases[28], nil, true},
{"test-bucket-list-object", "", "Asia/India/Karnataka/Bangalore/Koramangala/pics", "/", 10, resultCases[29], nil, true},
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Test with prefix and delimiter set to '/'. (60)
XL: ListObjects should not list when delimiter and prefix are '/'. (#1777)
2016-05-28 06:43:51 +08:00
{"test-bucket-list-object", "/", "", "/", 10, resultCases[30], nil, true},
tests: Implemented more tests for fs-v1*.go (#2686)
2016-09-17 04:06:49 +08:00
// Test with invalid prefix (61)
api: Relax object name validation. (#2814) Fixes #2812
2016-10-01 07:56:36 +08:00
{"test-bucket-list-object", "\\", "", "/", 10, resultCases[30], ObjectNameInvalid{Bucket: "test-bucket-list-object", Object: "\\"}, false},
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
for i, testCase := range testCases {
Use context.Background() instead of nil Rename Context[Get|Set] -> [Get|Set]Context
2018-03-16 04:27:16 +08:00
result, err := obj.ListObjects(context.Background(), testCase.bucketName, testCase.prefix, testCase.marker, testCase.delimeter, int(testCase.maxKeys))
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
if err != nil && testCase.shouldPass {
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
t.Errorf("Test %d: %s: Expected to pass, but failed with: <ERROR> %s", i+1, instanceType, err.Error())
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
if err == nil && !testCase.shouldPass {
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
t.Errorf("Test %d: %s: Expected to fail with <ERROR> \"%s\", but passed instead", i+1, instanceType, testCase.err.Error())
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
// Failed as expected, but does it fail for the expected reason.
if err != nil && !testCase.shouldPass {
xl/fs: Split object layer into interface. (#1415)
2016-04-30 05:24:10 +08:00
if !strings.Contains(err.Error(), testCase.err.Error()) {
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
t.Errorf("Test %d: %s: Expected to fail with error \"%s\", but instead failed with error \"%s\" instead", i+1, instanceType, testCase.err.Error(), err.Error())
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
}
object: move go-routine listing from posix to objectLayer. (#1491)
2016-05-06 03:51:56 +08:00
// Since there are cases for which ListObjects fails, this is
// necessary. Test passes as expected, but the output values
// are verified for correctness here.
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
if err == nil && testCase.shouldPass {
object: move go-routine listing from posix to objectLayer. (#1491)
2016-05-06 03:51:56 +08:00
// The length of the expected ListObjectsResult.Objects
// should match in both expected result from test cases
// and in the output. On failure calling t.Fatalf,
// otherwise it may lead to index out of range error in
// assertion following this.
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
if len(testCase.result.Objects) != len(result.Objects) {
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
t.Fatalf("Test %d: %s: Expected number of object in the result to be '%d', but found '%d' objects instead", i+1, instanceType, len(testCase.result.Objects), len(result.Objects))
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
for j := 0; j < len(testCase.result.Objects); j++ {
if testCase.result.Objects[j].Name != result.Objects[j].Name {
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
t.Errorf("Test %d: %s: Expected object name to be \"%s\", but found \"%s\" instead", i+1, instanceType, testCase.result.Objects[j].Name, result.Objects[j].Name)
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
xl: Fix removing an empty directory (#6421) Removing an empty directory is not working because of xl.DeleteObject() was only checking if the passed prefix is an actual object but it should also check if it is an empty directory.
2018-09-06 07:38:03 +08:00
//FIXME: we should always check for ETag
if result.Objects[j].ETag == "" && !strings.HasSuffix(result.Objects[j].Name, slashSeparator) {
t.Errorf("Test %d: %s: Expected ETag to be not empty, but found empty instead (%v)", i+1, instanceType, result.Objects[j].Name)
Use getObjectInfo() in both FS and XL ListObjects() to simplify and to return complete object information (#3247)
2016-11-14 03:48:02 +08:00
}
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
if testCase.result.IsTruncated != result.IsTruncated {
Adding XL Object layer validation for existing unit tests of single node (#1499) object layer. Adding isBucketExist check for GetObjectInfo in the XL layer.
2016-05-07 02:57:04 +08:00
t.Errorf("Test %d: %s: Expected IsTruncated flag to be %v, but instead found it to be %v", i+1, instanceType, testCase.result.IsTruncated, result.IsTruncated)
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
s3: Don't set NextMarker when listing is not truncated (#7012) Setting NextMarker when IsTruncated is not set seems to be confusing AWS C++ SDK, this commit will avoid setting any string in NextMarker.
2018-12-21 05:30:25 +08:00
if testCase.result.IsTruncated && result.NextMarker == "" {
t.Errorf("Test %d: %s: Expected NextContinuationToken to contain a string since listing is truncated, but instead found it to be empty", i+1, instanceType)
}
if !testCase.result.IsTruncated && result.NextMarker != "" {
t.Errorf("Test %d: %s: Expected NextContinuationToken to be empty since listing is not truncated, but instead found `%v`", i+1, instanceType, result.NextMarker)
}
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
// Take ListObject treeWalk go-routine to completion, if available in the treewalk pool.
if result.IsTruncated {
Use context.Background() instead of nil Rename Context[Get|Set] -> [Get|Set]Context
2018-03-16 04:27:16 +08:00
_, err = obj.ListObjects(context.Background(), testCase.bucketName, testCase.prefix, result.NextMarker, testCase.delimeter, 1000)
tests: Using listObjects clean up remaining tree walk go routines. (#2278) * fs: Set nextMarker independent of it having a slash or not. * tests: Using listObjects clean up remaining tree walk go routines. * tests: Use slices to hold data instead of enumerating test cases by hand ... also fixed numbering of test cases.
2016-07-25 06:52:12 +08:00
if err != nil {
t.Fatal(err)
}
}
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
}
Fix List-Object Benchmark failure. (#3142)
2016-11-01 04:57:30 +08:00
// Initialize FS backend for the benchmark.
server: Add more elaborate startup messages. (#2731) These messages based on our prep stage during XL and prints more informative message regarding drive information. This change also does a much needed refactoring.
2016-10-06 03:48:07 +08:00
func initFSObjectsB(disk string, t *testing.B) (obj ObjectLayer) {
fs: Re-implement object layer to remember the fd (#3509) This patch re-writes FS backend to support shared backend sharing locks for safe concurrent access across multiple servers.
2017-01-17 09:05:00 +08:00
var err error
Add NAS gateway support (#5516)
2018-02-21 04:21:12 +08:00
obj, err = NewFSObjectLayer(disk)
server: Add more elaborate startup messages. (#2731) These messages based on our prep stage during XL and prints more informative message regarding drive information. This change also does a much needed refactoring.
2016-10-06 03:48:07 +08:00
if err != nil {
t.Fatal("Unexpected err: ", err)
}
return obj
}
Fix List-Object Benchmark failure. (#3142)
2016-11-01 04:57:30 +08:00
// BenchmarkListObjects - Run ListObject Repeatedly and benchmark.
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
func BenchmarkListObjects(b *testing.B) {
refactor: refactor code to separate fs into object-layer and fs layer. (#1305)
2016-04-09 01:37:38 +08:00
// Make a temporary directory to use as the obj.
tests: Use testTmpDir var to specify tmp directory (#3459) To be able to specify the directory where tests will be done. This way, it will be easier to run Minio tests on a mounted directory like NFS, ..
2016-12-16 14:25:05 +08:00
directory, err := ioutil.TempDir(globalTestTmpDir, "minio-list-benchmark")
xl/fs: Split object layer into interface. (#1415)
2016-04-30 05:24:10 +08:00
if err != nil {
b.Fatal(err)
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
}
posix: Deprecate custom removeAll/mkdirAll implementations. (#4808) Since go1.8 os.RemoveAll and os.MkdirAll both support long path names i.e UNC path on windows. The code we are carrying was directly borrowed from `pkg/os` package and doesn't need to be in our repo anymore. As a side affect this also addresses our codecoverage issue. Refer #4658
2017-08-13 10:25:43 +08:00
defer os.RemoveAll(directory)
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
refactor: refactor code to separate fs into object-layer and fs layer. (#1305)
2016-04-09 01:37:38 +08:00
// Create the obj.
server: Add more elaborate startup messages. (#2731) These messages based on our prep stage during XL and prints more informative message regarding drive information. This change also does a much needed refactoring.
2016-10-06 03:48:07 +08:00
obj := initFSObjectsB(directory, b)
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
Fix List-Object Benchmark failure. (#3142)
2016-11-01 04:57:30 +08:00
bucket := "ls-benchmark-bucket"
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
// Create a bucket.
Use context.Background() instead of nil Rename Context[Get|Set] -> [Get|Set]Context
2018-03-16 04:27:16 +08:00
err = obj.MakeBucketWithLocation(context.Background(), bucket, "")
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
if err != nil {
b.Fatal(err)
}
Fix List-Object Benchmark failure. (#3142)
2016-11-01 04:57:30 +08:00
// Insert objects to be listed and benchmarked later.
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
for i := 0; i < 20000; i++ {
key := "obj" + strconv.Itoa(i)
Move metadata into ObjectOptions for NewMultipart and PutObject (#7060)
2019-02-09 13:31:06 +08:00
_, err = obj.PutObject(context.Background(), bucket, key, mustGetPutObjReader(b, bytes.NewBufferString(key), int64(len(key)), "", ""), ObjectOptions{})
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
if err != nil {
b.Fatal(err)
}
}
b.ResetTimer()
// List the buckets over and over and over.
for i := 0; i < b.N; i++ {
Use context.Background() instead of nil Rename Context[Get|Set] -> [Get|Set]Context
2018-03-16 04:27:16 +08:00
_, err = obj.ListObjects(context.Background(), bucket, "", "obj9000", "", -1)
Fix for Istruncated set to true under certain conditions. Optimizing List Objects by using binary sort to discard entries in cases where prefix or marker is set. Adding test coverage to ListObjects. Adding benchmark to ListObjects.
2016-03-17 08:30:22 +08:00
if err != nil {
b.Fatal(err)
}
}
}