b2293c2bf4
XL: Rename, cleanup and add more comments. ( #1769 )
...
- xl-v1-bucket.go - removes a whole bunch of code.
- {xl-v1,fs-v1}-metadata.go - add a lot of comments and rename functions
appropriately.
2016-05-28 15:15:53 -07:00
3550660163
Return error for empty parts in multipartupload complete ( #1758 )
2016-05-28 15:15:53 -07:00
ee6645f421
XL: Add additional PartNumber variable as part of `xl.json` ( #1750 )
...
This is needed for verification of incoming parts and to
support variadic part uploads. Which should be sorted
properly.
Fixes #1740
2016-05-28 15:15:53 -07:00
b48b2e7f7c
Part ID check ( #1730 )
...
* Added check in PutObjectPartHandler to make sure part ID does not exceed 10000. ErrInvalidMaxParts written to response if part ID exceeds the maximum value.
2016-05-24 01:52:47 -07:00
9fdb69563d
handler: CopyObject should save metadata. ( #1698 )
...
- Content-Type
- Content-Encoding
- ETag
Fixes #1682
2016-05-19 17:10:08 -07:00
7d6ed50fc2
objects: Save all the incoming metadata properly. ( #1688 )
...
For both multipart and single put operation
2016-05-18 19:54:25 -07:00
9472299308
logging: Log only for unhandled errors, remove all the debug logging. ( #1652 )
...
This patch brings in the removal of debug logging altogether, instead
we bring in the functionality of being able to trace the errors properly
pointing back to the origination of the problem.
To enable tracing you need to enable "MINIO_TRACE" set to "1" or "true"
environment variable which would print back traces whenever there is an
error which is unhandled or at the handler layer.
By default this tracing is turned off and only user level logging is
provided.
2016-05-16 14:31:28 -07:00
498ce1e9bb
handler: Add a waitgroup to avoid expect100Continue crash. ( #1623 )
...
This waitgroup allows for safe blocking operation where we can cleanly
control the flow of the writes and the underlying pipe altogether.
Fixes #1553
2016-05-14 17:18:00 -07:00
5b29cefd40
api: DeleteObject should always return 204. ( #1645 )
...
Fixes #1643
2016-05-14 15:47:19 -07:00
d4745c7d6a
object: PutObjectHandler should set the md5Sum properly. ( #1604 )
...
Additionally add a test case as well for validating for us
to reply BadDigest properly.
Fixes #1603
2016-05-11 16:13:37 -07:00
26e2c4bf4d
Replacing fastsha256 with crypto/sha256 package from golang standard package ( #1584 )
2016-05-10 14:20:11 -07:00
091c1e8456
copyObject: No need to verify md5sum. ( #1520 )
...
Multipart objects are kept in non hex md5sum format.
This format doesn't comply with hex, so decoding
would fail invariably.
This is not necessary to validate and its not expected
error during a CopyObject operation.
Fixes #1519
2016-05-07 03:43:08 -07:00
751fa972f5
xl/fs: Multipart re-org introduce "uploads.json" ( #1505 )
...
Fixes #1457
2016-05-07 02:08:03 -07:00
17868ccd7f
handlers: overhaul entire writErrorResponse, simplify. ( #1472 )
2016-05-05 20:24:29 -07:00
4e34e03dd4
xl/fs: Split object layer into interface. ( #1415 )
2016-04-29 14:24:10 -07:00
a98a7fb1ad
Implement XL layer - preliminary work.
2016-04-25 12:47:31 -07:00
e0f8fed011
object: handle Error responses and handle errDiskFull. ( #1331 )
2016-04-19 02:42:10 -07:00
be002ac01e
fs/object: Fix issues from review comments.
2016-04-16 17:57:14 -07:00
8457af5708
fs: Add proper volume and path validation.
2016-04-16 16:25:53 -07:00
caa35f68fa
listMultipart: implement support for marker. ( #1313 )
2016-04-16 16:25:53 -07:00
30b0b4deba
storage/server/client: Enable storage server, enable client storage.
2016-04-16 16:25:53 -07:00
01a439f95b
refactor: add multipart code to the object layer.
2016-04-16 16:25:53 -07:00
3c48537f20
refactor: refactor code to separate fs into object-layer and fs layer. ( #1305 )
2016-04-16 16:25:53 -07:00
b182e94acc
signature: Handle presigned payload if set.
...
Validate payload with incoming content.
Fixes #1288
2016-04-07 03:04:18 -07:00
e8cd1aad8d
accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs + small fixes
...
* accessPolicy: copy object should require PutObject
* accessPolicy: cite mpu perms doc only for relevant operations
* accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs
2016-04-06 18:31:40 -07:00
8b4a5f07b4
accessPolicy: allow anonymous HEAD for Getable objects
...
* accessPolicy: allow anonymous HEAD for Getable objects
* accessPolicy: allow anonymous HEAD of Listable Buckets
2016-04-06 16:40:54 -07:00
c69fdf0cf2
listObjects: Cleanup and naming conventions.
...
- Marker should be escaped outside in handlers.
- Delimiter should be handled outside in handlers.
- Add missing comments and change the function names.
- Handle case of 'maxKeys' when its set to '0', its a valid
case and should be treated as such.
2016-04-04 19:55:07 -07:00
0479d4976b
objectAPI: Fix object API interface, remove unnecessary structs.
...
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
2016-04-03 15:25:01 -07:00
a6a4e7e297
signature: No need to validate region for getBucketLocation and listBuckets.
...
This type of check is added for making sure that we can support
custom regions.
ListBuckets and GetBucketLocation are always "us-east-1" rest
should look for the configured region.
Fixes #1278
2016-04-02 18:42:32 -07:00
efc80343e3
fs: Break fs package to top-level and introduce ObjectAPI interface.
...
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.
The new interface for any ObjectAPI layer is as below
```
// ObjectAPI interface.
type ObjectAPI interface {
// Bucket resource API.
DeleteBucket(bucket string) *probe.Error
ListBuckets() ([]BucketInfo, *probe.Error)
MakeBucket(bucket string) *probe.Error
GetBucketInfo(bucket string) (BucketInfo, *probe.Error)
// Bucket query API.
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)
// Object resource API.
GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
DeleteObject(bucket, object string) *probe.Error
// Object query API.
NewMultipartUpload(bucket, object string) (string, *probe.Error)
PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
2016-04-01 15:58:39 -07:00
02ad48466d
error: Signature errors should be returned with APIErrorCode.
...
The reasoning is that we can reply back with wide range of
S3 error responses, which would provide more richer context
to S3 client.
Fixes #1267
2016-03-31 23:28:40 -07:00
9dca46e156
signature: Use a layered approach for signature verification.
...
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.
In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
2016-03-26 15:21:05 -07:00
34f2c5bcdf
verify before writing
...
merge
verify headers before writing
2016-03-16 18:03:23 -07:00
9a5e3299fc
api/object: Add CopyObject to support match/modified copy headers
...
Adds support for the following request headers:
- x-amz-copy-source-if-match
- x-amz-copy-source-if-none-match
- x-amz-copy-source-if-unmodified-since
- x-amz-copy-source-if-modified-since
Fixes #1176
2016-03-12 10:54:23 -08:00
52751d81cb
cleanup: Rename ObjectMetadata as ObjectInfo.
...
Fixes #1215
2016-03-11 16:58:08 -08:00
fdf3d64793
error: Add proper prefixes for s3Error codes.
...
This patch adds 'Err' prefix for all standard API
error codes and also adds a proper type for them.
2016-03-10 18:38:46 -08:00
d5057b3c51
accessPolicy: Implement Put, Get, Delete access policy.
...
This patch implements Get,Put,Delete bucket policies
Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html
Currently supports following actions.
"*": true,
"s3:*": true,
"s3:GetObject": true,
"s3:ListBucket": true,
"s3:PutObject": true,
"s3:CreateBucket": true,
"s3:GetBucketLocation": true,
"s3:DeleteBucket": true,
"s3:DeleteObject": true,
"s3:AbortMultipartUpload": true,
"s3:ListBucketMultipartUploads": true,
"s3:ListMultipartUploadParts": true,
following conditions for "StringEquals" and "StringNotEquals"
"s3:prefix", "s3:max-keys"
2016-03-08 17:44:50 -08:00
aed62788d9
api: Implement multiple objects Delete api - fixes #956
...
This API takes input XML input in following form.
```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
<Quiet>true</Quiet>
<Object>
<Key>Key</Key>
</Object>
<Object>
<Key>Key</Key>
</Object>
...
</Delete>
```
and responds the list of successful deletes, list of errors
for all the deleted objects.
```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/ ">
<Deleted>
<Key>sample1.txt</Key>
</Deleted>
<Error>
<Key>sample2.txt</Key>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
</Error>
</DeleteResult>
```
2016-03-06 18:31:50 -08:00
adf74ffdb0
api: DRY code and add new test
...
This commit makes code cleaner and reduces the repetitions in the code
base. Specifically, it reduces the clutter in setObjectHeaders. It also
merges encodeSuccessResponse and encodeErrorResponse together because
they served no purpose differently. Finally, it adds a simple test for
generateRequestID.
2016-03-06 13:26:27 -08:00
f111997184
multipart: remove proper MD5, rather create MD5 based on parts to be s3 compatible.
...
This increases the performance phenominally.
2016-03-02 14:20:49 -08:00
17d145df3a
signature: Fix signature handling of parallel requests.
...
Signature struct should be immutable, this fixes an issue
with AWS cli not being able to do multipart put operations.
2016-03-02 11:49:50 -08:00
ee1b86e517
api: Implement support for additional request headers.
...
Now GetObject and HeadObject both support
- If-Modified-Since, If-Unmodified-Since
- If-Match, If-None-Match
request headers.
These headers are used to further handle the responses for GetObject
and HeadObject API.
Fixes #1098
2016-02-28 19:34:20 -08:00
3ff8a1b719
api: Implement CopyObject s3 API, doing server side copy.
...
Fixes #1172
2016-02-27 19:51:59 -08:00
91a092792a
presigned: Fix a bug in presigned request verification.
...
Additionally add Docker proxy configuration.
2016-02-18 02:23:12 -08:00
dd9aaa855c
web/rpc: Merge ports with API server.
...
Fixes #1081 and #1130
2016-02-17 20:28:15 -08:00
5a9333a67b
signature: Rewrite signature handling and move it into a library.
2016-02-16 17:28:16 -08:00
62f6ffb6db
xl: Moved to minio/minio - fixes #1112
2016-02-11 15:43:36 -08:00
99fbc0fcb3
getObject: Add support for special response headers.
...
Supports now response-content-type, response-content-disposition,
response-cache-control, response-expires.
2016-02-07 03:55:16 -08:00
a4c005ce30
multipart: Code cleanup
...
- More locking cleanup. Fix naming convention.
- Simplify concatenation and blocking calls.
2016-02-05 14:42:09 -08:00
7a3409c309
fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err'
...
- Replace the ACL checks back, remove them when bucket
policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
2016-02-04 13:43:52 -08:00
Harshavardhana
Krishnan Parthasarathi
Aakash Muttineni
karthic rao
Harshavardhana
Krishna Srinivas
Krishna Srinivas
Donald Guy
awwalker
Brendan Ashworth