root
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Actions 1 Packages Projects Releases Wiki Activity
12,512 Commits 7 Branches 523 Tags 189 MiB
Commit Graph

1715 Commits

Author SHA1 Message Date
Harshavardhana e5c83535af
chore: upgrade deps (#19819) 
Signed-off-by: Harshavardhana <harsha@minio.io>
 2024-05-28 02:27:44 -07:00
Minio Trusted 8f266e0772 Update yaml files to latest version RELEASE.2024-05-27T19-17-46Z 2024-05-27 23:52:43 +00:00
Harshavardhana e0fe7cc391
fix: information disclosure bug in preconditions GET (#19810) 
precondition check was being honored before, validating
if anonymous access is allowed on the metadata of an
object, leading to metadata disclosure of the following
headers.

```
Last-Modified
Etag
x-amz-version-id
Expires:
Cache-Control:
```

although the information presented is minimal in nature,
and of opaque nature. It still simply discloses that an
object by a specific name exists or not without even having
enough permissions.
 2024-05-27 12:17:46 -07:00
Aditya Manthramurthy 5f78691fcf
ldap: Add user DN attributes list config param (#19758) 
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.

A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.

This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
 2024-05-24 16:05:23 -07:00
Shireesh Anjal a591e06ae5
Add cluster scanner metrics in metrics-v3 (#19517) 
endpoint: /minio/metrics/v3/cluster/scanner
metrics:
 - bucket_scans_finished (counter)
 - bucket_scans_started (counter)
 - directories_scanned (counter)
 - last_activity_nano_seconds (gauge)
 - objects_scanned (counter)
 - versions_scanned (counter)
 2024-05-24 12:29:25 -07:00
Shireesh Anjal 5659cddc84
Add cluster config metrics in metrics-v3 (#19507) 
endpoint: /minio/metrics/v3/cluster/config
metrics:
- write_quorum
- rrs_parity
- standard_parity
 2024-05-24 05:50:46 -07:00
Shubhendu 1654a9b7e6
Use point in time values for `gauge` metrics in graphs (#19690) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-05-24 04:11:51 -07:00
Shireesh Anjal 673a521711
Change endpoint of v3 notification metrics (#19804) 
from /cluster/notification to /notification
 2024-05-24 04:10:24 -07:00
Klaus Post b92ac55250
Add multipart combination to xl-meta (#19780) 
Add combination of multiple parts.

Parts will be reconstructed and saved separately and can manually be combined to the complete object.

Parts will be named `(version_id)-(filename).(partnum).(in)complete`.
 2024-05-23 09:37:31 -07:00
Shireesh Anjal 7981509cc8
Add cluster and bucket replication metrics in metrics-v3 (#19546) 
endpoint: /minio/metrics/v3/cluster/replication
metrics:
- average_active_workers
- average_queued_bytes
- average_queued_count
- average_transfer_rate
- current_active_workers
- current_transfer_rate
- last_minute_queued_bytes
- last_minute_queued_count
- max_active_workers
- max_queued_bytes
- max_queued_count
- max_transfer_rate
- recent_backlog_count

endpoint: /minio/metrics/v3/api/bucket/replication
metrics:
- last_hour_failed_bytes
- last_hour_failed_count
- last_minute_failed_bytes
- last_minute_failed_count
- latency_ms
- proxied_delete_tagging_requests_total
- proxied_get_requests_failures
- proxied_get_requests_total
- proxied_get_tagging_requests_failures
- proxied_get_tagging_requests_total
- proxied_head_requests_failures
- proxied_head_requests_total
- proxied_put_tagging_requests_failures
- proxied_put_tagging_requests_total
- sent_bytes
- sent_count
- total_failed_bytes
- total_failed_count
- proxied_delete_tagging_requests_failures
 2024-05-23 00:41:18 -07:00
Klaus Post 2c7bcee53f
Add cross-version remapped merges to xl-meta (#19765) 
Adds `-xver` which can be used with `-export` and `-combine` to attempt to combine files across versions if data is suspected to be the same. Overlapping data is compared.

Bonus: Make `inspect` accept wildcards.
 2024-05-19 08:31:54 -07:00
Poorna e947a844c9
Fix test scripts to use mc ready (#19768) 2024-05-18 11:19:01 -07:00
Shubhendu de4d3dac00
Added tests for IAM policies for bucket operations (#19734) 
* Added tests for bucket access policies

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

* move to correct category of tests

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

---------

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-05-14 08:43:07 -07:00
Shireesh Anjal 3bab4822f3
Add logger webhook metrics in metrics-v3 (#19515) 
endpoint: /minio/metrics/v3/cluster/webhook
metrics:
- failed_messages (counter)
- online (gauge)
- queue_length (gauge)
- total_messages (counter)
 2024-05-14 00:27:33 -07:00
Shireesh Anjal 5808190398
Add more metrics to v3/cluster/erasure-set (#19714) 
Metrics being added:

- read_tolerance: No of drive failures that can be tolerated without
  disrupting read operations
- write_tolerance: No of drive failures that can be tolerated without
  disrupting write operations
- read_health: Health of the erasure set in a pool for read operations
  (1=healthy, 0=unhealthy)
- write_health: Health of the erasure set in a pool for write operations
  (1=healthy, 0=unhealthy)
 2024-05-14 00:25:56 -07:00
Shireesh Anjal b2a82248b1
Move /system/go to /debug/go (#19707) 2024-05-14 00:25:37 -07:00
dependabot[bot] 4e5fcca8b9 build(deps): bump golang.org/x/net (#23) 
Bumps the go_modules group with 1 update in the /docs/debugging/s3-verify directory: [golang.org/x/net](https://github.com/golang/net).


Updates `golang.org/x/net` from 0.24.0 to 0.25.0
- [Commits](https://github.com/golang/net/compare/v0.24.0...v0.25.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-05-13 10:59:52 -07:00
Shireesh Anjal 074d70112d
Consolidate drive health related metrics into single metric (#19706) 
Instead of having "online" and "healing" as two metrics, replace with a
single metric "health" which can have following values:

0 = offline
1 = healthy
2 = healing
 2024-05-12 10:23:50 -07:00
Shireesh Anjal 60d7e8143a
Move /cluster/audit to /audit (#19708) 
As the audit metrics are server level and not 
overall cluster level.
 2024-05-10 07:50:39 -07:00
Shubhendu abae30f9e1
Added decom test with KES using sse-s3 and sse-kms (#19695) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-05-10 01:24:14 -07:00
Minio Trusted f9311bc9d1 Update yaml files to latest version RELEASE.2024-05-10T01-41-38Z 2024-05-10 02:00:49 +00:00
Harshavardhana 3549e583a6
results must be a single channel to avoid overwriting `healing.bin` (#19702) 2024-05-09 10:15:03 -07:00
Minio Trusted b9bdc17465 Update yaml files to latest version RELEASE.2024-05-07T06-41-25Z 2024-05-07 16:59:52 +00:00
Olli Janatuinen b413ff9fdb
Support user certificate based authentication on SFTP (#19650) 2024-05-06 23:41:25 -07:00
Harshavardhana 1526e7ece3
extend server config.yaml to support per pool set drive count (#19663) 
This is to support deployments migrating from a multi-pooled
wider stripe to lower stripe. MINIO_STORAGE_CLASS_STANDARD
is still expected to be same for all pools. So you can satisfy
adding custom drive count based pools by adjusting the storage
class value.

```
version: v2
address: ':9000'
rootUser: 'minioadmin'
rootPassword: 'minioadmin'
console-address: ':9001'
pools: # Specify the nodes and drives with pools
  -
    args:
        - 'node{11...14}.example.net/data{1...4}'
  -
    args:
        - 'node{15...18}.example.net/data{1...4}'
  -
    args:
        - 'node{19...22}.example.net/data{1...4}'
  -
    args:
        - 'node{23...34}.example.net/data{1...10}'
    set-drive-count: 6
```
 2024-05-03 08:54:03 -07:00
Shireesh Anjal 04f92f1291
Change endpoint format for per-bucket metrics (#19655) 
Per-bucket metrics endpoints always start with /bucket and the bucket
name is appended to the path. e.g. if the collector path is /bucket/api,
the endpoint for the bucket "mybucket" would be
/minio/metrics/v3/bucket/api/mybucket

Change the existing bucket api endpoint accordingly from /api/bucket to
/bucket/api
 2024-05-02 10:37:57 -07:00
Bala FA e5b16adb1c
Add cluster IAM metrics in metrics-v3 (#19595) 
Signed-off-by: Bala.FA <bala@minio.io>
 2024-05-02 01:20:42 -07:00
Aditya Manthramurthy f3d61c51fc
fix: Filter out cust. AssumeRole `Token` for audit (#19646) 
The `Token` parameter is a sensitive value that should not be output in the Audit log for STS AssumeRoleWithCustomToken API.

Bonus: Add a simple tool that echoes audit logs to the console.
 2024-05-01 14:31:13 -07:00
Minio Trusted a75f42344b Update yaml files to latest version RELEASE.2024-05-01T01-11-10Z 2024-05-01 02:45:52 +00:00
Klaus Post f64dea2aac
Allow custom SFTP algorithm selection (#19636) 
Algorithms are comma separated.
Note that valid values does not in all cases represent default values.

`--sftp=pub-key-algos=...` specifies the supported client public key
authentication algorithms. Note that this doesn't include certificate types
since those use the underlying algorithm. This list is sent to the client if
it supports the server-sig-algs extension. Order is irrelevant.

Valid values
```
ssh-ed25519
sk-ssh-ed25519@openssh.com
sk-ecdsa-sha2-nistp256@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
rsa-sha2-256
rsa-sha2-512
ssh-rsa
ssh-dss
```

`--sftp=kex-algos=...` specifies the supported key-exchange algorithms in preference order.

Valid values:

```
curve25519-sha256
curve25519-sha256@libssh.org
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521
diffie-hellman-group14-sha256
diffie-hellman-group16-sha512
diffie-hellman-group14-sha1
diffie-hellman-group1-sha1
```

`--sftp=cipher-algos=...` specifies the allowed cipher algorithms.
If unspecified then a sensible default is used.

Valid values:
```
aes128-ctr
aes192-ctr
aes256-ctr
aes128-gcm@openssh.com
aes256-gcm@openssh.com
chacha20-poly1305@openssh.com
arcfour256
arcfour128
arcfour
aes128-cbc
3des-cbc
```

`--sftp=mac-algos=...` specifies a default set of MAC algorithms in preference order.
This is based on RFC 4253, section 6.4, but with hmac-md5 variants removed because they have
reached the end of their useful life.

Valid values:

```
hmac-sha2-256-etm@openssh.com
hmac-sha2-512-etm@openssh.com
hmac-sha2-256
hmac-sha2-512
hmac-sha1
hmac-sha1-96
```
 2024-04-30 08:15:45 -07:00
Minio Trusted 2e38bb5175 Update yaml files to latest version RELEASE.2024-04-28T17-53-50Z 2024-04-29 17:09:28 +00:00
Shireesh Anjal 4caa3422bd
Add process metrics in `metrics-v3` (#19612) 
endpoint: /minio/metrics/v3/system/process
metrics:
- locks_read_total
- locks_write_total
- cpu_total_seconds
- go_routine_total
- io_rchar_bytes
- io_read_bytes
- io_wchar_bytes
- io_write_bytes
- start_time_seconds
- uptime_seconds
- file_descriptor_limit_total
- file_descriptor_open_total
- syscall_read_total
- syscall_write_total
- resident_memory_bytes
- virtual_memory_bytes
- virtual_memory_max_bytes

Since the standard process collector implements only a subset of these
metrics, remove it and implement our own custom process collector that
captures all the process metrics we need.
 2024-04-26 09:07:23 -07:00
Harshavardhana f4f1c42cba
deprecate usage of sha256-simd (#19621) 
go1.21 already implements the necessary optimizations
 2024-04-25 23:31:35 -07:00
Poorna e7aa26dc29
fix: allow DeleteObject unversioned objects with insufficient read quorum (#19581) 
Since the object is being permanently deleted, the lack of read quorum should not
matter as long as sufficient disks are online to complete the deletion with parity
requirements.

If several pools have the same object with insufficient read quorum, attempt to
delete object from all the pools where it exists
 2024-04-25 17:31:12 -07:00
Harshavardhana c54ffde568
add metrics ioerror counter for alerts on I/O errors (#19618) 2024-04-25 15:01:31 -07:00
Klaus Post fbfeb59658
xl-meta: Allow combining multiple unversioned objects (#19604) 
When inspecting files like `.minio.sys/pool.bin` that may be present on multiple sets, use signature to separate them.

Also fixes null versions to actually be useful with `-export -combine`.
 2024-04-24 10:56:22 -07:00
Bala FA 14cdadfb56
Add cluster notification metrics in metrics-v3 (#19533) 
Signed-off-by: Bala.FA <bala@minio.io>
 2024-04-23 21:10:35 -07:00
Shireesh Anjal f7b665347e
Add system CPU metrics to metrics-v3 (#19560) 
endpoint: /minio/metrics/v3/system/cpu

metrics:
- minio_system_cpu_avg_idle
- minio_system_cpu_avg_iowait
- minio_system_cpu_load
- minio_system_cpu_load_perc
- minio_system_cpu_nice
- minio_system_cpu_steal
- minio_system_cpu_system
- minio_system_cpu_user
 2024-04-23 16:56:12 -07:00
Harshavardhana 9693c382a8
make renameData() more defensive during overwrites (#19548) 
instead upon any error in renameData(), we still
preserve the existing dataDir in some form for
recoverability in strange situations such as out
of disk space type errors.

Bonus: avoid running list and heal() instead allow
versions disparity to return the actual versions,
uuid to heal. Currently limit this to 100 versions
and lesser disparate objects.

an undo now reverts back the xl.meta from xl.meta.bkp
during overwrites on such flaky setups.

Bonus: Save N depth syscalls via skipping the parents
upon overwrites and versioned updates.

Flaky setup examples are stretch clusters with regular
packet drops etc, we need to add some defensive code
around to avoid dangling objects.
 2024-04-23 10:15:52 -07:00
dependabot[bot] ca1350b092
build(deps): bump golang.org/x/net from 0.19.0 to 0.23.0 in /docs/debugging/s3-verify (#19559) 
build(deps): bump golang.org/x/net in /docs/debugging/s3-verify

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.19.0 to 0.23.0.
- [Commits](https://github.com/golang/net/compare/v0.19.0...v0.23.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-04-19 14:24:47 -07:00
Klaus Post 5f774951b1
Store object EC in metadata header (#19534) 
Keep the EC in header, so it can be retrieved easily for dynamic quorum calculations.

To not force a full metadata decode on every read the value will be 0/0 for data written in previous versions.

Size is expected to increase by 2 bytes per version, since all valid values can be represented with 1 byte each.

Example:
```
λ xl-meta xl.meta
{
  "Versions": [
    {
      "Header": {
        "EcM": 4,
        "EcN": 8,
        "Flags": 6,
        "ModTime": "2024-04-17T11:46:25.325613+02:00",
        "Signature": "0a409875",
        "Type": 1,
        "VersionID": "8e03504e11234957b2727bc53eda0d55"
      },
...
```

Not used for operations yet.
 2024-04-19 09:43:43 -07:00
Harshavardhana 2ca9befd2a
add ILM + site-replication tests (#19554) 2024-04-19 05:48:19 -07:00
Sveinn 108e6f92d4
updating tests to use new mc --enc flags (#19508) 2024-04-19 01:43:09 -07:00
Minio Trusted 01bfdf949a Update yaml files to latest version RELEASE.2024-04-18T19-09-19Z 2024-04-18 20:45:59 +00:00
Shireesh Anjal ca5fab8656
Add cluster audit metrics in metrics-v3 (#19514) 
endpoint: /minio/metrics/v3/cluster/audit
metrics:
- failed_messages (counter)
- total_messages (counter)
- target_queue_length (gauge)
 2024-04-17 02:18:02 -07:00
Shireesh Anjal 6df76ca73c
Add system memory metrics in v3 (#19486) 
Following memory metrics will be added under /system/memory

- available
- buffers
- cache
- free
- shared
- total
- used
- used_perc
 2024-04-16 22:10:25 -07:00
Markus Wagner 0cf3d93360
removed hardcoded datasource uid (#19477) 2024-04-15 03:03:01 -07:00
guangwu 87299eba10
fix: close sessionPolicyFile in the sts-assume-role example (#19428) 2024-04-12 09:09:55 -07:00
Shubhendu d3a07c29ba
Correct sample for node scrape configuration (#19491) 
As node metrics should be scraped per node basis, use a sample
configuartion using all the nodes in targets.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-04-12 08:49:30 -07:00
Klaus Post 5206c0e883
Inspect: Add error if no results (#19476) 
When no results match or another error occurs, add an error to the stream. Keep the "inspect-input.txt" as the only thing in the zip for reference.

Example:

```
λ mc support inspect --airgap myminio/testbucket/fjghfjh/**
mc: Using public key from C:\Users\klaus\mc\support_public.pem
File data successfully downloaded as inspect-data.enc

λ inspect inspect-data.enc
Using private key from support_private.pem
output written to inspect-data.zip
2024/04/11 14:10:51 next stream: GetRawData: No files matched the given pattern

λ unzip -l inspect-data.zip
Archive:  inspect-data.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      222  2024-04-11 14:10   inspect-input.txt
---------                     -------
      222                     1 file

λ
```

Modifies inspect to read until end of stream to report the error.

Bonus: Add legacy commandline params
 2024-04-11 14:22:47 -07:00
Harshavardhana 41ec038523
remove permission denied error for being drive error (#19478) 2024-04-11 14:22:15 -07:00
Shireesh Anjal 08d3d06a06
Add drive metrics in metrics-v3 (#19452) 
Add following metrics:

- used_inodes
- total_inodes
- healing
- online
- reads_per_sec
- reads_kb_per_sec
- reads_await
- writes_per_sec
- writes_kb_per_sec
- writes_await
- perc_util

To be able to calculate the `per_sec` values, we capture the IOStats-related 
data in the beginning (along with the time at which they were captured), 
and compare them against the current values subsequently. This is because 
dividing by "time since server uptime." doesn't work in k8s environments.
 2024-04-11 10:46:34 -07:00
Harshavardhana aa8d25797b
update versioning tests to cover CopyObject() (#19472) 
adds tests to cover #19468
 2024-04-11 02:50:52 -07:00
jiuker 9496c17e13
doc: add Content-Type to s3zip (#19455) 2024-04-10 09:28:27 -07:00
Minio Trusted 51fc145161 Update yaml files to latest version RELEASE.2024-04-06T05-26-02Z 2024-04-06 06:44:30 +00:00
Shubhendu d96d696841
Dont use deprecated angular (#19396) 
Support for Angular would be stopped with newer versions of grafana

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-04-03 19:01:53 -07:00
Minio Trusted 06929258bc Update yaml files to latest version RELEASE.2024-03-30T09-41-56Z 2024-03-30 18:32:38 +00:00
Shubhendu 468a9fae83
Enable replication of SSE-C objects (#19107) 
If site replication enabled across sites, replicate the SSE-C
objects as well. These objects could be read from target sites
using the same client encryption keys.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-03-28 10:44:56 -07:00
Shubhendu d87f91720b
Split the replication dashboard in cluster and node level (#19374) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-03-28 10:15:39 -07:00
Shubhendu d63e603040
Pre populate the server names using a query (#19367) 
User doesn't need to remember and enter the server values,
rather they can select from the pre populated list.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-03-28 08:14:26 -07:00
Shubhendu 3d4fc28ec9
Render node graphs by node (#19356) 
As total drives count, online vs offline are per node basis, its
corect to select node for which graphs need to be rendered.

Set prometheus scrape jobs to fetch metrics from all nodes. A sample
scrape job for node metrics could be as below

```
- job_name: minio-job-node
  bearer_token: <token>
  metrics_path: /minio/v2/metrics/node
  scheme: https
  tls_config:
    insecure_skip_verify: true
  static_configs:
  - targets: [tenant1-ss-0-0.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-1.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-2.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-3.tenant1-hl.tenant-ns.svc.cluster.local:9000]
```

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-03-27 10:41:08 -07:00
Minio Trusted 428155add9 Update yaml files to latest version RELEASE.2024-03-26T22-10-45Z 2024-03-26 23:12:03 +00:00
Shubhendu 53a14c7301
Adding dashboard for MinIO node metrics (#19329) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-03-26 08:01:28 -07:00
Minio Trusted 15b930be1f Update yaml files to latest version RELEASE.2024-03-21T23-13-43Z 2024-03-22 20:08:28 +00:00
Minio Trusted d2373d5d6c Update yaml files to latest version RELEASE.2024-03-15T01-07-19Z 2024-03-15 02:47:20 +00:00
Harshavardhana 062f0cffad
fix: do not look for non-existent bucket in decom tests (#19261) 2024-03-14 08:54:11 -07:00
Klaus Post a2f6252b2f
xl-meta: Add inline data bitrot check (#19240) 
When using `-data` also perform a bitrot check on the data.

Example:

```
λ xl-meta -data net.zip
{
        "minio-1.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-1.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-1.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-1.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}}
}
```
 2024-03-11 10:57:11 -07:00
Dennis Marttinen 6c964fede5
Improve handling of compression inclusion for objects (#19234) 2024-03-11 04:55:34 -07:00
Aditya Manthramurthy b2c5b75efa
feat: Add Metrics V3 API (#19068) 
Metrics v3 is mainly a reorganization of metrics into smaller groups of
metrics and the removal of internal aggregation of metrics received from
peer nodes in a MinIO cluster.

This change adds the endpoint `/minio/metrics/v3` as the top-level metrics
endpoint and under this, various sub-endpoints are implemented. These
are currently documented in `docs/metrics/v3.md`

The handler will serve metrics at any path
`/minio/metrics/v3/PATH`, as follows:

when PATH is a sub-endpoint listed above => serves the group of
metrics under that path; or when PATH is a (non-empty) parent 
directory of the sub-endpoints listed above => serves metrics
from each child sub-endpoint of PATH. otherwise, returns a no 
resource found error

All available metrics are listed in the `docs/metrics/v3.md`. More will
be added subsequently.
 2024-03-10 01:15:15 -08:00
Minio Trusted 2dfa9adc5d Update yaml files to latest version RELEASE.2024-03-10T02-53-48Z 2024-03-10 08:42:35 +00:00
Minio Trusted 8ab3dac4f2 Update yaml files to latest version RELEASE.2024-03-07T00-43-48Z 2024-03-07 06:25:40 +00:00
Poorna 837a2a3d4b
sr: use service account cred for claims check (#19209) 
PR #19111 overlaid service account secret with site replicator secret
during token claims check.

Fixes : #19206
 2024-03-06 16:19:24 -08:00
Harshavardhana 74ccee6619
avoid too much auditing during decom/rebalance make it more robust (#19174) 
there can be a sudden spike in tiny allocations,
due to too much auditing being done, also don't hang
on the

```
h.logCh <- entry
```

after initializing workers if you do not have a way to
dequeue for some reason.
 2024-03-06 03:43:16 -08:00
Minio Trusted e3e3d92241 Update yaml files to latest version RELEASE.2024-03-05T04-48-44Z 2024-03-05 06:20:43 +00:00
Aditya Manthramurthy 9a4d003ac7
Add common middleware to S3 API handlers (#19171) 
The middleware sets up tracing, throttling, gzipped responses and
collecting API stats.

Additionally, this change updates the names of handler functions in
metric labels to be the same as the name derived from Go lang reflection
on the handler name.

The metric api labels are now stored in memory the same as the handler
name - they will be camelcased, e.g. `GetObject` instead of `getobject`.

For compatibility, we lowercase the metric api label values when emitting the metrics.
 2024-03-04 10:05:56 -08:00
Minio Trusted 18b0b7299a Update yaml files to latest version RELEASE.2024-03-03T17-50-39Z 2024-03-03 18:33:43 +00:00
Ravind Kumar f3e7c42425
Update metrics list.md with new metrics from RELEASE.2024-01-05 (#19161) 2024-02-29 14:53:54 -08:00
Shubhendu f46bee242c
Re-organized grafana dashboards (#19157) 
Moved different dashboards to their specific directories. Also
mentioned that these dashbards are examples of how to create
graphs using MinIO provided and metrics and customers should
change / add graphs on their specific need basis.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-02-29 10:35:20 -08:00
Anis Eleuch 828d4df6f0
debug: Add --search to print only specific goroutines (#19158) 
Easier to filter goroutines belonging to a specific subsystem
 2024-02-29 08:28:18 -08:00
Poorna b1351e2dee sr: use site replicator svcacct to sign STS session tokens (#19111) 
This change is to decouple need for root credentials to match between
 site replication deployments.

 Also ensuring site replication config initialization is re-tried until
 it succeeds, this deoendency is critical to STS flow in site replication
 scenario.
 2024-02-26 13:30:28 -08:00
Minio Trusted 85bcb5874a Update yaml files to latest version RELEASE.2024-02-26T09-33-48Z 2024-02-26 10:28:02 +00:00
Minio Trusted b49ce1713f Update yaml files to latest version RELEASE.2024-02-24T17-11-14Z 2024-02-25 05:23:21 +00:00
schmittey c44f311c4f
Add missing yaml syntax highlighting in prometheus README.md (#19087) 2024-02-20 16:22:37 -08:00
Shubhendu cb7dab17cb
Graph cluster and bucket replication proxied requests (#19078) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-02-20 01:45:00 -08:00
Minio Trusted 23c10350f3 Update yaml files to latest version RELEASE.2024-02-17T01-15-57Z 2024-02-17 01:37:10 +00:00
Harshavardhana b6e98aed01
fix: found races in accessing globalLocalDrives (#19069) 
make a copy before accessing globalLocalDrives

Bonus: update console v0.46.0

Signed-off-by: Harshavardhana <harsha@minio.io>
 2024-02-16 17:15:57 -08:00
Minio Trusted b5791e6f28 Update yaml files to latest version RELEASE.2024-02-14T21-36-02Z 2024-02-14 21:51:12 +00:00
Harshavardhana 00cb58eaf3
add customer specific hotfixes to 'registry.min.dev' (#19057) 
```
REPO="registry.min.dev/<customer>" CRED_DIR=/media/builder/minio make docker-hotfix-push
```
 2024-02-14 13:36:02 -08:00
Poorna 912a0031b7
fix sr tests to capture all server logs (#19051) 2024-02-13 20:51:23 -08:00
Minio Trusted f8e15e7d09 Update yaml files to latest version RELEASE.2024-02-13T15-35-11Z 2024-02-13 16:01:38 +00:00
Harshavardhana 0c068b15c7
add missing handler for reloading site replication config on peers (#19042) 2024-02-13 06:55:54 -08:00
Minio Trusted 6b63123ca9 Update yaml files to latest version RELEASE.2024-02-12T21-02-27Z 2024-02-12 21:40:49 +00:00
Harshavardhana eac4e4b279
honor replaced disk properly by updating globalLocalDrives (#19038) 
globalLocalDrives seem to be not updated during the
HealFormat() leads to a requirement where the server
needs to be restarted for the healing to continue.
 2024-02-12 13:00:20 -08:00
Minio Trusted 8e69f3cb89 Update yaml files to latest version RELEASE.2024-02-09T21-25-16Z 2024-02-09 22:48:07 +00:00
Harshavardhana 997ba3a574
introduce reader deadlines for net.Conn (#19023) 
Bonus: set "retry-after" header for AWS SDKs if possible to honor them.
 2024-02-09 13:25:16 -08:00
Aditya Manthramurthy a29c66ed74
Update IAM access manager plugin demo (#19007) 
Now prints the JSON payload for easier debugging.
 2024-02-08 09:15:20 -08:00
Minio Trusted eead4db1d2 Update yaml files to latest version RELEASE.2024-02-06T21-36-22Z 2024-02-07 12:11:36 +00:00
Poorna 27d02ea6f7
metrics: add replication metrics on proxied requests (#18957) 2024-02-05 22:00:45 -08:00
Minio Trusted 8414aff424 Update yaml files to latest version RELEASE.2024-02-04T22-36-13Z 2024-02-05 00:46:41 +00:00
Minio Trusted ee0055b929 Update yaml files to latest version RELEASE.2024-01-31T20-20-33Z 2024-01-31 20:35:33 +00:00
Daniel Valdivia 403ec7cf21
fix: metrics URI path in prometheus docs (#18907) 2024-01-29 14:34:21 -08:00
Minio Trusted ff4f4d4649 Update yaml files to latest version RELEASE.2024-01-29T03-56-32Z 2024-01-29 05:33:23 +00:00
Minio Trusted ff8269575a Update yaml files to latest version RELEASE.2024-01-28T22-35-53Z 2024-01-29 01:22:56 +00:00
Harshavardhana 944f3c1477
remove local disk metrics from cluster metrics (#18886) 
local disk metrics were polluting cluster metrics
Please remove them instead of adding relevant ones.

- batch job metrics were incorrectly kept at bucket
  metrics endpoint, move it to cluster metrics.

- add tier metrics to cluster peer metrics from the node.

- fix missing set level cluster health metrics
 2024-01-28 12:53:59 -08:00
Cesar N 1a91edecae
Update list.md node_cpu wording (#18878) 2024-01-26 18:57:58 -08:00
Harshavardhana e11d851aee
add new drive I/O waiting/tokens metric (#18836) 
Bonus: add virtual memory used as well part of the system resource metrics.
 2024-01-19 14:51:36 -08:00
Minio Trusted c66c5828ea Update yaml files to latest version RELEASE.2024-01-18T22-51-28Z 2024-01-19 11:35:32 +00:00
Harshavardhana dd2542e96c
add codespell action (#18818) 
Original work here, #18474,  refixed and updated.
 2024-01-17 23:03:17 -08:00
Harshavardhana 9588978028
fix: HealBucket regression for empty buckets, simplify it (#18815) 2024-01-17 15:19:09 -08:00
Minio Trusted 8ccd1ee34a Update yaml files to latest version RELEASE.2024-01-16T16-07-38Z 2024-01-16 16:33:40 +00:00
Klaus Post ca258c04cb
xl-meta: Clean output (#18794) 
* Sort files.
* Sort maps.
* Remove annoying stray ','
* Hide -ndjson - doesn't do what it claims.

Sample output:

```
{
	"/tmp/xl1/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":0,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":5,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl10/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":14,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl11/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":0,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":15,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl12/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":16,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl13/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":1,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl14/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":2,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl15/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":3,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl16/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":4,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl2/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":6,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl3/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":7,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl4/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":8,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl5/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":9,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl6/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":10,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl7/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":11,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl8/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":12,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl9/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":13,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]}
}
```
 2024-01-16 08:07:38 -08:00
Harshavardhana 38637897ba
fix: listing SSE encrypted multipart objects (#18786) 
GetActualSize() was heavily relying on o.Parts()
to be non-empty to figure out if the object is multipart or not, 
However, we have many indicators of whether an object is multipart 
or not.

Blindly assuming that o.Parts == nil is not a multipart, is an 
incorrect expectation instead, multipart must be obtained via

- Stored metadata value indicating this is a multipart encrypted object.

- Rely on <meta>-actual-size metadata to get the object's actual size.
  This value is preserved for additional reasons such as these.

- ETag != 32 length
 2024-01-15 00:57:49 -08:00
Minio Trusted c727c8b684 Update yaml files to latest version RELEASE.2024-01-13T07-53-03Z 2024-01-13 09:24:52 +00:00
Minio Trusted 5ce68ad7fd Update yaml files to latest version RELEASE.2024-01-11T07-46-16Z 2024-01-11 09:39:57 +00:00
Shubhendu e31081d79d
Heal buckets at node level (#18612) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-01-09 20:34:04 -08:00
Minio Trusted a0b10c05e5 Update yaml files to latest version RELEASE.2024-01-05T22-17-24Z 2024-01-06 06:51:06 +00:00
Harshavardhana f4710948c4
fix: an odd crash when deleting `null` DEL markers (#18727) 
fixes #18724

A regression was introduced in #18547, that attempted
to file adding a missing `null` marker however we
should not skip returning based on versionID instead
it must be based on if we are being asked to create
a DEL marker or not.

The PR also has a side-affect for replicating `null`
marker permanent delete, as it may end up adding a
`null` marker while removing one.

This PR should address both scenarios.
 2024-01-02 15:08:18 -08:00
Shubhendu 9434fff215
Added list of scanner metrics to document (#18731) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2024-01-02 10:41:33 -08:00
Minio Trusted 695962fae8 Update yaml files to latest version RELEASE.2024-01-01T16-36-33Z 2024-01-02 01:46:48 +00:00
Tsuzuki Tsuchiya c1cae51fb5
doc: fix the name of RoleArn param in AssumeRoleWithWebIdentity requests (#18711) 2023-12-30 13:25:15 -08:00
Harshavardhana a50ea92c64
feat: introduce list_quorum="auto" to prefer quorum drives (#18084) 
NOTE: This feature is not retro-active; it will not cater to previous transactions
on existing setups. 

To enable this feature, please set ` _MINIO_DRIVE_QUORUM=on` environment
variable as part of systemd service or k8s configmap. 

Once this has been enabled, you need to also set `list_quorum`. 

```
~ mc admin config set alias/ api list_quorum=auto` 
```

A new debugging tool is available to check for any missing counters.
 2023-12-29 15:52:41 -08:00
Mario Bros fbd8dfe60f
Adding ~ to match job when multiple jobs (#18706) 2023-12-27 15:39:20 -08:00
Harshavardhana 60aff22931
clarify maximum number of buckets (#18718) 2023-12-27 13:52:19 -08:00
Minio Trusted ba245c6c46 Update yaml files to latest version RELEASE.2023-12-23T07-19-11Z 2023-12-23 10:30:11 +00:00
Shubhendu 9d7660b409
Graph cluster wide where applicable (#18705) 
Graph the maximum value reported across nodes at cluster
level for applicable scenarios.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-12-22 08:14:32 -08:00
Krishnan Parthasarathi 56b7045c20
Export tier metrics (#18678) 
minio_node_tier_ttlb_seconds - Distribution of time to last byte for streaming objects from warm tier
minio_node_tier_requests_success - Number of requests to download object from warm tier that were successful
minio_node_tier_requests_failure - Number of requests to download object from warm tier that failed
 2023-12-20 20:13:40 -08:00
Minio Trusted b1a109a611 Update yaml files to latest version RELEASE.2023-12-20T01-00-02Z 2023-12-20 08:43:46 +00:00
Harshavardhana f4389fb322
update dependencies for https://pkg.go.dev/vuln/GO-2023-2402 (#18676) 2023-12-18 18:35:20 -08:00
Anugrah Vijay 6acf038a84
docs: fix bucket metrics API\ path in docs (#18661) 2023-12-18 08:21:08 -08:00
Minio Trusted 54bc995f0a Update yaml files to latest version RELEASE.2023-12-14T18-51-57Z 2023-12-14 19:54:40 +00:00
Praveen raj Mani 10ca0a6936
Label the notification target metrics by their target IDs (#18633) 
This patch adds the targetID to the existing notification target metrics
and deprecates the current target metrics which points to the overall
event notification subsystem
 2023-12-14 09:09:26 -08:00
Minio Trusted 3b9a948045 Update yaml files to latest version RELEASE.2023-12-13T23-28-55Z 2023-12-14 03:04:03 +00:00
Poorna 3781a0f9ad
replication: Pass metadata timestamps in CopyObject call (#18647) 
Regression from #18285. CopyObject options were inheriting source MTime
for metadata timestamps if unspecified, removing this prevented metadata
updates from being applied on target.
 2023-12-13 15:28:55 -08:00
Poorna e79b289325
fix datadir missing check on HeadObject (#18646) 
versions pending purge in replication were seeing a errFileCorrupt
that prevents permanent deletion after replication.

Regression from PR#18477
 2023-12-13 14:54:01 -08:00
Shubhendu 6d4c1156d6
Changed the expression to render the value (#18627) 
The metrics `minio_bucket_replication_received_bytes` and
`minio_bucket_replication_sent_bytes` are additive in nature
and rendering the value as is looks fine.

Also added sort order for few graphs for better reading of tool
tips as keeping ones with highest value at top helps.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-12-13 10:05:47 -08:00
Minio Trusted 5fe7f9fa93 Update yaml files to latest version RELEASE.2023-12-09T18-17-51Z 2023-12-09 21:03:28 +00:00
Anis Eleuch 6f97663174
yml-config: Add support of rootUser and rootPassword (#18615) 
Users can define the root user and password in the yaml configuration
file; Root credentials defined in the environment variable still take
precedence
 2023-12-08 12:04:54 -08:00
Anis Eleuch 2e23e61a45
Add support of conf file to pass arguments and options (#18592) 2023-12-07 01:33:56 -08:00
Minio Trusted 9cdf490bc5 Update yaml files to latest version RELEASE.2023-12-07T04-16-00Z 2023-12-07 06:28:15 +00:00
Shireesh Anjal 7350a29fec
Capture percentage of cpu load and memory used (#18596) 
By default the cpu load is the cumulative of all cores. Capture the
percentage load (load * 100 / cpu-count)

Also capture the percentage memory used (used * 100 / total)
 2023-12-06 13:19:59 -08:00
Minio Trusted e99a597899 Update yaml files to latest version RELEASE.2023-12-06T09-09-22Z 2023-12-06 10:16:37 +00:00
Harshavardhana 73dde66dbe
stick to go1.19 go.mod (#18600) 2023-12-06 01:09:22 -08:00
Harshavardhana e98172d72d
avoid hot-tier SLA to be tied to warm-tier SLA (#18581) 
it is okay if the warm-tier cannot keep up, we should continue
to take I/O at hot-tier, only fail hot-tier or block it when
we are disk full.

Bonus: add metrics counter for these missed tasks, we will
know for sure if one of the node is lagging behind or is
losing too many tasks during transitioning.
 2023-12-02 13:02:12 -08:00
Minio Trusted f2d063e7b9 Update yaml files to latest version RELEASE.2023-12-02T10-51-33Z 2023-12-02 11:14:35 +00:00
jiuker 34187e047d
feat: support elasticsearch notification endpoint compression codec (#18562) 2023-11-30 00:25:03 -08:00
Shubhendu ce62980d4e
Fixed transition rules getting overwritten while healing (#18542) 
While healing the latest changes of expiry rules across sites
if target had pre existing transition rules, they were getting
overwritten as cloned latest expiry rules from remote site were
getting written as is. Fixed the same and added test cases as
well.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-11-28 10:38:35 -08:00
Harshavardhana fba883839d
feat: bring new HDD related performance enhancements (#18239) 
Optionally allows customers to enable 

- Enable an external cache to catch GET/HEAD responses 
- Enable skipping disks that are slow to respond in GET/HEAD 
  when we have already achieved a quorum
 2023-11-22 13:46:17 -08:00
Shubhendu 58306a9d34
Replicate Expiry ILM configs while site replication (#18130) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-11-21 09:48:06 -08:00
Harshavardhana a4cfb5e1ed
return errors if dataDir is missing during HeadObject() (#18477) 
Bonus: allow replication to attempt Deletes/Puts when
the remote returns quorum errors of some kind, this is
to ensure that MinIO can rewrite the namespace with the
latest version that exists on the source.
 2023-11-20 21:33:47 -08:00
Minio Trusted 8bedb419a9 Update yaml files to latest version RELEASE.2023-11-20T22-40-07Z 2023-11-21 00:54:16 +00:00