root
/
gitlab-ce
mirror of https://gitlab.com/free-releases/gitlab-ce.git
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

Add latest changes from gitlab-org/gitlab@master

This commit is contained in:
GitLab Bot 2023-11-03 06:12:08 +00:00
parent 08c02e9cd5
commit ea044b0c4c
8 changed files with 101 additions and 67 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
.gitlab/CODEOWNERS
View File

@ -1445,23 +1445,6 @@ ee/lib/ee/api/entities/project.rb
/ee/app/views/shared/icons/_icon_audit_events_purple.svg
/ee/app/views/shared/promotions/_promote_audit_events.html.haml
/ee/app/workers/audit_events/audit_event_streaming_worker.rb
/ee/config/events/1652263097_groups__audit_events__index_click_streams_tab.yml
/ee/config/events/202108302307_admin_audit_logs_index_click_date_range_button.yml
/ee/config/events/202108302307_groups__audit_events_controller_search_audit_event.yml
/ee/config/events/202108302307_profiles_controller_search_audit_event.yml
/ee/config/events/202108302307_projects__audit_events_controller_search_audit_event.yml
/ee/config/events/202111041910_admin__audit_logs_controller_search_audit_event.yml
/ee/config/metrics/counts_28d/20210216183930_g_compliance_audit_events_monthly.yml
/ee/config/metrics/counts_28d/20210216183934_i_compliance_audit_events_monthly.yml
/ee/config/metrics/counts_28d/20210216183942_a_compliance_audit_events_api_monthly.yml
/ee/config/metrics/counts_28d/20211130085433_g_manage_compliance_audit_event_destinations.yml
/ee/config/metrics/counts_7d/20210216183906_g_compliance_audit_events.yml
/ee/config/metrics/counts_7d/20210216183908_i_compliance_audit_events.yml
/ee/config/metrics/counts_7d/20210216183912_a_compliance_audit_events_api.yml
/ee/config/metrics/counts_7d/20210216183928_g_compliance_audit_events_weekly.yml
/ee/config/metrics/counts_7d/20210216183932_i_compliance_audit_events_weekly.yml
/ee/config/metrics/counts_7d/20210216183940_a_compliance_audit_events_api_weekly.yml
/ee/config/metrics/counts_all/20211130085433_g_manage_compliance_audit_event_destinations.yml
/ee/lib/api/audit_events.rb
/ee/lib/audit/
/ee/lib/ee/api/entities/audit_event.rb

1
db/docs/activity_pub_releases_subscriptions.yml
View File

@ -7,4 +7,5 @@ feature_categories:
description: Stores subscriptions from external users through ActivityPub for project
releases
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/132889
milestone: '16.6'
gitlab_schema: gitlab_main

1
db/docs/p_ci_job_annotations.yml
View File

@ -6,4 +6,5 @@ feature_categories:
- build_artifacts
description: Stores user provided annotations for jobs. Currently storing extra information for a given job feed by API.
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/117319
milestone: '16.1'
gitlab_schema: gitlab_ci

116
doc/architecture/blueprints/secret_detection/index.md
View File

@ -26,28 +26,22 @@ job logs, and project management features such as issues, epics, and MRs.
### Goals
- Support asynchronous secret detection for the following scan targets:
- push events
- issuable creation
- issuable updates
- issuable comments
- Support platform-wide detection of tokens to avoid secret leaks
- Prevent exposure by rejecting detected secrets
- Provide scalable means of detection without harming end user experience
See [target types](#target-types) for scan target priorities.
### Non-Goals
The current proposal is limited to asynchronous detection and alerting only.
**Blocking** secrets on push events is high-risk to a critical path and
would require extensive performance profiling before implementing. See
[a recent example](https://gitlab.com/gitlab-org/gitlab/-/issues/246819#note_1164411983)
of a customer incident where this was attempted.
Initial proposal is limited to detection and alerting across plaform, with rejection only
during preceive Git interactions.
Secret revocation and rotation is also beyond the scope of this new capability.
Scanned object types beyond the scope of this MVC include:
- Media types (JPEGs, PDFs,...)
- Snippets
- Wikis
See [target types](#target-types) for scan target priorities.
#### Management UI
@ -69,7 +63,13 @@ which remain focused on active detection.
## Proposal
To achieve scalable secret detection for a variety of domain objects a dedicated
The first iteration of the experimental capability will feature a blocking
pre-receive hook implemented within the Rails application. This iteration
will be released in an experimental state to select users and provide
opportunity for the team to profile the capability before considering extraction
into a dedicated service.
In the future state, to achieve scalable secret detection for a variety of domain objects a dedicated
scanning service must be created and deployed alongside the GitLab distribution.
This is referred to as the `SecretScanningService`.
@ -94,10 +94,10 @@ as self-managed instances.
The critical paths as outlined under [goals above](#goals) cover two major object
types: Git blobs (corresponding to push events) and arbitrary text blobs.
The detection flow for push events relies on subscribing to the PostReceive hook
to enqueue Sidekiq requests to the `SecretScanningService`. The `SecretScanningService`
service fetches enqueued refs, queries Gitaly for the ref blob contents, scans
the commit contents, and notifies the Rails application when a secret is detected.
The detection flow for push events relies on subscribing to the PreReceive hook
to scan commit data using the [PushCheck interface](https://gitlab.com/gitlab-org/gitlab/blob/3f1653f5706cd0e7bbd60ed7155010c0a32c681d/lib/gitlab/checks/push_check.rb). This `SecretScanningService`
service fetches the specified blob contents from Gitaly, scans
the commit contents, and rejects the push when a secret is detected.
See [Push event detection flow](#push-event-detection-flow) for sequence.
The detection flow for arbitrary text blobs, such as issue comments, relies on
@ -112,13 +112,33 @@ storage. See discussion [in this issue](https://gitlab.com/groups/gitlab-org/-/e
around scanning during streaming and the added complexity in buffering lookbacks
for arbitrary trace chunks.
In any case of detection, the Rails application manually creates a vulnerability
In the case of a push detection, the commit is rejected and error returned to the end user.
In any other case of detection, the Rails application manually creates a vulnerability
using the `Vulnerabilities::ManuallyCreateService` to surface the finding in the
existing Vulnerability Management UI.
See [technical discovery](https://gitlab.com/gitlab-org/gitlab/-/issues/376716)
for further background exploration.
### Target types
Target object types refer to the scanning targets prioritized for detection of leaked secrets.
In order of priority this includes:
1. non-binary Git blobs
1. job logs
1. issuable creation (issues, MRs, epics)
1. issuable updates (issues, MRs, epics)
1. issuable comments (issues, MRs, epics)
Targets out of scope for the initial phases include:
- Media types (JPEGs, PDFs,...)
- Snippets
- Wikis
- Container images
### Token types
The existing Secret Detection configuration covers ~100 rules across a variety
@ -140,9 +160,10 @@ for all secret scanning in pipeline contexts. By using its `--no-git` configurat
we can scan arbitrary text blobs outside of a repository context and continue to
utilize it for non-pipeline scanning.
Given our existing familiarity with the tool and its extensibility, it should
remain our engine of choice. Changes to the detection engine are out of scope
unless benchmarking unveils performance concerns.
In the case of prereceive detection, we rely on a combination of keyword/substring matches
for prefiltering and `re2` for regex detections. See [spike issue](https://gitlab.com/gitlab-org/gitlab/-/issues/423832) for initial benchmarks
Changes to the detection engine are out of scope until benchmarking unveils performance concerns.
Notable alternatives include high-performance regex engines such as [hyperscan](https://github.com/intel/hyperscan) or it's portable fork [vectorscan](https://github.com/VectorCamp/vectorscan).
@ -167,37 +188,42 @@ for past discussion around scaling approaches.
sequenceDiagram
autonumber
actor User
User->>+Workhorse: git push
User->>+Workhorse: git push with-secret
Workhorse->>+Gitaly: tcp
Gitaly->>+Rails: grpc
Sidekiq->>+Rails: poll job
Rails->>-Sidekiq: PostReceive worker
Sidekiq-->>+Sidekiq: enqueue PostReceiveSecretScanWorker
Gitaly->>+Rails: PreReceive
Rails->>-Gitaly: ListAllBlobs
Gitaly->>-Rails: ListAllBlobsResponse
Sidekiq->>+Rails: poll job
loop PostReceiveSecretScanWorker
Rails->>-Sidekiq: PostReceiveSecretScanWorker
Sidekiq->>+SecretScanningSvc: ScanBlob(ref)
SecretScanningSvc->>+Sidekiq: accepted
Note right of SecretScanningSvc: Scanning job enqueued
Sidekiq-->>+Rails: done
SecretScanningSvc->>+Gitaly: retrieve blob
SecretScanningSvc->>+SecretScanningSvc: scan blob
SecretScanningSvc->>+Rails: secret found
end
Rails->>+GitLabSecretDetection: Scan(blob)
GitLabSecretDetection->>-Rails: found
Rails->>User: rejected: secret found
User->>+Workhorse: git push without-secret
Workhorse->>+Gitaly: tcp
Gitaly->>+Rails: PreReceive
Rails->>-Gitaly: ListAllBlobs
Gitaly->>-Rails: ListAllBlobsResponse
Rails->>+GitLabSecretDetection: Scan(blob)
GitLabSecretDetection->>-Rails: not_found
Rails->>User: OK
```
## Iterations
- ✓ Define [requirements for detection coverage and actions](https://gitlab.com/gitlab-org/gitlab/-/issues/376716)
- ✓ Implement [Clientside detection of GitLab tokens in comments/issues](https://gitlab.com/gitlab-org/gitlab/-/issues/368434)
- PoC of secret scanning service
- Benchmarking of issuables, comments, job logs and blobs to gain confidence that the total costs will be viable
- Capacity planning for addition of service component to Reference Architectures headroom
- Service capabilities
- ✓ Implement [Browser-based detection of GitLab tokens in comments/issues](https://gitlab.com/gitlab-org/gitlab/-/issues/368434)
- ✓ [PoC of secret scanning service](https://gitlab.com/gitlab-org/secure/pocs/secret-detection-go-poc/)
- ✓ [PoC of secret scanning gem](https://gitlab.com/gitlab-org/gitlab/-/issues/426823)
- [Pre Production Performance Profiling for pre-receive PoCs](https://gitlab.com/gitlab-org/gitlab/-/issues/428499)
- Profiling service capabilities
- ✓ [Benchmarking regex performance between Ruby and Go approaches](https://gitlab.com/gitlab-org/gitlab/-/issues/423832)
- gRPC commit retrieval from Gitaly
- blob scanning
- transfer latency, CPU, and memory footprint
- Implementation of secret scanning service MVC (targeting individual commits)
- Capacity planning for addition of service component to Reference Architectures headroom
- Security and readiness review
- Deployment and monitoring
- Implementation of secret scanning service MVC (targeting arbitrary text blobs)

8
doc/user/application_security/vulnerabilities/index.md
View File

@ -104,7 +104,13 @@ When dismissing a vulnerability, one of the following reasons must be chosen to
- **Used in tests**: The finding is not a vulnerability because it is part of a test or is test data.
- **Not applicable**: The vulnerability is known, and has not been remediated or mitigated, but is considered to be in a part of the application that will not be updated.
## Change status of a vulnerability
## Change the status of a vulnerability
> In GitLab 16.4 the ability for `Developers` to change the status of a vulnerability (`admin_vulnerability`) was [deprecated](../../../update/deprecations.md#deprecate-change-vulnerability-status-from-the-developer-role). The `admin_vulnerability` permission will be removed, by default, from all `Developer` roles in GitLab 17.0.
Prerequisites:
- You must have at least the Developer role for the project.
To change a vulnerability's status from its Vulnerability Page:

9
doc/user/permissions.md
View File

@ -201,10 +201,10 @@ The following table lists project permissions available for each role:
| [Requirements Management](project/requirements/index.md):<br>Import / export | | ✓ | ✓ | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Create issue from vulnerability finding | | | ✓ | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Create vulnerability from vulnerability finding | | | ✓ | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability | | | ✓ | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability finding | | | ✓ | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Resolve vulnerability | | | ✓ | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Revert vulnerability to detected state | | | ✓ | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability | | | ✓ (24) | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability finding | | | ✓ | ✓ (24) | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Resolve vulnerability | | | ✓ (24) | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Revert vulnerability to detected state | | | ✓ (24) | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>Use security dashboard | | | ✓ | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability | | | ✓ | ✓ | ✓ |
| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability findings in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ |
@ -249,6 +249,7 @@ The following table lists project permissions available for each role:
21. Authors of tasks can delete them even if they don't have the Owner role, but they have to have at least the Guest role for the project.
22. You must have permission to [view the epic](group/epics/manage_epics.md#who-can-view-an-epic).
23. In GitLab 15.9 and later, users with the Guest role and an Ultimate license can view private repository content if an administrator (on self-managed) or group owner (on GitLab.com) gives those users permission. The administrator or group owner can create a [custom role](custom_roles.md) through the API and assign that role to the users.
24. In GitLab 16.4 the ability for `Developers` to change the status of a vulnerability (`admin_vulnerability`) was [deprecated](../update/deprecations.md#deprecate-change-vulnerability-status-from-the-developer-role). The `admin_vulnerability` permission will be removed, by default, from all `Developer` roles in GitLab 17.0.
<!-- markdownlint-enable MD029 -->

3
locale/gitlab.pot
View File

@ -56920,6 +56920,9 @@ msgstr ""
msgid "is not in the group enforcing Group Managed Account"
msgstr ""
msgid "is not linked to a SAML account or has an inactive SCIM identity. For information on how to resolve this error, see the %{troubleshoot_link_start}troubleshooting SCIM documentation%{troubleshoot_link_end}."
msgstr ""
msgid "is not one of"
msgstr ""

13
qa/qa/specs/features/api/1_manage/migration/gitlab_migration_large_project_spec.rb
View File

@ -48,6 +48,19 @@ module QA
create(:sandbox, api_client: admin_api_client, path: "qa-sandbox-#{SecureRandom.hex(4)}")
end
let!(:api_client) do
Runtime::API::Client.new(
user: user,
is_new_session: false,
personal_access_token: Resource::PersonalAccessToken.fabricate_via_api! do |pat|
pat.user = user
# importing very large project can take multiple days
# token must not expire while we still poll for import result
pat.expires_at = (Time.now.to_date + 5)
end.token
)
end
# Source objects
#
let(:source_project) { source_group.projects(auto_paginate: true).find { |project| project.name == gitlab_source_project }.reload! }