root
/
gitlab-ce
mirror of https://gitlab.com/free-releases/gitlab-ce.git
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

Add latest changes from gitlab-org/gitlab@master

This commit is contained in:
GitLab Bot 2024-01-31 03:07:07 +00:00
parent a88bc207c2
commit 9aa50adcfa
13 changed files with 211 additions and 88 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
config/feature_flags/development/use_primary_and_secondary_stores_for_sidekiq_status_migrator.yml Normal file
View File

@ -0,0 +1,9 @@
---
name: use_primary_and_secondary_stores_for_sidekiq_status_migrator
feature_issue_url: https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/923
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/141743
rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/439553
milestone: '16.9'
group: group::scalability
type: development
default_enabled: false

9
config/feature_flags/development/use_primary_store_as_default_for_sidekiq_status_migrator.yml Normal file
View File

@ -0,0 +1,9 @@
---
name: use_primary_store_as_default_for_sidekiq_status_migrator
feature_issue_url: https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/923
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/141743
rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/439553
milestone: '16.9'
group: group::scalability
type: development
default_enabled: false

27
data/deprecations/16-9-secure-analyzers-bump.yml Normal file
View File

@ -0,0 +1,27 @@
- title: "Secure analyzers major version update"
announcement_milestone: "16.9"
removal_milestone: "17.0"
breaking_change: true
reporter: thiagocsf
stage: secure
issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/438123
body: | # Do not modify this line, instead modify the lines below.
The Secure stage will be bumping the major versions of its analyzers in tandem with the GitLab
17.0 release.
If you are not using the default included templates, or have pinned your analyzer versions you
must update your CI/CD job definition to either remove the pinned version or to update
the latest major version.
Users of GitLab 16.0-16.11 will continue to experience analyzer updates as normal until the
release of GitLab 17.0, following which all newly fixed bugs and released features will be
released only in the new major version of the analyzers.
We do not backport bugs and features to deprecated versions as per our maintenance policy. As
required, security patches will be backported within the latest 3 minor releases.
Specifically, the following analyzers are being deprecated and will no longer be updated after
the GitLab 17.0 release:
- Container Scanning: version 6
- Dependency Scanning: version 4

62
doc/administration/audit_event_schema.md
View File

@ -12,23 +12,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Audit events have a predictable schema in the body of the response.
| Field | Description | Notes |
Streaming Only Field |
|------------------|------------------------------------------------------------|-----------------------------------------------------------------------------------|
-----------------------------------------------------------------------------------|
| `author_id` | User ID of the user who triggered the event | | |
| `author_name` | Human-readable name of the author that triggered the event | Helpful when the author no longer exists | :white_check_mark: |
| `created_at` | Timestamp when event was triggered | | |
| `details` | JSON object containing additional metadata | Has no defined schema but often contains additional information about an event | |
| `entity_id` | ID of the audit event's entity | | |
| `entity_path` | Full path of the entity affected by the auditable event | | :white_check_mark: |
| `entity_type` | String representation of the type of entity | Acceptable values include `User`, `Group`, and `Key`. This list is not exhaustive | |
| `event_type` | String representation of the type of audit event | | :white_check_mark: |
| `id` | Unique identifier for the audit event | Can be used for deduplication if required | |
| `ip_address` | IP address of the host used to trigger the event | | :white_check_mark: |
| `target_details` | Additional details about the target | | :white_check_mark: |
| `target_id` | ID of the audit event's target | | :white_check_mark: |
| `target_type` | String representation of the target's type | | :white_check_mark: |
| Field | Description | Notes | Streaming Only Field |
|------------------|------------------------------------------------------------|-----------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| `author_id` | User ID of the user who triggered the event | | **{dotted-circle}** No |
| `author_name` | Human-readable name of the author that triggered the event | Helpful when the author no longer exists | **{check-circle}** Yes |
| `created_at` | Timestamp when event was triggered | | **{dotted-circle}** No |
| `details` | JSON object containing additional metadata | Has no defined schema but often contains additional information about an event | **{dotted-circle}** No |
| `entity_id` | ID of the audit event's entity | | **{dotted-circle}** No |
| `entity_path` | Full path of the entity affected by the auditable event | | **{check-circle}** Yes |
| `entity_type` | String representation of the type of entity | Acceptable values include `User`, `Group`, and `Key`. This list is not exhaustive | **{dotted-circle}** No |
| `event_type` | String representation of the type of audit event | | **{check-circle}** Yes |
| `id` | Unique identifier for the audit event | Can be used for deduplication if required | **{dotted-circle}** No |
| `ip_address` | IP address of the host used to trigger the event | | **{check-circle}** Yes |
| `target_details` | Additional details about the target | | **{check-circle}** Yes |
| `target_id` | ID of the audit event's target | | **{check-circle}** Yes |
| `target_type` | String representation of the target's type | | **{check-circle}** Yes |
### Audit Event JSON schema
@ -74,25 +72,9 @@ Streaming Only Field
}
```
## Example: audit event streaming on Git operations
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default.
> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.0.
> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.1 by default.
> - `details.author_class` field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3.
> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101583) in GitLab 15.6. Feature flag `audit_event_streaming_git_operations` removed.
Streaming audit events can be sent when authenticated users push, pull, or clone a project's remote Git repositories:
- [Using SSH](../user/ssh.md).
- Using HTTP or HTTPS.
- Using **Download** (**{download}**) in GitLab UI.
Audit events are not captured for users that are not signed in. For example, when downloading a public project.
### Headers
> `X-Gitlab-Audit-Event-Type` [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86881) in GitLab 15.0.
> - `X-Gitlab-Audit-Event-Type` [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86881) in GitLab 15.0.
Headers are formatted as follows:
@ -104,9 +86,17 @@ X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN>
X-Gitlab-Audit-Event-Type: repository_git_operation
```
### Example payloads for Git over SSH events with Deploy Key
## Example: audit event streaming on Git operations
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3.
Streaming audit events can be sent when authenticated users push, pull, or clone a project's remote Git repositories:
- [Using SSH](../user/ssh.md).
- Using HTTP or HTTPS.
- Using **Download** (**{download}**) in GitLab UI.
Audit events are not captured for users that are not signed in. For example, when downloading a public project.
### Example: audit event payloads for Git over SSH events with Deploy Key
Fetch:

90
doc/development/ai_features/index.md
View File

@ -49,6 +49,10 @@ To implement a new AI action, connect to the preferred AI provider. You can conn
All AI features are experimental.
## Test self-managed AI features locally
See [below](#test-ai-features-with-ai-gateway-locally)
## Test SaaS-only AI features locally
**One-line setup**
@ -67,7 +71,7 @@ RAILS_ENV=development bundle exec rake gitlab:duo:setup['<test-group-name>']
Feature.enable(:ai_global_switch, type: :ops)
```
1. Ensure you have followed [the process to obtain an EE license](https://about.gitlab.com/handbook/developer-onboarding/#working-on-gitlab-ee-developer-licenses) for your local instance
1. Ensure you have followed [the process to obtain an EE license](https://handbook.gitlab.com/handbook/developer-onboarding/#working-on-gitlab-ee-developer-licenses) for your local instance and you applied this license.
1. Simulate the GDK to [simulate SaaS](../ee_features.md#simulate-a-saas-instance) and ensure the group you want to test has an Ultimate license
1. Enable `Experiment & Beta features`
1. Go to the group with the Ultimate license
@ -197,23 +201,20 @@ Therefore, a different setup is required from the [SaaS-only AI features](#test-
### Setup
1. Set up AI Gateway:
1. [Install it](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally).
1. Ensure that the following environment variables are set in the `.env` file:
```shell
AIGW_AUTH__BYPASS_EXTERNAL=true
ANTHROPIC_API_KEY="[REDACTED]" # IMPORTANT: Ensure you use Corp account. See https://gitlab.com/gitlab-org/gitlab/-/issues/435911#note_1701762954.
AIGW_VERTEX_TEXT_MODEL__PROJECT="[REDACTED]"
```
1. Run `poetry run ai_gateway`.
1. Visit OpenAPI playground (`http://0.0.0.0:5052/docs`), try an endpoint (e.g. `/v1/chat/agent`) and make sure you get a successful response.
If something went wrong, check `modelgateway_debug.log` if it contains error information.
1. Setup GitLab Development Kit (GDK):
1. [Install it](https://gitlab.com/gitlab-org/gitlab-development-kit#installation).
1. [Set up `gdk.test` hostname](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#set-up-gdktest-hostname).
1. [Activate GitLab Enterprise license](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#use-gitlab-enterprise-features) (e.g. Ultimate).
1. Setup CustomersDot:
1. Install CustomersDot: [internal video tutorial](https://youtu.be/_8wOMa_yGSw)
- This video loosely follows [official installation steps](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/doc/setup/installation_steps.md)
- It also offers guidance on how to create a self-managed subscription. You will receive a *cloud activation code* in return.
- Bookmark [this link](http://localhost:5000/subscriptions/new?plan_id=2c92c0f976d721ed0176db74d74f4fd4&test=capabilities) for creating Ultimate Self-Managed Subscription locally.
- A list of other subscription plan ids are available [here](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/doc/flows/buy_subscription.md).
1. Setup GitLab Development Kit (GDK): [internal video tutorial](https://youtu.be/rudS6KeQHcA)
1. [Install it](https://gitlab.com/gitlab-org/gitlab-development-kit#installation) as a separate GDK instance.
1. Run `gdk config set license.customer_portal_url 'http://localhost:5000'`
1. [Set up `gdk.test` hostname](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/local_network.md#local-interface).
1. Follow [Instruct your local CustomersDot instance to use the GitLab application](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/doc/setup/installation_steps.md#instruct-your-local-customersdot-instance-to-use-the-gitlab-application)
1. Activate GitLab Enterprise license
- To test Self Managed instances, follow [Cloud Activation steps](../../administration/license.md#activate-gitlab-ee) using the cloud activation code you received earlier.
- To test SaaS, follow [Activate GitLab Enterprise license](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/index.md#use-gitlab-enterprise-features) with your license file.
1. Export these environment variables in the same terminal session with `gdk start`:
```shell
@ -228,19 +229,25 @@ Therefore, a different setup is required from the [SaaS-only AI features](#test-
rake gitlab:duo:enable_feature_flags
```
1. Create a dummy access token via `gdk rails console` OR skip this step and setup GitLab or Customer Dot as OIDC provider (See the following section):
1. Set up AI Gateway: [internal video tutorial](https://youtu.be/ePoHqvw78oQ)
1. [Install it](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#how-to-run-the-server-locally).
1. Ensure that the following environment variables are set in the `.env` file:
```ruby
# Creating dummy token, and this will work as long as `AIGW_AUTH__BYPASS_EXTERNAL=true` in AI Gateway.
::CloudConnector::ServiceAccessToken.create!(token: 'dummy', expires_at: 1.month.from_now)
```
```shell
AIGW_AUTH__BYPASS_EXTERNAL=true
ANTHROPIC_API_KEY="[REDACTED]" # IMPORTANT: Ensure you use Corp account. See https://gitlab.com/gitlab-org/gitlab/-/issues/435911#note_1701762954.
```
1. Ensure GitLab-Rails can talk to the AI Gateway. Run `gdk rails console` and execute:
1. (Optional) [Configure OIDC](#set-oidc-provider-in-ai-gateway) if needed.
1. Run `poetry run ai_gateway`.
1. Visit OpenAPI playground (`http://0.0.0.0:5052/docs`), try an endpoint (e.g. `/v1/chat/agent`) and make sure you get a successful response.
If something went wrong, check `modelgateway_debug.log` if it contains error information.
```ruby
user = User.first
Gitlab::Llm::AiGateway::Client.new(user).stream(prompt: "\n\nHuman: Hi, how are you?\n\nAssistant:")
```
You are set, and should be able to verify AI feature by calling the following in GitLab-Rails console:
```ruby
Gitlab::Llm::AiGateway::Client.new(User.first).stream(prompt: "\n\nHuman: Hi, how are you?\n\nAssistant:")
```
#### Verify the setup with GraphQL
@ -282,33 +289,22 @@ Therefore, a different setup is required from the [SaaS-only AI features](#test-
If you can't fetch the response, check `graphql_json.log`, `sidekiq_json.log`, `llm.log` or `modelgateway_debug.log` if it contains error information.
### Use GitLab as OIDC provider in AI Gateway
1. Reconfigure AI Gateway:
1. Additionally, ensure that the following environment variables are set in the `.env` file:
```shell
AIGW_GITLAB_URL="http://gdk.test:3000/"
AIGW_GITLAB_API_URL="http://gdk.test:3000/api/v4/"
AIGW_AUTH__BYPASS_EXTERNAL=False
```
### Set OIDC provider in AI Gateway
1. Configure AI Gateway:
1. [Set the OIDC provider URLs](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist#set-oidc-providers).
Make sure to adapt to the domain you use.
Note that you can choose to only set either GitLab SaaS instance or CDot as a provider.
1. Restart AI Gateway.
1. Reconfigure GitLab Development Kit (GDK):
1. Additionally, export the following environment variables:
1. If GitLab instance is set as a provider, you need to configure GDK to run in SaaS mode:
1. Export the following environment variables:
```shell
export GITLAB_SIMULATE_SAAS=1 # Simulate a SaaS instance. See https://docs.gitlab.com/ee/development/ee_features.html#simulate-a-saas-instance.
export GITLAB_SIMULATE_SAAS=1 # Simulate a SaaS instance. See https://docs.gitlab.com/ee/development/ee_features.html#simulate-a-saas-instance.
```
1. Restart GDK.
### Use Customer Dot as OIDC provider in AI Gateway
1. AI Gateway:
1. Ensure `AIGW_CUSTOMER_PORTAL_URL` in the `.env` file points to your Customer Dot URL.
1. Restart
## Experimental REST API
Use the [experimental REST API endpoints](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/ai/experimentation) to quickly experiment and prototype AI features.

32
doc/update/deprecations.md
View File

@ -1308,6 +1308,38 @@ that is available now. We recommend this alternative solution because it provide
<div class="deprecation breaking-change" data-milestone="17.0">
### Secure analyzers major version update
<div class="deprecation-notes">
- Announced in GitLab <span class="milestone">16.9</span>
- Removal in GitLab <span class="milestone">17.0</span> ([breaking change](https://docs.gitlab.com/ee/update/terminology.html#breaking-change))
- To discuss this change or learn more, see the [deprecation issue](https://gitlab.com/gitlab-org/gitlab/-/issues/438123).
</div>
The Secure stage will be bumping the major versions of its analyzers in tandem with the GitLab
17.0 release.
If you are not using the default included templates, or have pinned your analyzer versions you
must update your CI/CD job definition to either remove the pinned version or to update
the latest major version.
Users of GitLab 16.0-16.11 will continue to experience analyzer updates as normal until the
release of GitLab 17.0, following which all newly fixed bugs and released features will be
released only in the new major version of the analyzers.
We do not backport bugs and features to deprecated versions as per our maintenance policy. As
required, security patches will be backported within the latest 3 minor releases.
Specifically, the following analyzers are being deprecated and will no longer be updated after
the GitLab 17.0 release:
- Container Scanning: version 6
- Dependency Scanning: version 4
</div>
<div class="deprecation breaking-change" data-milestone="17.0">
### Security policy field `newly_detected` is deprecated
<div class="deprecation-notes">

BIN
doc/user/analytics/img/issues_created_per_month_v14_8.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 17 KiB

2
doc/user/project/deploy_keys/index.md
View File

@ -69,7 +69,7 @@ As with all sensitive information, you should ensure only those who need access
For human interactions, use credentials tied to users such as Personal Access Tokens.
To help detect a potential secret leak, you can use the
[Audit Event](../../../administration/audit_event_schema.md#example-payloads-for-git-over-ssh-events-with-deploy-key) feature.
[Audit Event](../../../administration/audit_event_schema.md#example-audit-event-payloads-for-git-over-ssh-events-with-deploy-key) feature.
## View deploy keys

1
lib/gitlab/redis.rb
View File

@ -18,6 +18,7 @@ module Gitlab
Gitlab::Redis::RepositoryCache,
Gitlab::Redis::Sessions,
Gitlab::Redis::SharedState,
Gitlab::Redis::SidekiqStatusMigrator,
Gitlab::Redis::TraceChunks,
Gitlab::Redis::Chat,
Gitlab::Redis::Workhorse

14
lib/gitlab/redis/sidekiq_status_migrator.rb Normal file
View File

@ -0,0 +1,14 @@
# frozen_string_literal: true
module Gitlab
module Redis
class SidekiqStatusMigrator < ::Gitlab::Redis::MultiStoreWrapper
class << self
def multistore
# migrate from SharedState to QueuesMetadata
MultiStore.new(QueuesMetadata.pool, SharedState.pool, store_name)
end
end
end
end
end

5
lib/gitlab/sidekiq_status.rb
View File

@ -138,7 +138,10 @@ module Gitlab
end
def self.with_redis
if Feature.enabled?(:use_primary_and_secondary_stores_for_sidekiq_status) ||
if Feature.enabled?(:use_primary_and_secondary_stores_for_sidekiq_status_migrator) ||
Feature.enabled?(:use_primary_store_as_default_for_sidekiq_status_migrator)
Gitlab::Redis::SidekiqStatusMigrator.with { |redis| yield redis }
elsif Feature.enabled?(:use_primary_and_secondary_stores_for_sidekiq_status) ||
Feature.enabled?(:use_primary_store_as_default_for_sidekiq_status)
# TODO: Swap for Gitlab::Redis::SharedState after store transition
# https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/923

15
spec/lib/gitlab/redis/sidekiq_status_migrator_spec.rb Normal file
View File

@ -0,0 +1,15 @@
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Gitlab::Redis::SidekiqStatusMigrator, feature_category: :redis do
let(:instance_specific_config_file) { "config/redis.shared_state.yml" }
let(:rails_root) { "test" }
include_examples "multi_store_wrapper_shared_examples"
it 'migrates data from SharedState to QueuesMetadata' do
expect(described_class.multistore.primary_pool).to eq(Gitlab::Redis::QueuesMetadata.pool)
expect(described_class.multistore.secondary_pool).to eq(Gitlab::Redis::SharedState.pool)
end
end

33
spec/lib/gitlab/sidekiq_status_spec.rb
View File

@ -157,11 +157,34 @@ RSpec.describe Gitlab::SidekiqStatus, :clean_gitlab_redis_queues,
end
end
context 'with multi-store feature flags turned on' do
context 'with migrator class multi-store feature flags turned on' do
def with_redis(&block)
Gitlab::Redis::SidekiqStatusMigrator.with(&block)
end
it 'uses Gitlab::Redis::SharedState.with' do
expect(Gitlab::Redis::SidekiqStatusMigrator).to receive(:with).and_call_original
expect(Gitlab::Redis::SharedState).not_to receive(:with)
expect(Sidekiq).not_to receive(:redis)
described_class.job_status(%w[123 456 789])
end
it_behaves_like 'tracking status in redis'
end
context 'with sidekiq-status multi-store feature flags turned on' do
def with_redis(&block)
Gitlab::Redis::SharedState.with(&block)
end
before do
stub_feature_flags(
use_primary_and_secondary_stores_for_sidekiq_status_migrator: false,
use_primary_store_as_default_for_sidekiq_status_migrator: false
)
end
it 'uses Gitlab::Redis::SharedState.with' do
expect(Gitlab::Redis::SharedState).to receive(:with).and_call_original
expect(Sidekiq).not_to receive(:redis)
@ -178,8 +201,12 @@ RSpec.describe Gitlab::SidekiqStatus, :clean_gitlab_redis_queues,
end
before do
stub_feature_flags(use_primary_and_secondary_stores_for_sidekiq_status: false)
stub_feature_flags(use_primary_store_as_default_for_sidekiq_status: false)
stub_feature_flags(
use_primary_and_secondary_stores_for_sidekiq_status_migrator: false,
use_primary_store_as_default_for_sidekiq_status_migrator: false,
use_primary_and_secondary_stores_for_sidekiq_status: false,
use_primary_store_as_default_for_sidekiq_status: false
)
end
it 'uses Sidekiq.redis' do