root
/
gitlab-ce
mirror of https://gitlab.com/free-releases/gitlab-ce.git
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

Add latest changes from gitlab-org/gitlab@master

This commit is contained in:
GitLab Bot 2024-09-06 03:13:50 +00:00
parent 11501d600a
commit b28ebf3730
16 changed files with 278 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
app/controllers/import/source_users_controller.rb
View File

@ -15,9 +15,9 @@ module Import
if result.success?
flash[:raw] = banner('accept_invite')
redirect_to(dashboard_groups_path)
redirect_to(root_path)
else
redirect_to(dashboard_groups_path, alert: s_('UserMapping|The invitation could not be accepted.'))
redirect_to(root_path, alert: s_('UserMapping|The invitation could not be accepted.'))
end
end
@ -26,9 +26,9 @@ module Import
if result.success?
flash[:raw] = banner('reject_invite')
redirect_to(dashboard_groups_path)
redirect_to(root_path)
else
redirect_to(dashboard_groups_path, alert: s_('UserMapping|The invitation could not be declined.'))
redirect_to(root_path, alert: s_('UserMapping|The invitation could not be declined.'))
end
end
@ -40,7 +40,7 @@ module Import
return if source_user.awaiting_approval? && current_user_matches_invite?
flash[:raw] = banner('invalid_invite')
redirect_to(dashboard_groups_path)
redirect_to(root_path)
end
def current_user_matches_invite?

4
config/gitlab_loose_foreign_keys.yml
View File

@ -584,10 +584,6 @@ vulnerability_scanners:
- table: projects
column: project_id
on_delete: async_delete
vulnerability_state_transitions:
- table: ci_pipelines
column: state_changed_at_pipeline_id
on_delete: async_nullify
vulnerability_statistics:
- table: ci_pipelines
column: latest_pipeline_id

2
doc/.vale/gitlab_base/CodeBlockNesting.yml
View File

@ -4,7 +4,7 @@
# Ensures content nested in lists are spaced correctly.
#
extends: existence
message: "Use three spaces for lines under ordered lists, and two spaces under unordered lists"
message: "Items under an ordered list must be indented three spaces. Items under an unordered list must be indented two spaces."
link: https://docs.gitlab.com/ee/development/documentation/styleguide/#nesting-inside-a-list-item
level: error
nonword: true

1
doc/administration/repository_storage_paths.md
View File

@ -28,6 +28,7 @@ For more information on:
## Hashed storage
> - Support for legacy storage, where repository paths were generated based on the project path, has been completely removed in GitLab 14.0.
> - **Storage name** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128416) from **Gitaly storage name** and **Relative path** field [renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/128416) from **Gitaly relative path** in GitLab 16.3.
Hashed storage stores projects on disk in a location based on a hash of the project's ID. This makes the folder

15
doc/api/environments.md
View File

@ -163,6 +163,21 @@ Example of response
"runner": null,
"artifacts_expire_at": null
}
},
"cluster_agent": {
"id": 1,
"name": "agent-1",
"config_project": {
"id": 20,
"description": "",
"name": "test",
"name_with_namespace": "Administrator / test",
"path": "test",
"path_with_namespace": "root/test",
"created_at": "2022-03-20T20:42:40.221Z"
},
"created_at": "2022-04-20T20:42:40.221Z",
"created_by_user_id": 42
}
}
```

7
doc/development/file_storage.md
View File

@ -45,7 +45,7 @@ they are still not 100% standardized. You can see them below:
| User snippet attachments | yes | `uploads/-/system/personal_snippet/:id/:random_hex/:filename` | `PersonalFileUploader` | Snippet |
| Project avatars | yes | `uploads/-/system/project/avatar/:id/:filename` | `AvatarUploader` | Project |
| Topic avatars | yes | `uploads/-/system/projects/topic/avatar/:id/:filename` | `AvatarUploader` | Topic |
| Issues/MR/Notes Markdown attachments | yes | `uploads/:project_path_with_namespace/:random_hex/:filename` | `FileUploader` | Project |
| Issues/MR/Notes Markdown attachments | yes | `uploads/:hash_project_id/:random_hex/:filename` | `FileUploader` | Project |
| Issues/MR/Notes Legacy Markdown attachments | no | `uploads/-/system/note/attachment/:id/:filename` | `AttachmentUploader` | Note |
| Design Management design thumbnails | yes | `uploads/-/system/design_management/action/image_v432x230/:id/:filename` | `DesignManagement::DesignV432x230Uploader` | DesignManagement::Action |
| CI Artifacts (CE) | yes | `shared/artifacts/:disk_hash[0..1]/:disk_hash[2..3]/:disk_hash/:year_:month_:date/:job_id/:job_artifact_id` (`:disk_hash` is SHA256 digest of `project_id`) | `JobArtifactUploader` | Ci::JobArtifact |
@ -56,9 +56,8 @@ they are still not 100% standardized. You can see them below:
CI Artifacts and LFS Objects behave differently in CE and EE. In CE they inherit the `GitlabUploader`
while in EE they inherit the `ObjectStorage` and store files in and S3 API compatible object store.
In the case of Issues/MR/Notes Markdown attachments, there is a different approach using the [Hashed Storage](../administration/repository_storage_paths.md) layout,
instead of basing the path into a mutable variable `:project_path_with_namespace`, it's possible to use the
hash of the project ID instead, if project migrates to the new approach (introduced in 10.2).
Attachments for issues, merge requests (MR), and notes in Markdown use
[hashed storage](../administration/repository_storage_paths.md) with the hash of the project ID.
We provide an [all-in-one Rake task](../administration/raketasks/uploads/migrate.md)
to migrate all uploads to object storage in one go. If a new Uploader class or model

3
doc/development/git_object_deduplication.md
View File

@ -98,9 +98,6 @@ are as follows:
### Assumptions
- All repositories in a pool must use [hashed storage](../administration/repository_storage_paths.md).
This is so that we don't have to ever worry about updating paths in
`object/info/alternates` files.
- All repositories in a pool must be on the same Gitaly storage shard.
The Git alternates mechanism relies on direct disk access across
multiple repositories, and we can only assume direct disk access to

8
doc/user/application_security/policies/pipeline_execution_policies.md
View File

@ -192,6 +192,14 @@ The strategy takes precedence over other policies using the `inject_ci` strategy
This strategy allows users to include the project CI/CD configuration in the pipeline execution policy configuration, enabling them to customize the policy jobs. For example, by combining policy and project CI/CD configuration into one YAML file, users can override `before_script` configuration.
NOTE:
When a pipeline execution policy uses workflow rules that prevent policy jobs from running, the
project's original CI/CD configuration remains in effect instead of being overridden. You can
conditionally apply pipeline execution policies to control when the policy impacts the project's
CI/CD configuration. For example, if you set a workflow rule `if: $CI_PIPELINE_SOURCE ==
"merge_request_event"`, the project's CI configuration is only overridden when the pipeline source
is a merge request event.
### Include a project's CI/CD configuration in the pipeline execution policy configuration
When using `override_project_ci` strategy, the project configuration can be included into the pipeline execution policy configuration:

2
doc/user/project/issues/design_management.md
View File

@ -36,8 +36,6 @@ For a video overview, see [Design Management](https://www.youtube.com/watch?v=CC
Image thumbnails are stored as other uploads, and are not associated with a project but rather
with a specific design model.
Newly created projects use hashed storage by default.
A GitLab administrator can verify the relative path of a hashed-stored project by going to **Admin area > Projects**
and then selecting the project in question. The **Relative path** field contains `@hashed` in its value.

9
lib/api/entities/environment.rb
View File

@ -10,6 +10,15 @@ module API
expose :last_deployment, using: Entities::Deployment, if: { last_deployment: true }
expose :state, documentation: { type: 'string', example: 'available' }
expose :auto_stop_at, documentation: { type: 'dateTime', example: '2019-05-25T18:55:13.252Z' }
expose :cluster_agent, using: Entities::Clusters::Agent, if: ->(_, _) { can_read_cluster_agent? }
private
def can_read_cluster_agent?
return unless object.cluster_agent.present?
Ability.allowed?(options[:current_user], :read_cluster_agent, object.cluster_agent)
end
end
end
end

112
lib/import/placeholder_references/alias_resolver.rb
View File

@ -3,8 +3,13 @@
module Import
module PlaceholderReferences
module AliasResolver
extend self
MissingAlias = Class.new(StandardError)
NOTE_COLUMNS = { "author_id" => "author_id", "updated_by_id" => "updated_by_id",
"resolved_by_id" => "resolved_by_id" }.freeze
# A new version for a model should be defined when new entries must be
# mapped in a different way to data that already exists in the database.
# Context: https://gitlab.com/gitlab-org/gitlab/-/issues/478501
@ -30,7 +35,7 @@ module Import
"Ci::Build" => {
1 => {
model: Ci::Build,
columns: { "user_id" => "user_id" }
columns: { "user_id" => "user_id", "erased_by_id" => "erased_by_id" }
}
},
"Ci::Pipeline" => {
@ -39,28 +44,60 @@ module Import
columns: { "user_id" => "user_id" }
}
},
"Ci::PipelineSchedule" => {
1 => {
model: Ci::PipelineSchedule,
columns: { "owner_id" => "owner_id" }
}
},
"DesignManagement::Version" => {
1 => {
model: DesignManagement::Version,
columns: { "author_id" => "author_id" }
}
},
"DiffNote" => {
1 => {
model: DiffNote,
columns: NOTE_COLUMNS
}
},
"DiscussionNote" => {
1 => {
model: DiscussionNote,
columns: NOTE_COLUMNS
}
},
"Event" => {
1 => {
model: Event,
columns: { "author_id" => "author_id" }
}
},
"Epic" => {
1 => {
model: Epic,
columns: { "author_id" => "author_id", "assignee_id" => "assignee_id", "updated_by_id" => "updated_by_id",
"last_edited_by_id" => "last_edited_by_id", "closed_by_id" => "closed_by_id" }
}
},
"GenericCommitStatus" => {
1 => {
model: GenericCommitStatus,
columns: { "user_id" => "user_id" }
}
},
"LegacyDiffNote" => {
1 => {
model: LegacyDiffNote,
columns: NOTE_COLUMNS
}
},
"Issue" => {
1 => {
model: Issue,
columns: { "author_id" => "author_id" }
columns: { "author_id" => "author_id", "updated_by_id" => "updated_by_id",
"closed_by_id" => "closed_by_id", "last_edited_by_id" => "last_edited_by_id" }
}
},
"IssueAssignee" => {
@ -69,16 +106,23 @@ module Import
columns: { "user_id" => "user_id", "issue_id" => "issue_id" }
}
},
"List" => {
1 => {
model: List,
columns: { "user_id" => "user_id" }
}
},
"MergeRequest" => {
1 => {
model: MergeRequest,
columns: { "author_id" => "author_id" }
columns: { "author_id" => "author_id", "updated_by_id" => "updated_by_id",
"last_edited_by_id" => "last_edited_by_id", "merge_user_id" => "merge_user_id" }
}
},
"MergeRequest::Metrics" => {
1 => {
model: MergeRequest::Metrics,
columns: { "merged_by_id" => "merged_by_id" }
columns: { "merged_by_id" => "merged_by_id", "latest_closed_by_id" => "latest_closed_by_id" }
}
},
"MergeRequestAssignee" => {
@ -96,6 +140,36 @@ module Import
"Note" => {
1 => {
model: Note,
columns: NOTE_COLUMNS
}
},
"ProtectedTag::CreateAccessLevel" => {
1 => {
model: ProtectedTag::CreateAccessLevel,
columns: { "user_id" => "user_id" }
}
},
"ProtectedBranch::MergeAccessLevel" => {
1 => {
model: ProtectedBranch::MergeAccessLevel,
columns: { "user_id" => "user_id" }
}
},
"ProtectedBranch::PushAccessLevel" => {
1 => {
model: ProtectedBranch::PushAccessLevel,
columns: { "user_id" => "user_id" }
}
},
"ProjectSnippet" => {
1 => {
model: ProjectSnippet,
columns: { "author_id" => "author_id" }
}
},
"Release" => {
1 => {
model: Release,
columns: { "author_id" => "author_id" }
}
},
@ -117,6 +191,12 @@ module Import
columns: { "user_id" => "user_id" }
}
},
"Snippet" => {
1 => {
model: Snippet,
columns: { "author_id" => "author_id" }
}
},
"Timelog" => {
1 => {
model: Timelog,
@ -125,16 +205,22 @@ module Import
}
}.freeze
def self.version_for_model(model)
return ALIASES[model].keys.max if ALIASES[model]
private_constant :ALIASES
def aliases
ALIASES
end
def version_for_model(model)
return aliases[model].keys.max if aliases[model]
track_error_for_missing(model: model)
1
end
def self.aliased_model(model, version:)
aliased_model = ALIASES.dig(model, version, :model)
def aliased_model(model, version:)
aliased_model = aliases.dig(model, version, :model)
return aliased_model if aliased_model.present?
track_error_for_missing(model: model, version: version)
@ -142,8 +228,8 @@ module Import
model.safe_constantize
end
def self.aliased_column(model, column, version:)
aliased_column = ALIASES.dig(model, version, :columns, column)
def aliased_column(model, column, version:)
aliased_column = aliases.dig(model, version, :columns, column)
return aliased_column if aliased_column.present?
track_error_for_missing(model: model, column: column, version: version)
@ -151,8 +237,8 @@ module Import
column
end
def self.models_with_columns
ALIASES.values
def models_with_columns
aliases.values
.map { |versions| versions[versions.keys.max] }
.map { |data| [data[:model], data[:columns].values] }
end
@ -166,3 +252,5 @@ module Import
end
end
end
Import::PlaceholderReferences::AliasResolver.extend_mod

1
spec/db/schema_spec.rb
View File

@ -179,6 +179,7 @@ RSpec.describe 'Database schema', feature_category: :database do
vulnerability_identifiers: %w[external_id],
vulnerability_occurrence_identifiers: %w[project_id],
vulnerability_scanners: %w[external_id],
vulnerability_state_transitions: %w[state_changed_at_pipeline_id],
security_scans: %w[pipeline_id project_id], # foreign key is not added as ci_pipeline table will be moved into different db soon
dependency_list_exports: %w[pipeline_id], # foreign key is not added as ci_pipeline table is in different db
vulnerability_reads: %w[cluster_agent_id namespace_id], # namespace_id is a denormalization of `project.namespace`

10
spec/fixtures/api/schemas/public_api/v4/environment.json vendored
View File

@ -60,6 +60,16 @@
},
"project": {
"$ref": "project.json"
},
"cluster_agent": {
"oneOf": [
{
"type": "null"
},
{
"$ref": "agent.json"
}
]
}
},
"additionalProperties": false

110
spec/lib/import/placeholder_references/alias_resolver_spec.rb
View File

@ -3,7 +3,21 @@
require "spec_helper"
RSpec.describe Import::PlaceholderReferences::AliasResolver, feature_category: :importers do
describe "ALIASES" do
describe ".aliases" do
def missing_attribute_message(model, attribute)
<<-MSG
#{model}##{attribute} references a user and it is not defined in #{described_class}::ALIASES.
Please add the attribute in the columns key in the #{described_class}::ALIASES['#{model}'] hash.
MSG
end
def missing_alias_message(model)
<<-MSG
#{model} models references a user and it is not defined in #{described_class}::ALIASES.
Please define the mapping in #{described_class}::ALIASES.
MSG
end
it "points to real columns" do
def failure_message(model, column)
<<-MSG
@ -13,7 +27,7 @@ RSpec.describe Import::PlaceholderReferences::AliasResolver, feature_category: :
MSG
end
described_class::ALIASES.each_value.flat_map(&:values).each do |model_alias|
described_class.aliases.each_value.flat_map(&:values).each do |model_alias|
model = model_alias[:model]
column_names = model.columns.map(&:name)
@ -22,6 +36,86 @@ RSpec.describe Import::PlaceholderReferences::AliasResolver, feature_category: :
end
end
end
shared_examples 'define aliases' do
def relation_class(relation_key)
relation_key.to_s.classify.constantize
rescue NameError
relation_key.to_s.constantize
end
def extract_relation_names(hash, keys = [])
keys += hash.keys
hash.each_value do |value|
keys += extract_relation_names(value, keys)
end
keys.uniq
end
it "defines aliases for imported resources that references users", :eager_load do
relation_names = extract_relation_names(config_tree).reject { |name| ignore_relations.include?(name) }
relation_names.each do |relation_name|
relation_name = overrides[relation_name] || relation_name
model_class = relation_class(relation_name)
table_columns = model_class.columns.collect(&:name)
user_associations = model_class.reflect_on_all_associations(:belongs_to)
.reject(&:polymorphic?)
.filter { |association| association.klass == User }
.reject { |association| table_columns.exclude?(association.foreign_key) }
next unless user_associations.any?
expect(described_class.aliases[model_class.to_s]).to be_present, missing_alias_message(model_class)
user_associations.each do |association|
foreign_key = association.foreign_key
last_version = described_class.aliases[model_class.to_s].keys.max
alias_definition = described_class.aliases[model_class.to_s][last_version]
expect(alias_definition[:columns]).to include(foreign_key),
missing_attribute_message(model_class, foreign_key)
end
end
end
end
describe 'group aliases' do
let(:overrides) { Gitlab::ImportExport::Group::RelationFactory.overrides }
let(:ignore_relations) { %i[members user_contributions author user] }
let(:config_tree) do
Gitlab::ImportExport::Config.new(config: Gitlab::ImportExport.group_config_file).to_h[:tree][:group]
end
it_behaves_like 'define aliases'
end
describe 'project aliases' do
let(:overrides) { Gitlab::ImportExport::Project::RelationFactory.overrides }
let(:ignore_relations) { %i[project_members user_contributions author user] }
let(:config_tree) do
Gitlab::ImportExport::Config.new(config: Gitlab::ImportExport.config_file).to_h[:tree][:project]
end
it_behaves_like 'define aliases'
end
it "defines aliases for all note descendants apart from synthetic notes" do
user_associations = Note.reflect_on_all_associations(:belongs_to)
.reject(&:polymorphic?)
.filter { |association| association.klass == User }
.reject { |association| Note.columns.collect(&:name).exclude?(association.foreign_key) }
(Note.descendants - SyntheticNote.descendants - [SyntheticNote]).each do |descendant|
expect(described_class.aliases[descendant.to_s]).to be_present, missing_alias_message(descendant)
user_associations.each do |association|
foreign_key = association.foreign_key
last_version = described_class.aliases[descendant.to_s].keys.max
alias_definition = described_class.aliases[descendant.to_s][last_version]
expect(alias_definition[:columns]).to include(foreign_key),
missing_attribute_message(descendant, foreign_key)
end
end
end
end
describe ".version_for_model" do
@ -41,7 +135,7 @@ RSpec.describe Import::PlaceholderReferences::AliasResolver, feature_category: :
end
before do
stub_const("#{described_class}::ALIASES", aliases)
allow(described_class).to receive(:aliases).and_return(aliases)
end
it "returns the max version available for the model" do
@ -80,7 +174,7 @@ RSpec.describe Import::PlaceholderReferences::AliasResolver, feature_category: :
end
before do
stub_const("#{described_class}::ALIASES", aliases)
allow(described_class).to receive(:aliases).and_return(aliases)
end
it "returns the value for the right version" do
@ -106,7 +200,7 @@ RSpec.describe Import::PlaceholderReferences::AliasResolver, feature_category: :
end
before do
stub_const("#{described_class}::ALIASES", aliases)
allow(described_class).to receive(:aliases).and_return(aliases)
end
it "returns the new model name" do
@ -160,7 +254,7 @@ RSpec.describe Import::PlaceholderReferences::AliasResolver, feature_category: :
end
before do
stub_const("#{described_class}::ALIASES", aliases)
allow(described_class).to receive(:aliases).and_return(aliases)
end
it "returns the value for the right version" do
@ -184,7 +278,7 @@ RSpec.describe Import::PlaceholderReferences::AliasResolver, feature_category: :
end
before do
stub_const("#{described_class}::ALIASES", aliases)
allow(described_class).to receive(:aliases).and_return(aliases)
end
it "returns the new column name" do
@ -229,7 +323,7 @@ RSpec.describe Import::PlaceholderReferences::AliasResolver, feature_category: :
end
before do
stub_const("#{described_class}::ALIASES", aliases)
allow(described_class).to receive(:aliases).and_return(aliases)
end
it "only includes the last version" do

24
spec/requests/api/environments_spec.rb
View File

@ -6,7 +6,8 @@ RSpec.describe API::Environments, feature_category: :continuous_delivery do
let_it_be(:user) { create(:user) }
let_it_be(:developer) { create(:user) }
let_it_be(:non_member) { create(:user) }
let_it_be(:project) { create(:project, :private, :repository, namespace: user.namespace, maintainers: user, developers: developer) }
let_it_be(:reporter) { create(:user) }
let_it_be(:project) { create(:project, :private, :repository, namespace: user.namespace, maintainers: user, developers: developer, reporters: reporter) }
let_it_be_with_reload(:environment) { create(:environment, project: project) }
describe 'GET /projects/:id/environments', :aggregate_failures do
@ -383,11 +384,14 @@ RSpec.describe API::Environments, feature_category: :continuous_delivery do
deployable: job
)
environment.update!(cluster_agent: create(:cluster_agent, project: project))
get api("/projects/#{project.id}/environments/#{environment.id}", user)
expect(response).to have_gitlab_http_status(:ok)
expect(response).to match_response_schema('public_api/v4/environment')
expect(json_response['last_deployment']).to be_present
expect(json_response['cluster_agent']).to be_present
end
end
@ -443,6 +447,18 @@ RSpec.describe API::Environments, feature_category: :continuous_delivery do
end
end
context 'as a reporter' do
it 'does not expose the cluster agent' do
environment.update!(cluster_agent: create(:cluster_agent, project: project))
get api("/projects/#{project.id}/environments/#{environment.id}", reporter)
expect(response).to have_gitlab_http_status(:ok)
expect(response).to match_response_schema('public_api/v4/environment')
expect(json_response['cluster_agent']).not_to be_present
end
end
context 'as non member' do
shared_examples 'environment will not be found' do
it 'returns a 404 status code' do
@ -518,12 +534,6 @@ RSpec.describe API::Environments, feature_category: :continuous_delivery do
end
context "as a reporter" do
let(:reporter) { create(:user) }
before do
project.add_reporter(reporter)
end
it "rejects the request" do
delete api("/projects/#{project.id}/environments/review_apps", reporter)

12
spec/requests/import/source_users_controller_spec.rb
View File

@ -31,7 +31,7 @@ RSpec.describe Import::SourceUsersController, feature_category: :importers do
expect { subject }.not_to change { source_user.reload.status }
expect(response).to redirect_to(dashboard_groups_path)
expect(response).to redirect_to(root_path)
expect(flash[:raw]).to match(/Reassignment not available/)
end
end
@ -42,7 +42,7 @@ RSpec.describe Import::SourceUsersController, feature_category: :importers do
expect { subject }.not_to change { source_user.reload.status }
expect(response).to redirect_to(dashboard_groups_path)
expect(response).to redirect_to(root_path)
expect(flash[:raw]).to match(/Reassignment not available/)
end
end
@ -72,7 +72,7 @@ RSpec.describe Import::SourceUsersController, feature_category: :importers do
it 'redirects with a notice when accepted' do
accept_invite
expect(response).to redirect_to(dashboard_groups_path)
expect(response).to redirect_to(root_path)
expect(flash[:raw]).to match(/Reassignment approved/)
end
@ -82,7 +82,7 @@ RSpec.describe Import::SourceUsersController, feature_category: :importers do
accept_invite
expect(response).to redirect_to(dashboard_groups_path)
expect(response).to redirect_to(root_path)
expect(flash[:alert]).to match(/The invitation could not be accepted/)
end
@ -112,7 +112,7 @@ RSpec.describe Import::SourceUsersController, feature_category: :importers do
it 'redirects with a notice' do
reject_invite
expect(response).to redirect_to(dashboard_groups_path)
expect(response).to redirect_to(root_path)
expect(flash[:raw]).to match(/Reassignment rejected/)
end
@ -122,7 +122,7 @@ RSpec.describe Import::SourceUsersController, feature_category: :importers do
reject_invite
expect(response).to redirect_to(dashboard_groups_path)
expect(response).to redirect_to(root_path)
expect(flash[:alert]).to match(/The invitation could not be declined/)
end