root
/
gitlab-ce
mirror of https://gitlab.com/free-releases/gitlab-ce.git
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

Add latest changes from gitlab-org/gitlab@master

This commit is contained in:
GitLab Bot 2023-07-18 09:10:04 +00:00
parent 6e2c0f81af
commit 16887b928c
10 changed files with 213 additions and 130 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
app/assets/javascripts/diffs/components/commit_item.vue
View File

@ -2,7 +2,7 @@
import { GlButtonGroup, GlButton, GlTooltipDirective, GlFormCheckbox } from '@gitlab/ui';
import SafeHtml from '~/vue_shared/directives/safe_html';
import CommitPipelineStatus from '~/projects/tree/components/commit_pipeline_status_component.vue';
import CommitPipelineStatus from '~/projects/tree/components/commit_pipeline_status.vue';
import ModalCopyButton from '~/vue_shared/components/modal_copy_button.vue';
import TimeAgoTooltip from '~/vue_shared/components/time_ago_tooltip.vue';
import UserAvatarLink from '~/vue_shared/components/user_avatar/user_avatar_link.vue';

2
app/assets/javascripts/pages/projects/blob/show/index.js
View File

@ -11,7 +11,7 @@ import createDefaultClient from '~/lib/graphql';
import initBlob from '~/pages/projects/init_blob';
import ForkInfo from '~/repository/components/fork_info.vue';
import initWebIdeLink from '~/pages/projects/shared/web_ide_link';
import CommitPipelineStatus from '~/projects/tree/components/commit_pipeline_status_component.vue';
import CommitPipelineStatus from '~/projects/tree/components/commit_pipeline_status.vue';
import BlobContentViewer from '~/repository/components/blob_content_viewer.vue';
import '~/sourcegraph/load';
import createStore from '~/code_navigation/store';

0
app/assets/javascripts/projects/tree/components/commit_pipeline_status_component.vue → app/assets/javascripts/projects/tree/components/commit_pipeline_status.vue
View File

201
doc/update/background_migrations.md
View File

@ -6,92 +6,25 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Background migrations and upgrades **(FREE SELF)**
> - Batched background migrations [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51332) in GitLab 13.11 [with a flag](../user/feature_flags.md) named `execute_batched_migrations_on_schedule`. Disabled by default.
> - Feature flag `execute_batched_migrations_on_schedule` [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511) in GitLab 13.12.
> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](../development/database/batched_background_migrations.md#enable-or-disable-background-migrations).
Certain releases may require different migrations to be finished before you
update to the newer version. Two kinds of migrations exist. They differ, and you
should check that both are complete before upgrading GitLab:
- [Batched background migrations](#batched-background-migrations), available in GitLab 14.0 and later.
- [Batched background migrations](#batched-background-migrations), most
commonly used in GitLab 14.0 and later.
- [Background migrations](#background-migrations) that are not batched.
Most commonly used in GitLab 13.11 and earlier.
To decrease the time required to complete these migrations, increase the number of
[Sidekiq workers](../administration/sidekiq/extra_sidekiq_processes.md)
that can process jobs in the `background_migration` queue.
## Background migrations
### Check for pending background migrations
To check for pending background migrations:
::Tabs
:::TabTitle Linux package (Omnibus)
```shell
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count'
```
:::TabTitle Self-compiled (source)
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count'
```
::EndTabs
### Check for failed background migrations
To check for background migrations that have failed:
::Tabs
:::TabTitle Linux package (Omnibus)
For GitLab versions 14.10 and later:
```shell
sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count'
```
For GitLab versions 14.0-14.9:
```shell
sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count'
```
:::TabTitle Self-compiled (source)
For GitLab versions 14.10 and later:
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count'
```
For GitLab versions 14.0-14.9:
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count'
```
::EndTabs
## Batched background migrations
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51332) in GitLab 13.11, [behind a feature flag](../user/feature_flags.md), disabled by default.
> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511) in GitLab 13.12.
> - Enabled on GitLab.com.
> - Recommended for production use.
> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](../development/database/batched_background_migrations.md#enable-or-disable-background-migrations).
WARNING:
There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features).
Refer to this feature's version history for more details.
To update database tables in batches, GitLab can use batched background migrations. These migrations
are created by GitLab developers and run automatically on upgrade. However, such migrations are
limited in scope to help with migrating some `integer` database columns to `bigint`. This is needed to
@ -163,7 +96,17 @@ To query the database directly for the status of batched background migrations:
WHERE status <> 3;
```
### Pause batched background migrations in GitLab 14.x
### Enable or disable advanced features
Batched background migrations provide feature flags that enable you to customize
migrations or pause them entirely. These feature flags should only be disabled by
advanced users who understand the risks of doing so.
#### Pause batched background migrations in GitLab 14.x
WARNING:
There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features).
Refer to each feature's version history for more details.
To pause an ongoing batched background migration,
[disable the batched background migrations feature](../development/database/batched_background_migrations.md#enable-or-disable-background-migrations).
@ -214,47 +157,25 @@ Use the following database queries to see the state of the current batched backg
command above) to proceed with the batch when ready. On larger instances,
background migrations can take as long as 48 hours to complete each batch.
### Automatic batch size optimization
#### Automatic batch size optimization
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60133)
> in GitLab 13.12, [behind a feature flag](../user/feature_flags.md),
> [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511).
> - Enabled on GitLab.com.
> - Recommended for production use.
> - For GitLab self-managed instances, GitLab administrators can opt to
> [disable it](#enable-or-disable-automatic-batch-size-optimization).
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60133) in GitLab 13.2 [with a flag](../administration/feature_flags.md) named `optimize_batched_migrations`. Enabled by default.
WARNING:
There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features).
Refer to this feature's version history for more details.
FLAG:
On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `optimize_batched_migrations`.
On GitLab.com, this feature is available.
To maximize throughput of batched background migrations (in terms of the number of tuples updated per time unit), batch sizes are automatically adjusted based on how long the previous batches took to complete.
### Enable or disable automatic batch size optimization
#### Parallel execution
Automatic batch size optimization for batched background migrations is under development but ready for production use.
It is deployed behind a feature flag that is **enabled by default**.
[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md)
can opt to disable it.
To enable it:
```ruby
Feature.enable(:optimize_batched_migrations)
```
To disable it:
```ruby
Feature.disable(:optimize_batched_migrations)
```
### Parallel execution
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104027)
> in GitLab 15.7, [behind a feature flag](../user/feature_flags.md),
> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/372316) in GitLab 15.11.
> - Feature flag `batched_migrations_parallel_execution` removed in GitLab 16.1.
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104027) in GitLab 15.7 [with a flag](../administration/feature_flags.md) named `batched_migrations_parallel_execution`. Disabled by default.
> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/372316) in GitLab 15.11.
> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/120808) in GitLab 16.1. Feature flag `batched_migrations_parallel_execution` removed.
WARNING:
There can be [risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features).
@ -314,6 +235,72 @@ To monitor the retried batched background migrations, you can
[check the status of batched background migrations](#check-the-status-of-batched-background-migrations)
on a regular interval.
## Background migrations
In GitLab 13, background migrations were not batched. In GitLab 14 and later, this
type of migration was superseded by batched background migrations.
### Check for pending background migrations
To check for pending non-batched background migrations:
::Tabs
:::TabTitle Linux package (Omnibus)
```shell
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count'
```
:::TabTitle Self-compiled (source)
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count'
```
::EndTabs
### Check for failed background migrations
To check for non-batched background migrations that have failed:
::Tabs
:::TabTitle Linux package (Omnibus)
For GitLab versions 14.10 and later:
```shell
sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count'
```
For GitLab versions 14.0-14.9:
```shell
sudo gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count'
```
:::TabTitle Self-compiled (source)
For GitLab versions 14.10 and later:
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count'
```
For GitLab versions 14.0-14.9:
```shell
cd /home/git/gitlab
sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.failed.count'
```
::EndTabs
## Troubleshooting
### Database migrations failing because of batched background migration not finished

36
lib/gitlab/issues/rebalancing/state.rb
View File

@ -53,7 +53,7 @@ module Gitlab
end
def can_start_rebalance?
rebalance_in_progress? || too_many_rebalances_running?
rebalance_in_progress? || concurrent_rebalance_within_limit?
end
def cache_issue_ids(issue_ids)
@ -100,11 +100,11 @@ module Gitlab
def refresh_keys_expiration
with_redis do |redis|
Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
redis.multi do |multi|
multi.expire(issue_ids_key, REDIS_EXPIRY_TIME)
multi.expire(current_index_key, REDIS_EXPIRY_TIME)
multi.expire(current_project_key, REDIS_EXPIRY_TIME)
multi.expire(CONCURRENT_RUNNING_REBALANCES_KEY, REDIS_EXPIRY_TIME)
redis.pipelined do |pipeline|
pipeline.expire(issue_ids_key, REDIS_EXPIRY_TIME)
pipeline.expire(current_index_key, REDIS_EXPIRY_TIME)
pipeline.expire(current_project_key, REDIS_EXPIRY_TIME)
pipeline.expire(CONCURRENT_RUNNING_REBALANCES_KEY, REDIS_EXPIRY_TIME)
end
end
end
@ -113,16 +113,20 @@ module Gitlab
def cleanup_cache
value = "#{rebalanced_container_type}/#{rebalanced_container_id}"
# The clean up is done sequentially to be compatible with Redis Cluster
# Do not use a pipeline as it fans-out in a Redis-Cluster setting and forego ordering guarantees
with_redis do |redis|
Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
redis.multi do |multi|
multi.del(issue_ids_key)
multi.del(current_index_key)
multi.del(current_project_key)
multi.srem?(CONCURRENT_RUNNING_REBALANCES_KEY, value)
multi.set(self.class.recently_finished_key(rebalanced_container_type, rebalanced_container_id), true, ex: 1.hour)
end
end
# srem followed by .del(issue_ids_key) to ensure that any subsequent redis errors would
# result in a no-op job retry since current_index_key still exists
redis.srem?(CONCURRENT_RUNNING_REBALANCES_KEY, value)
redis.del(issue_ids_key)
# delete current_index_key to ensure that subsequent redis errors would
# result in a fresh job retry
redis.del(current_index_key)
# setting recently_finished_key last after job details is cleaned up
redis.set(self.class.recently_finished_key(rebalanced_container_type, rebalanced_container_id), true, ex: 1.hour)
end
end
@ -159,7 +163,7 @@ module Gitlab
attr_accessor :root_namespace, :projects, :rebalanced_container_type, :rebalanced_container_id
def too_many_rebalances_running?
def concurrent_rebalance_within_limit?
concurrent_running_rebalances_count <= MAX_NUMBER_OF_CONCURRENT_REBALANCES
end

2
qa/qa/resource/project_imported_from_github.rb
View File

@ -69,7 +69,7 @@ module QA
single_endpoint_notes_import: full_notes_import,
attachments_import: attachments_import
}
}
}.compact
end
def transform_api_resource(api_resource)

2
spec/frontend/commit/commit_pipeline_status_component_spec.js → spec/frontend/commit/commit_pipeline_status_spec.js
View File

@ -5,7 +5,7 @@ import { nextTick } from 'vue';
import fixture from 'test_fixtures/pipelines/pipelines.json';
import { createAlert } from '~/alert';
import Poll from '~/lib/utils/poll';
import CommitPipelineStatus from '~/projects/tree/components/commit_pipeline_status_component.vue';
import CommitPipelineStatus from '~/projects/tree/components/commit_pipeline_status.vue';
import CiIcon from '~/vue_shared/components/ci_icon.vue';
jest.mock('~/lib/utils/poll');

2
spec/frontend/diffs/components/commit_item_spec.js
View File

@ -5,7 +5,7 @@ import { TEST_HOST } from 'helpers/test_constants';
import { trimText } from 'helpers/text_helper';
import Component from '~/diffs/components/commit_item.vue';
import { getTimeago } from '~/lib/utils/datetime_utility';
import CommitPipelineStatus from '~/projects/tree/components/commit_pipeline_status_component.vue';
import CommitPipelineStatus from '~/projects/tree/components/commit_pipeline_status.vue';
const TEST_AUTHOR_NAME = 'test';
const TEST_AUTHOR_EMAIL = 'test+test@gitlab.com';

3
spec/lib/gitlab/issues/rebalancing/state_spec.rb
View File

@ -160,12 +160,11 @@ RSpec.describe Gitlab::Issues::Rebalancing::State, :clean_gitlab_redis_shared_st
before do
generate_and_cache_issues_ids(count: 3)
rebalance_caching.cache_current_index(123)
rebalance_caching.cache_current_project_id(456)
rebalance_caching.track_new_running_rebalance
end
it 'removes cache keys' do
expect(check_existing_keys).to eq(4)
expect(check_existing_keys).to eq(3)
rebalance_caching.cleanup_cache

93
spec/services/issues/relative_position_rebalancing_service_spec.rb
View File

@ -156,5 +156,98 @@ RSpec.describe Issues::RelativePositionRebalancingService, :clean_gitlab_redis_s
subject.execute
end
end
shared_examples 'no-op on the retried job' do
it 'does not update positions in the 2nd .execute' do
original_order = issues_in_position_order.map(&:id)
# preloads issue ids on both runs
expect(service).to receive(:preload_issue_ids).twice.and_call_original
# 1st run performs rebalancing
expect(service).to receive(:update_positions_with_retry).exactly(9).times.and_call_original
expect { service.execute }.to raise_error(StandardError)
# 2nd run is a no-op
expect(service).not_to receive(:update_positions_with_retry)
expect { service.execute }.to raise_error(StandardError)
# order is preserved
expect(original_order).to match_array(issues_in_position_order.map(&:id))
end
end
context 'when error is raised in cache cleanup step' do
let_it_be(:root_namespace_id) { project.root_namespace.id }
context 'when srem fails' do
before do
Gitlab::Redis::SharedState.with do |redis|
allow(redis).to receive(:srem?).and_raise(StandardError)
end
end
it_behaves_like 'no-op on the retried job'
end
context 'when delete issues ids sorted set fails' do
before do
Gitlab::Redis::SharedState.with do |redis|
allow(redis).to receive(:del).and_call_original
allow(redis).to receive(:del)
.with("#{Gitlab::Issues::Rebalancing::State::REDIS_KEY_PREFIX}:#{root_namespace_id}")
.and_raise(StandardError)
end
end
it_behaves_like 'no-op on the retried job'
end
context 'when delete current_index_key fails' do
before do
Gitlab::Redis::SharedState.with do |redis|
allow(redis).to receive(:del).and_call_original
allow(redis).to receive(:del)
.with("#{Gitlab::Issues::Rebalancing::State::REDIS_KEY_PREFIX}:#{root_namespace_id}:current_index")
.and_raise(StandardError)
end
end
it_behaves_like 'no-op on the retried job'
end
context 'when setting recently finished key fails' do
before do
Gitlab::Redis::SharedState.with do |redis|
allow(redis).to receive(:set).and_call_original
allow(redis).to receive(:set)
.with(
"#{Gitlab::Issues::Rebalancing::State::RECENTLY_FINISHED_REBALANCE_PREFIX}:2:#{project.id}",
anything,
anything
)
.and_raise(StandardError)
end
end
it 'reruns the next job in full' do
original_order = issues_in_position_order.map(&:id)
# preloads issue ids on both runs
expect(service).to receive(:preload_issue_ids).twice.and_call_original
# 1st run performs rebalancing
expect(service).to receive(:update_positions_with_retry).exactly(9).times.and_call_original
expect { service.execute }.to raise_error(StandardError)
# 2nd run performs rebalancing in full
expect(service).to receive(:update_positions_with_retry).exactly(9).times.and_call_original
expect { service.execute }.to raise_error(StandardError)
# order is preserved
expect(original_order).to match_array(issues_in_position_order.map(&:id))
end
end
end
end
end