Add latest changes from gitlab-org/gitlab@master

doc/administration/troubleshooting/elasticsearch.md

@@ -36,6 +36,7 @@ The type of problem will determine what steps to take. The possible troubleshoot
 - Indexing.
 - Integration.
 - Performance.
+- Background Migrations.
 
 ### Search Results workflow
 
@@ -147,6 +148,30 @@ graph TD;
   F7(Escalate to<br>GitLab support.)
 ```
 
+### Background Migrations workflow
+
+```mermaid
+graph TD;
+  D --> |No| D1
+  D --> |Yes| D2
+  D2 --> |No| D3
+  D2 --> |Yes| D4
+  D4 --> |No| D5
+  D4 --> |Yes| D6
+  D6 --> |No| D8
+  D6 --> |Yes| D7
+
+  D{Is there a halted migration?}
+  D1[Migrations run in the<br>background and will<br>stop when completed.]
+  D2{Does the elasticsearch.log<br>file contain errors?}
+  D3[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.]
+  D4{Have the errors<br>been addressed?}
+  D5[Have an Elasticsearch admin<br>review and address<br>the errors.]
+  D6{Has the migration<br>been retried?}
+  D7[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.]
+  D8[Retry the migration from<br>the Admin > Settings ><br>Advanced Search UI.]
+```
+
 ## Troubleshooting walkthrough
 
 Most Elasticsearch troubleshooting can be broken down into 4 categories:
@@ -155,6 +180,7 @@ Most Elasticsearch troubleshooting can be broken down into 4 categories:
 - [Troubleshooting indexing](#troubleshooting-indexing)
 - [Troubleshooting integration](#troubleshooting-integration)
 - [Troubleshooting performance](#troubleshooting-performance)
+- [Troubleshooting background migrations](#troubleshooting-background-migrations)
 
 Generally speaking, if it does not fall into those four categories, it is either:
 
@@ -330,6 +356,18 @@ dig further into these.
 Feel free to reach out to GitLab support, but this is likely to be something a skilled
 Elasticsearch admin has more experience with.
 
+### Troubleshooting background migrations
+
+Troubleshooting background migration failures can be difficult and may require contacting 
+an Elasticsearch admin or GitLab Support.
+
+The best place to start while debugging issues with a background migration is the 
+[`elasticsearch.log` file](../logs.md#elasticsearchlog). Migrations will
+print information while a migration is in progress and any errors encountered.
+Apply fixes for any errors found in the log and retry the migration.
+
+If you still encounter issues after retrying the migration, reach out to GitLab support.
+
 ## Common issues
 
 All common issues [should be documented](../../integration/elasticsearch.md#troubleshooting). If not,

doc/development/elasticsearch.md

@@ -216,6 +216,9 @@ cron worker sequentially.
 
 Any update to the Elastic index mappings should be replicated in [`Elastic::Latest::Config`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb).
 
+Migrations can be built with a retry limit and have the ability to be [failed and marked as halted](https://gitlab.com/gitlab-org/gitlab/-/blob/66e899b6637372a4faf61cfd2f254cbdd2fb9f6d/ee/lib/elastic/migration.rb#L40).
+Any data or index cleanup needed to support migration retries should be handled within the migration.
+
 ### Migration options supported by the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb)
 
 - `batched!` - Allow the migration to run in batches. If set, the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb)

doc/integration/elasticsearch.md

@@ -508,6 +508,15 @@ This should return something similar to:
 
 In order to debug issues with the migrations you can check the [`elasticsearch.log` file](../administration/logs.md#elasticsearchlog).
 
+### Retry a halted migration
+
+Some migrations are built with a retry limit. If the migration cannot finish within the retry limit,
+it will be halted and a notification will be displayed in the Advanced Search integration settings.
+It is recommended to check the [`elasticsearch.log` file](../administration/logs.md#elasticsearchlog) to
+debug why the migration was halted and make any changes before retrying the migration. Once you believe you've 
+fixed the cause of the failure, click "Retry migration", and the migration will be scheduled to be retried 
+in the background.
+
 ## GitLab Advanced Search Rake tasks
 
 Rake tasks are available to:

locale/gitlab.pot

@@ -3805,6 +3805,9 @@ msgstr ""
 msgid "Are you sure you want to reset the registration token?"
 msgstr ""
 
+msgid "Are you sure you want to retry this migration?"
+msgstr ""
+
 msgid "Are you sure you want to revoke this %{type}? This action cannot be undone."
 msgstr ""
 
@@ -5398,6 +5401,9 @@ msgstr ""
 msgid "Check the %{docs_link_start}documentation%{docs_link_end}."
 msgstr ""
 
+msgid "Check the elasticsearch.log file to debug why the migration was halted and make any changes before retrying the migration. When you fix the cause of the failure, click \"Retry migration\", and the migration will be scheduled to be retried in the background."
+msgstr ""
+
 msgid "Check your Docker images for known vulnerabilities."
 msgstr ""
 
@@ -18215,6 +18221,9 @@ msgstr ""
 msgid "Migrated %{success_count}/%{total_count} files."
 msgstr ""
 
+msgid "Migration has been scheduled to be retried"
+msgstr ""
+
 msgid "Migration successful."
 msgstr ""
 
@@ -24305,6 +24314,9 @@ msgstr ""
 msgid "Retry job"
 msgstr ""
 
+msgid "Retry migration"
+msgstr ""
+
 msgid "Retry this job"
 msgstr ""
 
@@ -28460,6 +28472,9 @@ msgstr ""
 msgid "There are running deployments on the environment. Please retry later."
 msgstr ""
 
+msgid "There is a halted Elasticsearch migration"
+msgstr ""
+
 msgid "There is already a To-Do for this design."
 msgstr ""