146 lines
8.0 KiB
Markdown
146 lines
8.0 KiB
Markdown
---
|
|
stage: Systems
|
|
group: Distribution
|
|
info: Any user with at least the Maintainer role can merge updates to this content. For details, see https://docs.gitlab.com/ee/development/development_processes.html#development-guidelines-review.
|
|
---
|
|
|
|
# Avoiding required stops
|
|
|
|
Required stops are any changes to GitLab [components](architecture.md) or
|
|
dependencies that result in the need to upgrade to and stop at a specific
|
|
`major.minor` version when [upgrading GitLab](../update/index.md).
|
|
|
|
While Development maintains a [maintenance policy](../policy/maintenance.md)
|
|
that results in a three-release (3 month) backport window - GitLab maintains a
|
|
much longer window of [version support](https://about.gitlab.com/support/statement-of-support/#version-support)
|
|
that includes the current major version, as well as the two previous major
|
|
versions. Based on the schedule of previous major releases, GitLab customers can
|
|
lag behind the current release for up to three years and still expect to have
|
|
support for upgrades.
|
|
|
|
For example, a GitLab user upgrading from GitLab 14.0.12 to GitLab 16.1,
|
|
which is a fully supported [upgrade path](../update/index.md#upgrade-paths), may have
|
|
the following required stops: `14.3.6`, `14.9.5`, `14.10.5`, `15.0.5`, `15.1.6`,
|
|
`15.4.6`, and `15.11.11` before upgrading to the latest `16.1.z` version.
|
|
|
|
Past required stops have not been discovered for months after their introduction,
|
|
often as the result of extensive troubleshooting and assistance from Support
|
|
Engineers, Customer Success Managers, and Development Engineers as users upgrade
|
|
across greater than 1-3 minor releases.
|
|
|
|
Wherever possible, a required stop should be avoided. If it can't be avoided,
|
|
the required stop should be aligned to a _scheduled_ required stop.
|
|
|
|
In cases where we are considering retroactively declaring an unplanned required stop,
|
|
contact the [Distribution team product manager](https://about.gitlab.com/handbook/product/categories/#distributionbuild-group) to advise on next steps. If there
|
|
is uncertainty about whether we should declare a required stop, the Distribution product
|
|
manager may escalate to GitLab product leadership (VP or Chief Product Officer) to make
|
|
a final determination. This may happen, for example, if a change might require a stop for
|
|
a small subset of very large self-managed installations and there are well-defined workarounds
|
|
if customers run into issues.
|
|
|
|
Scheduled required stops are often implemented for the previous `major`.`minor`
|
|
release just prior to a `major` version release in order to accommodate multiple
|
|
[planned deprecations](../update/terminology.md#deprecation) and known
|
|
[breaking changes](../update/terminology.md#breaking-change).
|
|
|
|
Additionally, as of GitLab 16, we have introduced
|
|
[_scheduled_ `major`.`minor` required stops](../update/index.md#upgrade-paths):
|
|
|
|
>>>
|
|
During GitLab 16.x, we are scheduling two or three required upgrade stops.
|
|
|
|
We will give at least two milestones of notice when we schedule a required
|
|
upgrade stop. The first planned required upgrade stop is scheduled for GitLab
|
|
16.3. If nothing is introduced requiring an upgrade stop, GitLab 16.3 will be
|
|
treated as a regular upgrade.
|
|
>>>
|
|
|
|
## Causes of required stops
|
|
|
|
### Inaccurate assumptions about completed migrations
|
|
|
|
The majority of required stops are due to assumptions about the state of the
|
|
data model in a given release, usually in the form of interdependent database
|
|
migrations, or code changes that assume that schema changes introduced in
|
|
prior migrations have completed by the time the code loads.
|
|
|
|
Designing changes and migrations for [backwards compatibility between versions](multi_version_compatibility.md) can mitigate stop concerns with continuous or
|
|
"zero-downtime" upgrades. However, the **contract** phase will likely introduce
|
|
a required stop when a migration/code change is introduced that requires
|
|
that background migrations have completed before running or loading.
|
|
|
|
WARNING:
|
|
If you're considering adding or removing a migration, or introducing code that
|
|
assumes that migrations have completed in a given release, first review
|
|
the database-related documentation on [required stops](database/required_stops.md).
|
|
|
|
#### Examples
|
|
|
|
- GitLab `12.1`: Introduced a background migration changing `merge_status` in
|
|
MergeRequests depending on the `state` value. The `state` attribute was removed
|
|
in `12.10`. It took until `13.6` to document the required stop.
|
|
- GitLab `13.8`: Includes a background migration to deal with duplicate service
|
|
records. In `13.9`, a unique index was applied in another migration that
|
|
depended on the background migration completing. Not discovered/documented until
|
|
GitLab `14.3`
|
|
- GitLab `14.3`: Includes a potentially long-running background migration against
|
|
`merge_request_diff_commits` that was foregrounded in `14.5`. This change resulted in
|
|
extensive downtime for users with large GitLab installations. Not documented
|
|
until GitLab `15.1`
|
|
- GitLab `14.9`: Includes a batched background migration for `namespaces` and `projects`
|
|
that needs to finish before another batched background migration added in `14.10` executes,
|
|
forcing a required stop. The migration can take hours or days to complete on
|
|
large GitLab installations.
|
|
|
|
Additional details as well as links to related issues and merge requests can be
|
|
found in: [Issue: Put in place measures to avoid addition/proliferation of GitLab upgrade path stops](https://gitlab.com/gitlab-org/gitlab/-/issues/375553)
|
|
|
|
### Removal of code workarounds and mitigations
|
|
|
|
Similar to assumptions about the data model/schema/migration state, required
|
|
`major.minor` stops have been introduced due to the intentional removal of
|
|
code implemented to workaround previously discovered issues.
|
|
|
|
#### Examples
|
|
|
|
- GitLab `13.1`: A security fix in Rails `6.0.3.1` introduced a CSRF token change
|
|
(causing a canary environment incident). We introduced code to maintain acceptance
|
|
of previously generated tokens, and removed the code in `13.2`, creating a known
|
|
required stop in `13.1`.
|
|
- GitLab `15.4`: Introduces a migration to fix an inaccurate `expires_at` timestamp
|
|
for job artifacts that had been mitigated in code since GitLab `14.9`. The
|
|
workaround was removed in GitLab `15.6`, causing `15.4` to be a required stop.
|
|
|
|
### Deprecations
|
|
|
|
Deprecations, particularly [breaking changes](../update/terminology.md#breaking-change)
|
|
can also cause required stops if they introduce long migration delays or require
|
|
manual actions on the part of GitLab administrators.
|
|
|
|
These are generally accepted as a required stop around a major release, either
|
|
stopping at the latest `major.minor` release immediately proceeding
|
|
a new `major` release, and potentially the latest `major.0` patch release, and
|
|
to date, discovered required stops related to deprecations have been limited to
|
|
these releases.
|
|
|
|
#### Examples
|
|
|
|
Examples of deprecations are too numerous to be listed here, but can found
|
|
in the [deprecations and removals by version](../update/deprecations.md) as well
|
|
as the [version-specific upgrading instructions](../update/index.md#version-specific-upgrading-instructions),
|
|
[version-specific changes for the GitLab package (Omnibus)](../update/package/index.md#version-specific-changes),
|
|
and [GitLab chart upgrade notes](https://docs.gitlab.com/charts/installation/upgrade.html).
|
|
|
|
## Further reading
|
|
|
|
- [Documentation: Database - Adding required stops](database/required_stops.md)
|
|
- [Documentation: Upgrading GitLab](../update/index.md)
|
|
- [Package (Omnibus) upgrade](../update/package/index.md)
|
|
- [Docker upgrade](../install/docker.md#upgrade)
|
|
- [GitLab chart](https://docs.gitlab.com/charts/installation/upgrade.html)
|
|
- [Issue: Put in place measures to avoid addition/proliferation of GitLab upgrade path stops](https://gitlab.com/gitlab-org/gitlab/-/issues/375553)
|
|
- [Issue: Brainstorm ways for background migrations to be finalized without introducing a required upgrade step](https://gitlab.com/gitlab-org/gitlab/-/issues/357561)
|
|
- [Issue: Scheduled required paths for GitLab upgrades to improve UX](https://gitlab.com/gitlab-org/gitlab/-/issues/358417)
|
|
- [Epic: GitLab Releases and Maintenance policies](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/988)
|