|
|
|
|
@ -28,15 +28,14 @@ Note: Please make sure to run the chatops commands in the Slack channel that get
|
|
|
|
|
|
|
|
|
|
### Rollout on non-production environments
|
|
|
|
|
|
|
|
|
|
- Verify the MR with the feature flag is merged to `master` and have been deployed to non-production environments with `/chatops run auto_deploy status <merge-commit-of-your-feature>`
|
|
|
|
|
- Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops run auto_deploy status <merge-commit-of-your-feature>`
|
|
|
|
|
<!-- Delete Incremental roll out if it is not relevant to this deploy -->
|
|
|
|
|
- [ ] Deploy the feature flag at a percentage (recommended percentage: 50%) with `/chatops run feature set <feature-flag-name> <rollout-percentage> --actors --dev --pre --staging --staging-ref`
|
|
|
|
|
- [ ] Monitor that the error rates did not increase (repeat with a different percentage as necessary).
|
|
|
|
|
<!-- End of block for deletes -->
|
|
|
|
|
- [ ] Enable the feature globally on non-production environments with `/chatops run feature set <feature-flag-name> true --dev --pre --staging --staging-ref`
|
|
|
|
|
- [ ] Verify that the feature works as expected.
|
|
|
|
|
The best environment to validate the feature in is [`staging-canary`](https://about.gitlab.com/handbook/engineering/infrastructure/environments/#staging-canary)
|
|
|
|
|
as this is the first environment deployed to. Make sure you are [configured to use canary](https://next.gitlab.com/).
|
|
|
|
|
The best environment to validate the feature in is [`staging-canary`](https://about.gitlab.com/handbook/engineering/infrastructure/environments/#staging-canary) as this is the first environment deployed to. Make sure you are [configured to use canary](https://next.gitlab.com/).
|
|
|
|
|
- [ ] If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking [deployments](https://about.gitlab.com/handbook/engineering/deployments-and-releases/deployments/).
|
|
|
|
|
- See [`#e2e-run-staging` Slack channel](https://gitlab.enterprise.slack.com/archives/CBS3YKMGD) and look for the following messages:
|
|
|
|
|
- test kicked off: `Feature flag <feature-flag-name> has been set to true on **gstg**`
|
|
|
|
|
@ -46,7 +45,7 @@ For assistance with end-to-end test failures, please reach out via the [`#test-p
|
|
|
|
|
|
|
|
|
|
### Specific rollout on production
|
|
|
|
|
|
|
|
|
|
For visibility, all `/chatops` commands that target production should be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796)
|
|
|
|
|
For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796)
|
|
|
|
|
and cross-posted (with the command results) to the responsible team's Slack channel.
|
|
|
|
|
|
|
|
|
|
- Ensure that the feature MRs have been deployed to both production and canary with `/chatops run auto_deploy status <merge-commit-of-your-feature>`
|
|
|
|
|
@ -54,6 +53,7 @@ and cross-posted (with the command results) to the responsible team's Slack chan
|
|
|
|
|
- For **project-actor**: `/chatops run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss,gitlab-com/www-gitlab-com <feature-flag-name> true`
|
|
|
|
|
- For **group-actor**: `/chatops run feature set --group=gitlab-org,gitlab-com <feature-flag-name> true`
|
|
|
|
|
- For **user-actor**: `/chatops run feature set --user=<gitlab-username-of-dri> <feature-flag-name> true`
|
|
|
|
|
- For **all internal users**: `/chatops run feature set --feature-group=gitlab_team_members <feature-flag-name> true`
|
|
|
|
|
- [ ] Verify that the feature works for the specific actors.
|
|
|
|
|
|
|
|
|
|
### Preparation before global rollout
|
|
|
|
|
@ -65,23 +65,17 @@ and cross-posted (with the command results) to the responsible team's Slack chan
|
|
|
|
|
- [ ] Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production.
|
|
|
|
|
If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the `@sre-oncall` Slack alias.
|
|
|
|
|
- [ ] Ensure that documentation exists for the feature, and the [version history text](https://docs.gitlab.com/ee/development/documentation/feature_flags.html#add-version-history-text) has been updated.
|
|
|
|
|
- [ ] Leave a comment on [the feature issue](<feature-issue-link>) announcing estimated time when this feature flag will be enabled on GitLab.com.
|
|
|
|
|
- [ ] Ensure that any breaking changes have been announced following the [release post process](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations-removals-and-breaking-changes) to ensure GitLab customers are aware.
|
|
|
|
|
- [ ] Notify the [`#support_gitlab-com` Slack channel](https://gitlab.slack.com/archives/C4XFU81LG) and your team channel ([more guidance when this is necessary in the dev docs](https://docs.gitlab.com/ee/development/feature_flags/controls.html#communicate-the-change)).
|
|
|
|
|
- [ ] Ensure that the feature flag rollout plan is reviewed by another developer familiar with the domain.
|
|
|
|
|
|
|
|
|
|
### Global rollout on production
|
|
|
|
|
|
|
|
|
|
For visibility, all `/chatops` commands that target production should be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796)
|
|
|
|
|
and cross-posted (with the command results) to the responsible team's Slack channel (`#<slack-channel-of-dri-team>`).
|
|
|
|
|
For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796)
|
|
|
|
|
and cross-posted (with the command results) to the responsible team's Slack channel.
|
|
|
|
|
|
|
|
|
|
- [ ] (Optional) [Incrementally roll out](https://docs.gitlab.com/ee/development/feature_flags/controls.html#process) the feature on production environment.
|
|
|
|
|
- [ ] [Incrementally roll out](https://docs.gitlab.com/ee/development/feature_flags/controls.html#process) the feature on production.
|
|
|
|
|
- Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net.
|
|
|
|
|
- Perform **actor-based** rollout: `/chatops run feature set <feature-flag-name> <rollout-percentage> --actors`
|
|
|
|
|
- [ ] Enable the feature globally on production environment: `/chatops run feature set <feature-flag-name> true`
|
|
|
|
|
- [ ] Observe appropriate graphs on https://dashboards.gitlab.net and verify that services are not affected.
|
|
|
|
|
- [ ] Leave a comment on [the feature issue](<feature-issue-link>) announcing that the feature has been globally enabled.
|
|
|
|
|
- [ ] Wait for [at least one day for the verification term](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release).
|
|
|
|
|
- [ ] After the feature has been 100% enabled, wait for [at least one day before releasing the feature](#release-the-feature).
|
|
|
|
|
|
|
|
|
|
### (Optional) Release the feature with the feature flag
|
|
|
|
|
|
|
|
|
|
@ -98,15 +92,15 @@ If you're still unsure whether the feature is [deemed stable](https://about.gitl
|
|
|
|
|
but want to release it in the current milestone, you can change the default state of the feature flag to be enabled.
|
|
|
|
|
To do so, follow these steps:
|
|
|
|
|
|
|
|
|
|
- [ ] Create a merge request with the following changes. Ask for review and merge it.
|
|
|
|
|
- [ ] Create a merge request with the following changes.
|
|
|
|
|
- [ ] If feature was enabled for various actors, ensure the feature has been enabled globally on production `/chatops run feature get <feature-flag-name>`. If the feature has not been globally enabled then enable the feature globally using: `/chatops run feature set <feature-flag-name> true`
|
|
|
|
|
- [ ] Set the `default_enabled` attribute in [the feature flag definition](https://docs.gitlab.com/ee/development/feature_flags/#feature-flag-definition-and-validation) to `true`.
|
|
|
|
|
- [ ] Decide [which changelog entry](https://docs.gitlab.com/ee/development/feature_flags/#changelog) is needed.
|
|
|
|
|
- [ ] Ensure that the default-enabling MR has been included in the release package.
|
|
|
|
|
If the merge request was deployed before [the monthly release was tagged](https://about.gitlab.com/handbook/engineering/releases/#self-managed-releases-1),
|
|
|
|
|
the feature can be officially announced in a release blog post: `/chatops run release check <merge-request-url> <milestone>`
|
|
|
|
|
- [ ] Consider cleaning up the feature flag from all environments by running these chatops command in `#production` channel. Otherwise these settings may override the default enabled: `/chatops run feature delete <feature-flag-name> --dev --pre --staging --staging-ref --production`
|
|
|
|
|
- [ ] Close [the feature issue](<feature-issue-link>) to indicate the feature will be released in the current milestone.
|
|
|
|
|
- [ ] After the default-enabling MR has been deployed, clean up the feature flag from all environments by running these chatops command in the `#production` channel: `/chatops run feature delete <feature-flag-name> --dev --pre --staging --staging-ref --production`
|
|
|
|
|
- [ ] Close [the feature issue][<feature-issue-link>] to indicate the feature will be released in the current milestone.
|
|
|
|
|
- [ ] Set the next milestone to this rollout issue for scheduling [the flag removal](#release-the-feature).
|
|
|
|
|
- [ ] (Optional) You can [create a separate issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20Flag%20Cleanup) for scheduling the steps below to [Release the feature](#release-the-feature).
|
|
|
|
|
- [ ] Set the title to "[Feature flag] Cleanup `<feature-flag-name>`".
|
|
|
|
|
@ -120,21 +114,21 @@ To do so, follow these steps:
|
|
|
|
|
|
|
|
|
|
After the feature has been [deemed stable](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release),
|
|
|
|
|
the [clean up](https://docs.gitlab.com/ee/development/feature_flags/controls.html#cleaning-up)
|
|
|
|
|
should be done as soon as possible to permanently enable the feature and reduce complexity in the
|
|
|
|
|
codebase.
|
|
|
|
|
should be done as soon as possible to permanently enable the feature and reduce
|
|
|
|
|
complexity in the codebase.
|
|
|
|
|
|
|
|
|
|
You can either [create a follow-up issue for Feature Flag Cleanup](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20Flag%20Cleanup) or use the checklist below in this same issue.
|
|
|
|
|
You can either [create a follow-up issue for Feature Flag Cleanup](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20Flag%20Cleanup)
|
|
|
|
|
or use the checklist below in this same issue.
|
|
|
|
|
|
|
|
|
|
<!-- The checklist here is to help stakeholders keep track of the feature flag status -->
|
|
|
|
|
- [ ] Create a merge request to remove the `<feature-flag-name>` feature flag. Ask for review/approval/merge as usual. The MR should include the following changes:
|
|
|
|
|
- Remove all references to the feature flag from the codebase.
|
|
|
|
|
- Remove the YAML definitions for the feature from the repository.
|
|
|
|
|
- Create [a changelog entry](https://docs.gitlab.com/ee/development/feature_flags/#changelog).
|
|
|
|
|
- [ ] Ensure that the cleanup MR has been included in the release package.
|
|
|
|
|
If the merge request was deployed before [the monthly release was tagged](https://about.gitlab.com/handbook/engineering/releases/#self-managed-releases-1),
|
|
|
|
|
the feature can be officially announced in a release blog post: `/chatops run release check <merge-request-url> <milestone>`
|
|
|
|
|
- [ ] Close [the feature issue](<feature-issue-link>) to indicate the feature will be released in the current milestone.
|
|
|
|
|
- [ ] Clean up the feature flag from all environments by running these chatops command in `#production` channel: `/chatops run feature delete <feature-flag-name> --dev --pre --staging --staging-ref --production`
|
|
|
|
|
- [ ] Close [the feature issue][<feature-issue-link>] to indicate the feature will be released in the current milestone.
|
|
|
|
|
- [ ] Once the cleanup MR has been deployed to production, clean up the feature flag from all environments by running these chatops command in `#production` channel: `/chatops run feature delete <feature-flag-name> --dev --pre --staging --staging-ref --production`
|
|
|
|
|
- [ ] Close this rollout issue.
|
|
|
|
|
|
|
|
|
|
## Rollback Steps
|
|
|
|
|
|