Add latest changes from gitlab-org/gitlab@master
This commit is contained in:
parent
d17f94b8d7
commit
20cb836449
|
|
@ -28,8 +28,8 @@ pages:
|
|||
- cp .public/assets/application-*.css public/application.css || true
|
||||
- mv $KNAPSACK_RSPEC_SUITE_REPORT_PATH public/$KNAPSACK_RSPEC_SUITE_REPORT_PATH || true
|
||||
- mv $FLAKY_RSPEC_SUITE_REPORT_PATH public/$FLAKY_RSPEC_SUITE_REPORT_PATH || true
|
||||
- mv $RSPEC_PACKED_TESTS_MAPPING_PATH.gz public/$RSPEC_PACKED_TESTS_MAPPING_PATH || true
|
||||
- mv $RSPEC_PACKED_TESTS_MAPPING_ALT_PATH.gz public/$RSPEC_PACKED_TESTS_MAPPING_ALT_PATH || true
|
||||
- mv $RSPEC_PACKED_TESTS_MAPPING_PATH.gz public/$RSPEC_PACKED_TESTS_MAPPING_PATH.gz || true
|
||||
- mv $RSPEC_PACKED_TESTS_MAPPING_ALT_PATH.gz public/$RSPEC_PACKED_TESTS_MAPPING_ALT_PATH.gz || true
|
||||
- mv $FRONTEND_FIXTURES_MAPPING_PATH public/$FRONTEND_FIXTURES_MAPPING_PATH || true
|
||||
- *compress-public
|
||||
artifacts:
|
||||
|
|
|
|||
|
|
@ -1,176 +1,11 @@
|
|||
---
|
||||
stage: Data Stores
|
||||
group: Database
|
||||
description: 'Database Scalability / Limit table sizes'
|
||||
redirect_to: 'https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/database_size_limits/'
|
||||
remove_date: '2025-07-19'
|
||||
---
|
||||
|
||||
# Database Scalability: Limit on-disk table size to < 100 GB for GitLab.com
|
||||
This document was moved to [another location](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/database_size_limits/).
|
||||
|
||||
This document is a proposal to work towards reducing and limiting table sizes on GitLab.com. We establish a **measurable target** by limiting table size to a certain threshold. This is used as an indicator to drive database focus and decision making. With GitLab.com growing, we continuously re-evaluate which tables need to be worked on to prevent or otherwise fix violations.
|
||||
|
||||
This is not meant to be a hard rule but rather a strong indication that work needs to be done to break a table apart or otherwise reduce its size.
|
||||
|
||||
This is meant to be read in context with the [Database Sharding blueprint](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64115),
|
||||
which paints the bigger picture. This proposal here is thought to be part of the "debloating step" below, as we aim to reduce storage requirements and improve data modeling. Partitioning is part of the standard tool-belt: where possible, we can already use partitioning as a solution to cut physical table sizes significantly. Both help to prepare efforts like decomposition (database usage is already optimized) and sharding (database is already partitioned along an identified data access dimension).
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
Fe(Pick feature) --> D
|
||||
D[(Database)] --> De
|
||||
De[Debloating] --> Dc
|
||||
Dc[Decomposition] --> P
|
||||
P[Partitioning] --> S
|
||||
S[Sharding] --> R
|
||||
P --> M
|
||||
M[Microservices] --> R
|
||||
R{Repeat?} --> Fe
|
||||
style De fill:#fca326
|
||||
style P fill:#fc6d26
|
||||
```
|
||||
|
||||
## Motivation: GitLab.com stability and performance
|
||||
|
||||
Large tables on GitLab.com are a major problem - for both operations and development. They cause a variety of problems:
|
||||
|
||||
1. **Query timings** and hence overall application performance suffers
|
||||
1. **Table maintenance** becomes much more costly. Vacuum activity has become a significant concern on GitLab.com - with large tables only seeing infrequent (once per day) processing and vacuum runs taking many hours to complete. This has various negative consequences and a very large table has potential to impact seemingly unrelated parts of the database and hence overall application performance suffers.
|
||||
1. **Data migrations** on large tables are significantly more complex to implement and incur development overhead. They have potential to cause stability problems on GitLab.com and take a long time to execute on large datasets.
|
||||
1. **Indexes size** is significant. This directly impacts performance as smaller parts of the index are kept in memory and also makes the indexes harder to maintain (think repacking).
|
||||
1. **Index creation times** go up significantly - in 2021, we see B-Tree creation take up to 6 hours for a single B-Tree index. This impacts our ability to deploy frequently and leads to vacuum-related problems (delayed cleanup).
|
||||
1. We tend to add **many indexes** to mitigate, but this eventually causes significant overhead, can confuse the query planner and a large number of indexes is a smell of a design problem.
|
||||
|
||||
## Examples
|
||||
|
||||
Most prominently, the `ci_builds` table is 1.5 TB in size as of June 2021 and has 31 indexes associated with it which sum up to 1 TB in size. The overall on-disk size for this table is 2.5 TB. Currently, this grows at 300 GB per month. By the end of the year, this is thought to be close to 5 TB if we don't take measures against.
|
||||
|
||||
The following examples show that very large tables often constitute the root cause of incidents on GitLab.com.
|
||||
|
||||
1. Infrequent and long running vacuum activity has led to [repeated degradation of query performance for CI queuing](https://gitlab.com/gitlab-com/gl-infra/production/-/issues?label_name%5B%5D=Service%3A%3ACI+Runners&label_name%5B%5D=incident&scope=all&search=shared_runner_queues&state=all)
|
||||
1. On large tables like `ci_builds`, index creation time varies between 1.5 to 6 hours during busy times. This process blocks deployments as migrations are being run synchronously - reducing our ability to deploy frequently.
|
||||
1. Creating a large index can lead to a burst of activity on the database primary:
|
||||
1. on `merge_request_diff_commits` table: caused [high network saturation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4823),
|
||||
1. regular reindexing activity on the weekend: causes [growing WAL queue](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4767) (impacts recovery objectives),
|
||||
1. `notes` table: Re-creating a GIN trigram index for maintenance reasons has become nearly unfeasible and had to be [aborted after 12 hours upon first try](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4633) as it was blocking other vacuum operation.
|
||||
|
||||
## Problematic tables on GitLab.com
|
||||
|
||||
This shows the TOP30 tables by their total size (includes index sizes) as of mid June 2021 on GitLab.com. `table_size, index_size` is the on-disk size of the actual data and associated indexes, respectively. `percentage_of_total_database_size` displays the ratio of total table size to database size.
|
||||
|
||||
As we can see, there are currently very large tables greater than 1 TB in size, which also tend to have very large indexes.
|
||||
|
||||
The other observation here is that there are also tables with a large number of indexes and total index size can be significantly larger than the data stored. For example, `deployments` is 30 GB in size plus additional 123 GB of index data spread across 24 indexes.
|
||||
|
||||
<!--
|
||||
select tablename,
|
||||
pg_size_pretty(pg_total_relation_size(t.schemaname || '.' || t.tablename)) as total_size,
|
||||
pg_size_pretty(pg_relation_size(t.schemaname || '.' || t.tablename)) as table_size,
|
||||
pg_size_pretty(pg_indexes_size(t.schemaname || '.' || t.tablename)) as index_size,
|
||||
count(*) as index_count,
|
||||
round(pg_total_relation_size(t.schemaname || '.' || t.tablename) / pg_database_size('gitlabhq_production')::numeric * 100, 1) as percentage_of_total_database_size
|
||||
from pg_indexes i
|
||||
join pg_tables t USING (tablename)
|
||||
group by 1,
|
||||
2,
|
||||
3,
|
||||
t.schemaname,
|
||||
t.tablename
|
||||
order by pg_total_relation_size(t.schemaname || '.' || t.tablename) desc
|
||||
limit 30;
|
||||
-->
|
||||
|
||||
| Table | Total size | Table size | Index size | Index count | Percentage of total database size |
|
||||
|------------------------------|------------|------------|------------|-------------|-----------------------------------|
|
||||
| `ci_builds` | 2975 GB | 1551 GB | 941 GB | 30 | 22.7 |
|
||||
| `merge_request_diff_commits` | 1890 GB | 1454 GB | 414 GB | 2 | 14.4 |
|
||||
| `ci_build_trace_sections` | 1123 GB | 542 GB | 581 GB | 3 | 8.6 |
|
||||
| `notes` | 748 GB | 390 GB | 332 GB | 13 | 5.7 |
|
||||
| `merge_request_diff_files` | 575 GB | 481 GB | 88 GB | 1 | 4.4 |
|
||||
| `events` | 441 GB | 95 GB | 346 GB | 12 | 3.4 |
|
||||
| `ci_job_artifacts` | 397 GB | 187 GB | 210 GB | 10 | 3.0 |
|
||||
| `ci_pipelines` | 266 GB | 66 GB | 200 GB | 23 | 2.0 |
|
||||
| `taggings` | 238 GB | 60 GB | 179 GB | 5 | 1.8 |
|
||||
| `ci_builds_metadata` | 237 GB | 88 GB | 149 GB | 5 | 1.8 |
|
||||
| `issues` | 219 GB | 47 GB | 150 GB | 28 | 1.7 |
|
||||
| `web_hook_logs_202103` | 186 GB | 122 GB | 8416 MB | 3 | 1.4 |
|
||||
| `ci_stages` | 182 GB | 58 GB | 124 GB | 6 | 1.4 |
|
||||
| `web_hook_logs_202105` | 180 GB | 115 GB | 7868 MB | 3 | 1.4 |
|
||||
| `merge_requests` | 176 GB | 44 GB | 125 GB | 36 | 1.3 |
|
||||
| `web_hook_logs_202104` | 176 GB | 115 GB | 7472 MB | 3 | 1.3 |
|
||||
| `web_hook_logs_202101` | 169 GB | 112 GB | 7231 MB | 3 | 1.3 |
|
||||
| `web_hook_logs_202102` | 167 GB | 111 GB | 7106 MB | 3 | 1.3 |
|
||||
| `sent_notifications` | 166 GB | 88 GB | 79 GB | 3 | 1.3 |
|
||||
| `web_hook_logs_202011` | 163 GB | 113 GB | 7125 MB | 3 | 1.2 |
|
||||
| `push_event_payloads` | 162 GB | 114 GB | 48 GB | 1 | 1.2 |
|
||||
| `web_hook_logs_202012` | 159 GB | 106 GB | 6771 MB | 3 | 1.2 |
|
||||
| `web_hook_logs_202106` | 156 GB | 101 GB | 6752 MB | 3 | 1.2 |
|
||||
| `deployments` | 155 GB | 30 GB | 125 GB | 24 | 1.2 |
|
||||
| `web_hook_logs_202010` | 136 GB | 98 GB | 6116 MB | 3 | 1.0 |
|
||||
| `web_hook_logs_202009` | 114 GB | 82 GB | 5168 MB | 3 | 0.9 |
|
||||
| `security_findings` | 109 GB | 21 GB | 88 GB | 8 | 0.8 |
|
||||
| `web_hook_logs_202008` | 92 GB | 66 GB | 3983 MB | 3 | 0.7 |
|
||||
| `resource_label_events` | 66 GB | 47 GB | 19 GB | 6 | 0.5 |
|
||||
| `merge_request_diffs` | 63 GB | 39 GB | 22 GB | 5 | 0.5 |
|
||||
|
||||
## Target: All physical tables on GitLab.com are < 100 GB including indexes
|
||||
|
||||
NOTE:
|
||||
In PostgreSQL context, a **physical table** is either a regular table or a partition of a partitioned table.
|
||||
|
||||
To maintain and improve operational stability and lessen development burden, we target a **table size less than 100 GB for a physical table on GitLab.com** (including its indexes). This has numerous benefits:
|
||||
|
||||
1. Improved query performance and more stable query plans
|
||||
1. Significantly reduce vacuum run times and increase frequency of vacuum runs to maintain a healthy state - reducing overhead on the database primary
|
||||
1. Index creation times are significantly faster (significantly less data to read per index)
|
||||
1. Indexes are smaller, can be maintained more efficiently and fit better into memory
|
||||
1. Data migrations are easier to reason about, take less time to implement and execute
|
||||
|
||||
This target is *pragmatic*: We understand table sizes depend on feature usage, code changes and other factors - which all change over time. We may not always find solutions where we can tightly limit the size of physical tables once and for all. That is acceptable though and we primarily aim to keep the situation on GitLab.com under control. We adapt our efforts to the situation present on GitLab.com and re-evaluate frequently.
|
||||
|
||||
While there are changes we can make that lead to a constant maximum physical table size over time, this doesn't need to be the case necessarily. Consider for example hash partitioning, which breaks a table down into a static number of partitions. With data growth over time, individual partitions also grow in size and may eventually reach the threshold size again. We strive to get constant table sizes, but it is acceptable to ship easier solutions that don't have this characteristic but improve the situation for a considerable amount of time.
|
||||
|
||||
As such, the target size of a physical table after refactoring depends on the situation and there is no hard rule for it. We suggest to consider historic data growth and forecast when physical tables reach the threshold of 100 GB again. This allows us to understand how long a particular solution is expected to last until the model has to be revisited.
|
||||
|
||||
## Solutions
|
||||
|
||||
There is no standard solution to reduce table sizes - there are many!
|
||||
|
||||
1. **Retention**: Delete unnecessary data, for example expire old and unneeded records.
|
||||
1. **Remove STI**: We still use [single-table inheritance](../../../development/database/single_table_inheritance.md) in a few places, which is considered an anti-pattern. Redesigning this, we can split data into multiple tables.
|
||||
1. **Index optimization**: Drop unnecessary indexes and consolidate overlapping indexes if possible.
|
||||
1. **Optimize data types**: Review data type decisions and optimize data types where possible (example: use integer instead of text for an enum column)
|
||||
1. **Partitioning**: Apply a partitioning scheme if there is a common access dimension.
|
||||
1. **Normalization**: Review relational modeling and apply normalization techniques to remove duplicate data
|
||||
1. **Vertical table splits**: Review column usage and split table vertically.
|
||||
1. **Externalize**: Move large data types out of the database entirely. For example, JSON documents, especially when not used for filtering, may be better stored outside the database, for example, in object storage.
|
||||
|
||||
NOTE:
|
||||
While we're targeting to limit physical table sizes, we consider retaining or improving performance a goal, too.
|
||||
|
||||
For solutions like normalization, this is a trade-off: Denormalized models can speed up queries when used appropriately, at the expense of table size. When normalizing models, splitting tables or externalizing data, we aim to understand the impact on performance and strive to find a solution to reduce table sizes that doesn't impact performance significantly.
|
||||
|
||||
### Example efforts
|
||||
|
||||
A few examples can be found below, many more are organized under the epic [Database efficiency](https://gitlab.com/groups/gitlab-org/-/epics/5585).
|
||||
|
||||
1. [Reduce number of indexes on `ci_builds`](https://gitlab.com/groups/gitlab-org/-/epics/6203)
|
||||
1. [Normalize and de-duplicate committer and author details in `merge_request_diff_commits`](https://gitlab.com/gitlab-org/gitlab/-/issues/331823)
|
||||
1. [Retention strategy for `ci_build_trace_sections`](https://gitlab.com/gitlab-org/gitlab/-/issues/32565#note_603138100)
|
||||
1. [Implement worker that hard-deletes old CI jobs metadata](https://gitlab.com/gitlab-org/gitlab/-/issues/215646)
|
||||
1. [`merge_request_diff_files` violates < 100 GB target](https://gitlab.com/groups/gitlab-org/-/epics/6215) (epic)
|
||||
|
||||
## Goal
|
||||
|
||||
The [epic for `~group::database`](https://gitlab.com/groups/gitlab-org/-/epics/6211) drives decision making to establish and communicate the target and to identify and propose necessary changes to reach it. Those changes should primarily be driven by the respective stage group owning the data (and the feature using it), with `~group::database` to support.
|
||||
|
||||
## Who
|
||||
|
||||
<!-- vale gitlab.Spelling = NO -->
|
||||
|
||||
Identifying solutions for offending tables is driven by the [GitLab Database Team](https://handbook.gitlab.com/handbook/engineering/infrastructure/core-platform/data_stores/database/) and respective stage groups.
|
||||
|
||||
| Role | Who |
|
||||
|--------------------|-----|
|
||||
| Author | Andreas Brandl |
|
||||
| Engineering Leader | Craig Gomes |
|
||||
|
||||
<!-- vale gitlab.Spelling = YES -->
|
||||
<!-- This redirect file can be deleted after <2025-07-19>. -->
|
||||
<!-- Redirects that point to other docs in the same project expire in three months. -->
|
||||
<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. -->
|
||||
<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
|
||||
|
|
|
|||
|
|
@ -44,7 +44,7 @@ the list of users testing this feature, do this thing. If you find a bug,
|
|||
|
||||
When the feature is ready for production, remove:
|
||||
|
||||
- The text in parentheses.
|
||||
- The **Status** from the availability details.
|
||||
- Any language about the feature not being ready for production in the body
|
||||
description.
|
||||
- The feature flag information if available.
|
||||
|
|
|
|||
|
|
@ -11,13 +11,13 @@ DETAILS:
|
|||
**Offering:** GitLab.com, Self-managed, GitLab Dedicated
|
||||
|
||||
To manage your infrastructure with GitLab, you can use the integration with
|
||||
Terraform to define resources that you can version, reuse, and share:
|
||||
Terraform or OpenTofu to define resources that you can version, reuse, and share:
|
||||
|
||||
- Manage low-level components like compute, storage, and networking resources.
|
||||
- Manage high-level components like DNS entries and SaaS features.
|
||||
- Incorporate GitOps deployments and Infrastructure-as-Code (IaC) workflows.
|
||||
- Use GitLab as a Terraform state storage.
|
||||
- Store and use Terraform modules to simplify common and complex infrastructure patterns.
|
||||
- Incorporate GitOps deployments and Infrastructure-as-Code (IaC) workflows.
|
||||
|
||||
<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [a video overview](https://www.youtube.com/watch?v=iGXjUrkkzDI) of the features GitLab provides with the integration with Terraform.
|
||||
|
||||
|
|
@ -34,7 +34,29 @@ For simplicity, the GitLab documentation refers primarily to Terraform.
|
|||
However, differences between the Terraform and OpenTofu integration
|
||||
are documented.
|
||||
|
||||
## Integrate your project with Terraform
|
||||
## Quickstart an OpenTofu project in pipelines
|
||||
|
||||
OpenTofu can integrate with all Terraform-specific GitLab features with the
|
||||
GitLab OpenTofu CI/CD component.
|
||||
|
||||
You can add a *validate*, *plan*, and *apply* workflow to your pipeline by including the component:
|
||||
|
||||
```yaml
|
||||
include:
|
||||
- component: gitlab.com/components/opentofu/validate-plan-apply@<VERSION>
|
||||
inputs:
|
||||
version: <VERSION>
|
||||
opentofu_version: <OPENTOFU_VERSION>
|
||||
root_dir: terraform/
|
||||
state_name: production
|
||||
|
||||
stages: [validate, build, deploy]
|
||||
```
|
||||
|
||||
See the [OpenTofu CI/CD component README](https://gitlab.com/components/opentofu) for detailed information
|
||||
about the usage of the component and all available templates and inputs.
|
||||
|
||||
## Quickstart a Terraform project in pipelines
|
||||
|
||||
WARNING:
|
||||
The Terraform CI/CD templates are deprecated and will be removed in GitLab 18.0.
|
||||
|
|
@ -124,31 +146,8 @@ project.
|
|||
|
||||
For GitLab-curated template recipes, see [Terraform template recipes](terraform_template_recipes.md).
|
||||
|
||||
## Integrate your project with OpenTofu
|
||||
|
||||
OpenTofu can integrate with all Terraform-specific GitLab features with the
|
||||
GitLab OpenTofu CI/CD component.
|
||||
|
||||
You can add a *validate*, *plan*, and *apply* workflow to your pipeline by including the component:
|
||||
|
||||
```yaml
|
||||
include:
|
||||
- component: gitlab.com/components/opentofu/validate-plan-apply@<VERSION>
|
||||
inputs:
|
||||
version: <VERSION>
|
||||
opentofu_version: <OPENTOFU_VERSION>
|
||||
root_dir: terraform/
|
||||
state_name: production
|
||||
|
||||
stages: [validate, build, deploy]
|
||||
```
|
||||
|
||||
See the [OpenTofu CI/CD component README](https://gitlab.com/components/opentofu) for detailed information
|
||||
about the usage of the component and all available templates and inputs.
|
||||
|
||||
## Related topics
|
||||
|
||||
- View [the images that contain the `gitlab-terraform` shell script](https://gitlab.com/gitlab-org/terraform-images).
|
||||
- Use GitLab as a [Terraform Module Registry](../../packages/terraform_module_registry/index.md).
|
||||
- To store state files in local storage or in a remote store, use the [GitLab-managed Terraform state](terraform_state.md).
|
||||
- To collaborate on Terraform code changes and Infrastructure-as-Code workflows, use the
|
||||
|
|
@ -159,3 +158,4 @@ about the usage of the component and all available templates and inputs.
|
|||
- [Create a new cluster on Amazon Elastic Kubernetes Service (EKS)](../clusters/connect/new_eks_cluster.md).
|
||||
- [Create a new cluster on Google Kubernetes Engine (GKE)](../clusters/connect/new_gke_cluster.md).
|
||||
- [Troubleshoot](troubleshooting.md) issues with GitLab and Terraform.
|
||||
- View [the images that contain the `gitlab-terraform` shell script](https://gitlab.com/gitlab-org/terraform-images).
|
||||
|
|
|
|||
|
|
@ -30,12 +30,12 @@ This setting ensures artifacts are accessible only to GitLab administrators and
|
|||
|
||||
## Configure Terraform report artifacts
|
||||
|
||||
GitLab [integrates with Terraform](index.md#integrate-your-project-with-terraform) and [OpenTofu](index.md#integrate-your-project-with-opentofu)
|
||||
GitLab [integrates with Terraform](index.md#quickstart-a-terraform-project-in-pipelines) and [OpenTofu](index.md#quickstart-an-opentofu-project-in-pipelines)
|
||||
through CI/CD templates and components that use GitLab-managed Terraform state and display Terraform changes on merge requests.
|
||||
|
||||
### Automatically configure Terraform report artifacts
|
||||
|
||||
You should use either the [Terraform CI/CD templates](index.md#integrate-your-project-with-terraform)
|
||||
You should use either the [Terraform CI/CD templates](index.md#quickstart-a-terraform-project-in-pipelines)
|
||||
or the [OpenTofu CI/CD component](https://gitlab.com/components/opentofu), which automatically configure the Terraform report artifacts
|
||||
in the `plan` jobs.
|
||||
|
||||
|
|
|
|||
|
|
@ -47,9 +47,6 @@ For self-managed GitLab, before you can use GitLab for your Terraform state file
|
|||
|
||||
## Initialize a Terraform state as a backend by using GitLab CI/CD
|
||||
|
||||
After you execute the `terraform init` command, you can use GitLab CI/CD
|
||||
to run `terraform` commands.
|
||||
|
||||
Prerequisites:
|
||||
|
||||
- To lock, unlock, and write to the state by using `terraform apply`, you must have at least the Maintainer role.
|
||||
|
|
@ -77,34 +74,30 @@ To configure GitLab CI/CD as a backend:
|
|||
```
|
||||
|
||||
1. In the root directory of your project repository, create a `.gitlab-ci.yml` file. Use the
|
||||
[`Terraform.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml)
|
||||
template to populate it.
|
||||
[OpenTofu CI/CD component](https://gitlab.com/components/opentofu) to from your `.gitlab-ci.yml`. Follow the [Terraform template recipes](terraform_template_recipes.md) documentation if you prefer using Terraform instead of OpenTofu.
|
||||
1. Push your project to GitLab. This action triggers a pipeline, which
|
||||
runs the `gitlab-terraform init`, `gitlab-terraform validate`, and
|
||||
`gitlab-terraform plan` commands.
|
||||
1. Trigger the manual `deploy` job from the previous pipeline, which runs `gitlab-terraform apply` command, to provision the defined infrastructure.
|
||||
runs the `gitlab-tofu init`, `gitlab-tofu validate`, and
|
||||
`gitlab-tofu plan` commands.
|
||||
1. Trigger the manual `deploy` job from the previous pipeline, which runs `gitlab-tofu apply` command, to provision the defined infrastructure.
|
||||
|
||||
The output from the above `terraform` commands should be viewable in the job logs.
|
||||
The output from the above commands should be viewable in the job logs.
|
||||
|
||||
The `gitlab-terraform` CLI is a wrapper around the `terraform` CLI. For more information,
|
||||
The `gitlab-tofu`/`gitlab-terraform` CLI is a wrapper around the `tofu`/`terraform` CLI, respectively. For more information,
|
||||
see [GitLab Terraform helpers](gitlab_terraform_helpers.md),
|
||||
or [view the source code of `gitlab-terraform`](https://gitlab.com/gitlab-org/terraform-images/-/blob/master/src/bin/gitlab-terraform.sh).
|
||||
|
||||
If you prefer to call the `terraform` commands explicitly, you can override
|
||||
the template, and instead, use it as reference for what you can achieve.
|
||||
|
||||
### Customizing your Terraform environment variables
|
||||
|
||||
When you use the `Terraform.gitlab-ci.yml` template, you can use [Terraform HTTP configuration variables](https://www.terraform.io/language/settings/backends/http#configuration-variables) when you define your CI/CD jobs.
|
||||
You can use [Terraform HTTP configuration variables](https://www.terraform.io/language/settings/backends/http#configuration-variables) when you define your CI/CD jobs.
|
||||
|
||||
To customize your `terraform init` and override the Terraform configuration,
|
||||
use environment variables instead of the `terraform init -backend-config=...` approach.
|
||||
To customize your `init` and override the Terraform configuration,
|
||||
use environment variables instead of the `init -backend-config=...` approach.
|
||||
When you use `-backend-config`, the configuration is:
|
||||
|
||||
- Cached in the output of the `terraform plan` command.
|
||||
- Usually passed forward to the `terraform apply` command.
|
||||
- Cached in the output of the `plan` command.
|
||||
- Usually passed forward to the `apply` command.
|
||||
|
||||
This configuration can lead to problems like [being unable to lock Terraform state files in CI jobs](troubleshooting.md#cant-lock-terraform-state-files-in-ci-jobs-for-terraform-apply-with-a-previous-jobs-plan).
|
||||
This configuration can lead to problems like [being unable to lock the state files in CI jobs](troubleshooting.md#cant-lock-terraform-state-files-in-ci-jobs-for-terraform-apply-with-a-previous-jobs-plan).
|
||||
|
||||
## Access the state from your local machine
|
||||
|
||||
|
|
@ -136,6 +129,8 @@ You should use a local terminal to run the commands needed for migrating to GitL
|
|||
|
||||
The following example demonstrates how to change the state name. The same workflow is needed to migrate to GitLab-managed Terraform state from a different state storage backend.
|
||||
|
||||
You should run these commands [on your local machine](#access-the-state-from-your-local-machine).
|
||||
|
||||
### Set up the initial backend
|
||||
|
||||
```shell
|
||||
|
|
|
|||
|
|
@ -248,10 +248,45 @@ From https://gitlab.com/gitlab-org/gitlab-foss.git
|
|||
...
|
||||
```
|
||||
|
||||
And to check out a particular merge request:
|
||||
To check out a particular merge request:
|
||||
|
||||
```shell
|
||||
git checkout origin/merge-requests/1
|
||||
```
|
||||
|
||||
All the above can be done with the [`git-mr`](https://gitlab.com/glensc/git-mr) script.
|
||||
|
||||
## Error: "source branch `<branch_name>` does not exist." when the branch exists
|
||||
|
||||
This error can happen if the GitLab cache does not reflect the actual state of the
|
||||
Git repository.
|
||||
|
||||
Prerequisite:
|
||||
|
||||
- You must be an administrator.
|
||||
|
||||
To force a cache update:
|
||||
|
||||
1. Open the GitLab Rails console with this command:
|
||||
|
||||
```shell
|
||||
sudo gitlab-rails console
|
||||
```
|
||||
|
||||
1. In the Rails console, run this script:
|
||||
|
||||
```ruby
|
||||
# Get the project
|
||||
project = Project.find_by_full_path('affected/project/path')
|
||||
|
||||
# Check if the affected branch exists in cache (it may return `false`)
|
||||
project.repository.branch_names.include?('affected_branch_name')
|
||||
|
||||
# Expire the branches cache
|
||||
project.repository.expire_branches_cache
|
||||
|
||||
# Check again if the affected branch exists in cache (this time it should return `true`)
|
||||
project.repository.branch_names.include?('affected_branch_name')
|
||||
```
|
||||
|
||||
1. Reload the merge request.
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
stage: Create
|
||||
group: Remote Development
|
||||
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
|
||||
description: "Configure the GitLab agent for remote development as part of setting up a workspace."
|
||||
description: "Configure the GitLab agent for workspaces."
|
||||
---
|
||||
|
||||
# GitLab agent configuration
|
||||
|
|
@ -15,7 +15,7 @@ DETAILS:
|
|||
> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/391543) in GitLab 16.0.
|
||||
> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136744) in GitLab 16.7. Feature flag `remote_development_feature_flag` removed.
|
||||
|
||||
As part of [setting up workspace infrastructure](configuration.md#set-up-workspace-infrastructure), you must configure a GitLab agent to support workspaces. This guide assumes that a GitLab Agent is already installed in the Kubernetes cluster.
|
||||
When you [set up workspace infrastructure](configuration.md#set-up-workspace-infrastructure), you must configure a GitLab agent to support workspaces. This guide assumes that a GitLab agent is already installed in the Kubernetes cluster.
|
||||
|
||||
Prerequisites:
|
||||
|
||||
|
|
@ -23,7 +23,7 @@ Prerequisites:
|
|||
- The agent must be **allowed** in a group for the purpose of creating workspaces. During workspace creation, users can select allowed agents that are associated with any parent group of the workspace project.
|
||||
- The workspace creator must have the Developer role to the project of the agent.
|
||||
|
||||
## Agent authorization in a Group for creating workspaces
|
||||
## Agent authorization in a group for creating workspaces
|
||||
|
||||
> - New authorization strategy [introduced](https://gitlab.com/groups/gitlab-org/-/epics/14025) in GitLab 17.2.
|
||||
|
||||
|
|
@ -41,7 +41,7 @@ graph TD;
|
|||
topGroup[Top Group, allowed Agent 1]
|
||||
subgroup1[Subgroup 1, allowed Agent 2]
|
||||
subgroup2[Subgroup 2, allowed Agent 3]
|
||||
wp(Woekspace Project, Agent 1, 2 & 3 all available)
|
||||
wp(Workspace Project, Agent 1, 2 & 3 all available)
|
||||
|
||||
topGroup --> subgroup1
|
||||
subgroup1 --> subgroup2
|
||||
|
|
@ -86,7 +86,8 @@ To remove an allowed cluster agent from a group:
|
|||
|
||||
The status of the selected agent is updated to **Blocked**, and the agent is removed from the **Allowed agents** tab.
|
||||
|
||||
Removing an allowed cluster agent from a group does not immediately stop running workspaces using this agent. Running workspaces will stop after the configured termation period, or being manually stopped.
|
||||
Removing an allowed cluster agent from a group does not immediately stop running workspaces using this agent.
|
||||
Running workspaces stop when they are automatically terminated or manually stopped.
|
||||
|
||||
## Legacy agent authorization strategy
|
||||
|
||||
|
|
|
|||
|
|
@ -2,18 +2,10 @@
|
|||
stage: Create
|
||||
group: Remote Development
|
||||
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
|
||||
description: "Setup an existing GitLab Agent for Workspaces"
|
||||
description: "Set up the GitLab agent for workspaces."
|
||||
---
|
||||
|
||||
# Tutorial: Set up the GitLab agent for the group
|
||||
|
||||
DETAILS:
|
||||
**Tier:** Premium, Ultimate
|
||||
**Offering:** GitLab.com, Self-managed, GitLab Dedicated
|
||||
|
||||
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/112397) in GitLab 15.11 [with a flag](../../administration/feature_flags.md) named `remote_development_feature_flag`. Disabled by default.
|
||||
> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/391543) in GitLab 16.0.
|
||||
> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/136744) in GitLab 16.7. Feature flag `remote_development_feature_flag` removed.
|
||||
# Tutorial: Set up the GitLab agent
|
||||
|
||||
In this tutorial, you'll learn how to set up a cluster agent for remote development workspace.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue