263 lines
13 KiB
Markdown
263 lines
13 KiB
Markdown
---
|
|
stage: Verify
|
|
group: Pipeline Authoring
|
|
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
|
|
title: Troubleshooting CI/CD variables
|
|
---
|
|
|
|
## List all variables
|
|
|
|
You can list all variables available to a script with the `export` command
|
|
in Bash or `dir env:` in PowerShell. This exposes the values of **all** available
|
|
variables, which can be a [security risk](_index.md#cicd-variable-security).
|
|
[Masked variables](_index.md#mask-a-cicd-variable) display as `[MASKED]`.
|
|
|
|
For example, with Bash:
|
|
|
|
```yaml
|
|
job_name:
|
|
script:
|
|
- export
|
|
```
|
|
|
|
Example job log output (truncated):
|
|
|
|
```shell
|
|
export CI_JOB_ID="50"
|
|
export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a"
|
|
export CI_COMMIT_SHORT_SHA="1ecfd275"
|
|
export CI_COMMIT_REF_NAME="main"
|
|
export CI_REPOSITORY_URL="https://gitlab-ci-token:[MASKED]@example.com/gitlab-org/gitlab.git"
|
|
export CI_COMMIT_TAG="1.0.0"
|
|
export CI_JOB_NAME="spec:other"
|
|
export CI_JOB_STAGE="test"
|
|
export CI_JOB_MANUAL="true"
|
|
export CI_JOB_TRIGGERED="true"
|
|
export CI_JOB_TOKEN="[MASKED]"
|
|
export CI_PIPELINE_ID="1000"
|
|
export CI_PIPELINE_IID="10"
|
|
export CI_PAGES_DOMAIN="gitlab.io"
|
|
export CI_PAGES_URL="https://gitlab-org.gitlab.io/gitlab"
|
|
export CI_PROJECT_ID="34"
|
|
export CI_PROJECT_DIR="/builds/gitlab-org/gitlab"
|
|
export CI_PROJECT_NAME="gitlab"
|
|
export CI_PROJECT_TITLE="GitLab"
|
|
...
|
|
```
|
|
|
|
## Enable debug logging
|
|
|
|
{{< alert type="warning" >}}
|
|
|
|
Debug logging can be a serious security risk. The output contains the content of
|
|
all variables available to the job. The output is uploaded to the
|
|
GitLab server and visible in job logs.
|
|
|
|
{{< /alert >}}
|
|
|
|
You can use debug logging to help troubleshoot problems with pipeline configuration
|
|
or job scripts. Debug logging exposes job execution details that are usually hidden
|
|
by the runner and makes job logs more verbose. It also exposes all variables and secrets
|
|
available to the job.
|
|
|
|
Before you enable debug logging, make sure only team members
|
|
can view job logs. You should also [delete job logs](../jobs/_index.md#view-jobs-in-a-pipeline)
|
|
with debug output before you make logs public again.
|
|
|
|
To enable debug logging, set the `CI_DEBUG_TRACE` variable to `true`:
|
|
|
|
```yaml
|
|
job_name:
|
|
variables:
|
|
CI_DEBUG_TRACE: "true"
|
|
```
|
|
|
|
Example output (truncated):
|
|
|
|
```plaintext
|
|
...
|
|
export CI_SERVER_TLS_CA_FILE="/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE"
|
|
if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then
|
|
echo $'\''\x1b[32;1mFetching changes...\x1b[0;m'\''
|
|
$'\''cd'\'' "/builds/gitlab-examples/ci-debug-trace"
|
|
$'\''git'\'' "config" "fetch.recurseSubmodules" "false"
|
|
$'\''rm'\'' "-f" ".git/index.lock"
|
|
$'\''git'\'' "clean" "-ffdx"
|
|
$'\''git'\'' "reset" "--hard"
|
|
$'\''git'\'' "remote" "set-url" "origin" "https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@example.com/gitlab-examples/ci-debug-trace.git"
|
|
$'\''git'\'' "fetch" "origin" "--prune" "+refs/heads/*:refs/remotes/origin/*" "+refs/tags/*:refs/tags/lds"
|
|
++ CI_BUILDS_DIR=/builds
|
|
++ export CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace
|
|
++ CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace
|
|
++ export CI_CONCURRENT_ID=87
|
|
++ CI_CONCURRENT_ID=87
|
|
++ export CI_CONCURRENT_PROJECT_ID=0
|
|
++ CI_CONCURRENT_PROJECT_ID=0
|
|
++ export CI_SERVER=yes
|
|
++ CI_SERVER=yes
|
|
++ mkdir -p /builds/gitlab-examples/ci-debug-trace.tmp
|
|
++ echo -n '-----BEGIN CERTIFICATE-----
|
|
-----END CERTIFICATE-----'
|
|
++ export CI_SERVER_TLS_CA_FILE=/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE
|
|
++ CI_SERVER_TLS_CA_FILE=/builds/gitlab-examples/ci-debug-trace.tmp/CI_SERVER_TLS_CA_FILE
|
|
++ export CI_PIPELINE_ID=52666
|
|
++ CI_PIPELINE_ID=52666
|
|
++ export CI_PIPELINE_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/pipelines/52666
|
|
++ CI_PIPELINE_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/pipelines/52666
|
|
++ export CI_JOB_ID=7046507
|
|
++ CI_JOB_ID=7046507
|
|
++ export CI_JOB_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/-/jobs/379424655
|
|
++ CI_JOB_URL=https://gitlab.com/gitlab-examples/ci-debug-trace/-/jobs/379424655
|
|
++ export CI_JOB_TOKEN=[MASKED]
|
|
++ CI_JOB_TOKEN=[MASKED]
|
|
++ export CI_REGISTRY_USER=gitlab-ci-token
|
|
++ CI_REGISTRY_USER=gitlab-ci-token
|
|
++ export CI_REGISTRY_PASSWORD=[MASKED]
|
|
++ CI_REGISTRY_PASSWORD=[MASKED]
|
|
++ export CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab.com/gitlab-examples/ci-debug-trace.git
|
|
++ CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab.com/gitlab-examples/ci-debug-trace.git
|
|
++ export CI_JOB_NAME=debug_trace
|
|
++ CI_JOB_NAME=debug_trace
|
|
++ export CI_JOB_STAGE=test
|
|
++ CI_JOB_STAGE=test
|
|
++ export CI_NODE_TOTAL=1
|
|
++ CI_NODE_TOTAL=1
|
|
++ export CI=true
|
|
++ CI=true
|
|
++ export GITLAB_CI=true
|
|
++ GITLAB_CI=true
|
|
++ export CI_SERVER_URL=https://gitlab.com:3000
|
|
++ CI_SERVER_URL=https://gitlab.com:3000
|
|
++ export CI_SERVER_HOST=gitlab.com
|
|
++ CI_SERVER_HOST=gitlab.com
|
|
++ export CI_SERVER_PORT=3000
|
|
++ CI_SERVER_PORT=3000
|
|
++ export CI_SERVER_SHELL_SSH_HOST=gitlab.com
|
|
++ CI_SERVER_SHELL_SSH_HOST=gitlab.com
|
|
++ export CI_SERVER_SHELL_SSH_PORT=22
|
|
++ CI_SERVER_SHELL_SSH_PORT=22
|
|
++ export CI_SERVER_PROTOCOL=https
|
|
++ CI_SERVER_PROTOCOL=https
|
|
++ export CI_SERVER_NAME=GitLab
|
|
++ CI_SERVER_NAME=GitLab
|
|
++ export GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,wip_limits,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal
|
|
++ GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,wip_limits,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal
|
|
++ export CI_PROJECT_ID=17893
|
|
++ CI_PROJECT_ID=17893
|
|
++ export CI_PROJECT_NAME=ci-debug-trace
|
|
++ CI_PROJECT_NAME=ci-debug-trace
|
|
...
|
|
```
|
|
|
|
### Access to debug logging
|
|
|
|
Access to debug logging is restricted to [users with at least the Developer role](../../user/permissions.md#cicd). Users with a lower role cannot see the logs when debug logging is enabled with a variable in:
|
|
|
|
- The [`.gitlab-ci.yml` file](_index.md#define-a-cicd-variable-in-the-gitlab-ciyml-file).
|
|
- The CI/CD variables set in the GitLab UI.
|
|
|
|
{{< alert type="warning" >}}
|
|
|
|
If you add `CI_DEBUG_TRACE` as a local variable to runners, debug logs generate and are visible
|
|
to all users with access to job logs. The permission levels are not checked by the runner,
|
|
so you should only use the variable in GitLab itself.
|
|
|
|
{{< /alert >}}
|
|
|
|
## `argument list too long` error
|
|
|
|
This issue occurs when the combined length of all CI/CD variables defined for a job exceeds the limit imposed by the
|
|
shell where the job executes. This includes the names and values of pre-defined and user defined variables. This limit
|
|
is typically referred to as `ARG_MAX`, and is shell and operating system dependent. This issue also occurs when the
|
|
content of a single [File-type](_index.md#use-file-type-cicd-variables) variable exceeds `ARG_MAX`.
|
|
|
|
For more information, see [issue 392406](https://gitlab.com/gitlab-org/gitlab/-/issues/392406#note_1414219596).
|
|
|
|
As a workaround you can either:
|
|
|
|
- Use [File-type](_index.md#use-file-type-cicd-variables) CI/CD variables for large environment variables where possible.
|
|
- If a single large variable is larger than `ARG_MAX`, try using [Secure Files](../secure_files/_index.md), or
|
|
bring the file to the job through some other mechanism.
|
|
|
|
## `Insufficient permissions to set pipeline variables` error for a downstream pipeline
|
|
|
|
When triggering a downstream pipeline, you might get this error unexpectedly:
|
|
|
|
```plaintext
|
|
Failed - (downstream pipeline can not be created, Insufficient permissions to set pipeline variables)
|
|
```
|
|
|
|
This error occurs when a downstream project has [restricted pipeline variables](_index.md#restrict-pipeline-variables) and the trigger job either:
|
|
|
|
- Has variables defined. For example:
|
|
|
|
```yaml
|
|
trigger-job:
|
|
variables:
|
|
VAR_FOR_DOWNSTREAM: "test"
|
|
trigger: my-group/my-project
|
|
```
|
|
|
|
- Receives variables from [default variables](../yaml/_index.md#default-variables) defined in a top-level `variables` section. For example:
|
|
|
|
```yaml
|
|
variables:
|
|
DEFAULT_VAR: "test"
|
|
|
|
trigger-job:
|
|
trigger: my-group/my-project
|
|
```
|
|
|
|
Variables passed to a downstream pipeline in a trigger job are [pipeline variables](_index.md#use-pipeline-variables),
|
|
so the workaround is to either:
|
|
|
|
- Remove the `variables` defined in the trigger job to avoid passing variables.
|
|
- [Prevent default variables from being passed to the downstream pipeline](../pipelines/downstream_pipelines.md#prevent-default-variables-from-being-passed).
|
|
|
|
## Default variable doesn't expand in job variable of the same name
|
|
|
|
You cannot use a default variable's value in a job variable of the same name. A default variable
|
|
is only made available to a job when the job does not have a variable defined with the same name.
|
|
If the job has a variable with the same name, the job's variable takes precedence
|
|
and the default variable is not available in the job.
|
|
|
|
For example, these two samples are equivalent:
|
|
|
|
- In this sample, `$MY_VAR` has no value because it's not defined anywhere:
|
|
|
|
```yaml
|
|
Job-with-variable:
|
|
variables:
|
|
MY_VAR: $MY_VAR
|
|
script: echo "Value is '$MY_VAR'"
|
|
```
|
|
|
|
- In this sample, `$MY_VAR` has no value because the default variable with the same name
|
|
is not available in the job:
|
|
|
|
```yaml
|
|
variables:
|
|
MY_VAR: "Default value"
|
|
|
|
Job-with-same-name-variable:
|
|
variables:
|
|
MY_VAR: $MY_VAR
|
|
script: echo "Value is '$MY_VAR'"
|
|
```
|
|
|
|
In both cases, the echo command outputs `Value is '$MY_VAR'`.
|
|
|
|
In general, you should use the default variable directly in a job rather than reassigning its value to a new variable.
|
|
If you need to do this, use variables with different names instead. For example:
|
|
|
|
```yaml
|
|
variables:
|
|
MY_VAR1: "Default value1"
|
|
MY_VAR2: "Default value2"
|
|
|
|
overwrite-same-name:
|
|
variables:
|
|
MY_VAR2_FROM_DEFAULTS: $MY_VAR2
|
|
script: echo "Values are '$MY_VAR1' and '$MY_VAR2_FROM_DEFAULTS'"
|
|
```
|