root
/
gitlab-ce
mirror of https://gitlab.com/free-releases/gitlab-ce.git
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity

Add latest changes from gitlab-org/gitlab@master

This commit is contained in:
GitLab Bot 2023-11-23 06:13:41 +00:00
parent 4822d600e3
commit 5dde7dc9b8
5 changed files with 369 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

293
doc/user/packages/package_registry/dependency_proxy/index.md Normal file
View File

@ -0,0 +1,293 @@
---
stage: Package
group: Package Registry
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Dependency proxy for packages **(PREMIUM ALL BETA)**
> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3610) in GitLab 16.6 [with a flag](../../../../administration/feature_flags.md) named `packages_dependency_proxy_maven`. Disabled by default.
> - This feature is in [Beta](../../../../policy/experiment-beta-support.md).
FLAG:
On self-managed GitLab, by default this feature is not available. To make it available, an administrator can [enable the feature flag](../../../../administration/feature_flags.md) named `packages_dependency_proxy_maven`.
On GitLab.com, this feature is not available.
The feature is not ready for production use.
The GitLab dependency proxy for packages is a local proxy for frequently pulled packages.
It is implemented as a pull-through cache that works at the project level.
Packages are pulled from the upstream package registry and automatically published to the
project's package registry. Subsequent identical requests are fulfilled with the project's
package registry. You can use the dependency proxy for packages to reduce unnecessary traffic
to the upstream registry.
## Enable the dependency proxy
To use the dependency proxy for packages, ensure your project is configured properly,
and that users who pull from the cache have the necessary authentication:
1. In the global configuration, if the following features are disabled, enable them:
- The [`package` feature](../../../../administration/packages/index.md#enable-or-disable-the-package-registry). Enabled by default.
- The [`dependency_proxy` feature](../../../../administration/packages/dependency_proxy.md#turn-on-the-dependency-proxy). Enabled by default.
1. In the project settings, if the [`package` feature](../../package_registry/index.md#disable-the-package-registry)
is disabled, enable it. It is enabled by default.
1. [Add an authentication method](#configure-a-client). The dependency proxy supports the same [authentication methods](../index.md#authenticate-with-the-registry) as the package registry:
- [Personal access token](../../../profile/personal_access_tokens.md)
- [Project deploy token](../../../project/deploy_tokens/index.md)
- [Group deploy token](../../../project/deploy_tokens/index.md)
- [Job token](../../../../ci/jobs/ci_job_token.md)
## Advanced caching
When possible, the dependency proxy for packages uses advanced caching to store packages in the project's package registry.
Advanced caching verifies the coherence between the project's package registry
and the upstream package registry. If the upstream registry has updated files,
the dependency proxy uses them to update the cached files.
When advanced caching is not supported, the dependency proxy falls back to the default behavior:
- If the requested file is found in the project's package registry, it is returned.
- If the file is not found, it is fetched from the upstream package registry.
Advanced caching support depends on how the upstream package registry
responds to dependency proxy requests, and on
which package format you use.
::Tabs
:::TabTitle Maven
| Package registry | Advanced caching supported? |
|------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------|
| [GitLab](../../maven_repository/index.md) | **{check-circle}** Yes |
| [Maven Central](https://mvnrepository.com/repos/central) | **{check-circle}** Yes |
| [Artifactory](https://jfrog.com/integration/maven-repository/) | **{check-circle}** Yes |
| [Sonatype Nexus](https://help.sonatype.com/repomanager3/nexus-repository-administration/formats/maven-repositories) | **{check-circle}** Yes |
| [GitHub Packages](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-apache-maven-registry) | **{dotted-circle}** No |
::EndTabs
### Permissions
When the dependency proxy pulls a file, the following occurs:
1. The dependency proxy searches for a file in the project's package registry.
This is a read operation.
1. The dependency proxy might publish a package file to the project's package registry.
This is a write operation.
Whether both steps are executed depends on user permissions.
The dependency proxy uses the [same permissions as the package registry](../index.md#package-registry-visibility-permissions).
| Project visibility | Minimum [role](../../../../user/permissions.md#roles) | Can read package files? | Can write package files? | Behavior |
|--------------------|-------------------------------------------------------|-------------------------|--------------------------|----------|
| Public | Anonymous | **{dotted-circle}** No | **{dotted-circle}** No | Request rejected. |
| Public | Guest | **{check-circle}** Yes | **{dotted-circle}** No | Package file returned from either the cache or the remote registry. |
| Public | Developer | **{check-circle}** Yes | **{check-circle}** Yes | Package file returned from either the cache or the remote registry. The file is published to the cache. |
| Internal | Anonymous | **{dotted-circle}** No | **{dotted-circle}** No | Request rejected |
| Internal | Guest | **{check-circle}** Yes | **{dotted-circle}** No | Package file returned from either the cache or the remote registry. |
| Internal | Developer | **{check-circle}** Yes | **{check-circle}** Yes | Package file returned from either the cache or the remote registry. The file is published to the cache. |
| Private | Anonymous | **{dotted-circle}** No | **{dotted-circle}** No | Request rejected |
| Private | Reporter | **{check-circle}** Yes | **{dotted-circle}** No | Package file returned from either the cache or the remote registry. |
| Internal | Developer | **{check-circle}** Yes | **{check-circle}** Yes | Package file returned from either the cache or the remote registry. The file is published to the cache. |
At a minimum, any user who can use the dependency proxy can also use the project's package registry.
To ensure the cache is properly filled over time, you should make sure a user with at least the Developer role pulls packages with the dependency proxy.
## Configure a client
Configuring a client for the dependency proxy is similar to configuring a client for the [package registry](../supported_functionality.md#pulling-packages).
### For Maven packages
For Maven packages, [all clients supported by the package registry](../../maven_repository/index.md) are supported by the dependency proxy:
- `mvn`
- `gradle`
- `sbt`
For authentication, the Maven dependency proxy access all authentication methods accepted by the [Maven package registry](../../maven_repository/index.md#edit-the-client-configuration).
You should use the [Basic HTTP authentication](../../maven_repository/index.md#basic-http-authentication) method as it is less complex.
To configure the client:
1. Follow the instructions in [Basic HTTP authentication](../../maven_repository/index.md#basic-http-authentication).
Make sure you use the endpoint URL `https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven`.
1. Complete the configuration for your client:
::Tabs
:::TabTitle mvn
In the `pom.xml` file add a `repository` element:
```xml
<repositories>
<repository>
<id>gitlab-maven</id>
<url>https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven</url>
</repository>
</repositories>
```
Where:
- `<project_id>` is the ID of the project to be used as a dependency proxy.
- `<id>` contains the name of the `<server>` used in the [authentication configuration](../../maven_repository/index.md#basic-http-authentication).
By default, Maven Central is checked first through the [Super POM](https://maven.apache.org/guides/introduction/introduction-to-the-pom.html#Super_POM).
However, you might want to force `mvn` to check the GitLab endpoint first. To do this, follow the instructions from the [request forward](../../maven_repository/index.md#additional-configuration-for-mvn).
:::TabTitle gradle
Add a `repositories` section to your [`build.gradle`](https://docs.gradle.org/current/userguide/tutorial_using_tasks.html) file.
- In Groovy DSL:
```groovy
repositories {
maven {
url "https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven"
name "GitLab"
credentials(PasswordCredentials) {
username = 'REPLACE_WITH_NAME'
password = gitLabPrivateToken
}
authentication {
basic(BasicAuthentication)
}
}
}
```
- In Kotlin DSL:
```kotlin
repositories {
maven {
url = uri("https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven")
name = "GitLab"
credentials(BasicAuthentication::class) {
username = "REPLACE_WITH_NAME"
password = findProperty("gitLabPrivateToken") as String?
}
authentication {
create("basic", BasicAuthentication::class)
}
}
}
```
In this example:
- `<project_id>` is the ID of the project to be used as a dependency proxy.
- `REPLACE_WITH_NAME` is explained in the [Basic HTTP authentication](../../maven_repository/index.md#basic-http-authentication) section.
:::TabTitle sbt
In your [`build.sbt`](https://www.scala-sbt.org/1.x/docs/Directories.html#sbt+build+definition+files), add the following lines:
```scala
resolvers += ("gitlab" at "https://gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven")
credentials += Credentials("GitLab Packages Registry", "<host>", "<name>", "<token>")
```
In this example:
- `<project_id>` is the ID of the project to be used as a dependency proxy.
- `<host>` is the host present in the `<endpoint url>` without the protocol scheme or the port. Example: `gitlab.example.com`.
- `<name>` and `<token>` are explained in the [Basic HTTP authentication](../../maven_repository/index.md#basic-http-authentication) section.
::EndTabs
## Configure the remote registry
The dependency proxy must be configured with:
- The URL of the remote package registry.
- Optional. The required credentials.
To set those parameters:
1. Use the related [GraphQL mutation](../../../../api/graphql/reference/index.md#mutationupdatedependencyproxypackagessettings).
A frontend to display and manage these settings is proposed by [issue 410726](https://gitlab.com/gitlab-org/gitlab/-/issues/410726).
1. Complete the configuration for your package format:
::Tabs
:::TabTitle Maven
Any Maven package registry can be connected to the dependency proxy. You can
authorize the connection with the Maven package registry username and password.
To set or update the remote Maven package registry, update the following fields on the settings object:
- `mavenExternalRegistryUrl` - The URL of the remote registry.
- `mavenExternalRegistryUsername` - Optional. The username to use with the remote registry.
- `mavenExternalRegistryPassword` - Optional. The password to use with the remote registry.
Example of the GraphQL mutation:
```graphql
mutation {
updateDependencyProxyPackagesSettings(input: { projectPath : <project full path>, enabled: true, mavenExternalRegistryUrl: <remote registry url>, mavenExternalRegistryUsername: <username>, mavenExternalRegistryPassword: <password> }) {
dependencyProxyPackagesSetting {
enabled
mavenExternalRegistryUrl
mavenExternalRegistryUsername
}
errors
}
}
```
::EndTabs
## Troubleshooting
### Manual file pull errors
You can pull files manually with cURL.
However, you might encounter one of the following responses:
- `404 Not Found` - The dependency proxy setting object was not found because it doesn't exist, or because the [requirements](#enable-the-dependency-proxy) were not fulfilled.
- `401 Unauthorized` - The user was properly authenticated but did not have the proper permissions to access the dependency proxy object.
- `403 Forbidden` - There was an issue with the [GitLab license level](#enable-the-dependency-proxy).
- `502 Bad Gateway` - The remote package registry could not fulfill the file request. Verify the [dependency proxy settings](#configure-the-remote-registry).
- `504 Gateway Timeout` - The remote package registry timed out. Verify the [dependency proxy settings](#configure-the-remote-registry).
::Tabs
:::TabTitle Maven
```shell
curl --verbose "https://<username>:<personal access token>@gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven/<group id and artifact id>/<version>/<file_name>"
```
- `<username>` and `<personal access token>` are the credentials to access the dependency proxy of the GitLab instance.
- `<project_id>` is the project ID.
- `<group id and artifact id>` are the [Maven package group ID and artifact ID](https://maven.apache.org/pom.html#Maven_Coordinates) joined with a forward slash.
- `<version>` is the package version.
- `file_name` is the exact name of the file.
For example, given a package with:
- group ID: `com.my_company`.
- artifact ID: `my_package`.
- version: `1.2.3`.
The request to manually pull a package is:
```shell
curl --verbose "https://<username>:<personal access token>@gitlab.example.com/api/v4/projects/<project_id>/dependency_proxy/packages/maven/com/my_company/my_package/1.2.3/my_package-1.2.3.pom"
```
::EndTabs

40
qa/.devcontainer/devcontainer.json Normal file
View File

@ -0,0 +1,40 @@
// this configuration starts docker container with qa code attached to docker test network
// this assumes omnibus instance has been started beforehand using `gitlab-qa` gem
{
"name": "gitlab-qa",
"privileged": true,
"build": {
"dockerfile": "../Dockerfile",
"context": "../../",
"target": "dev"
},
"runArgs": [
"--net=test"
],
"mounts": [
{
"type": "bind",
"source": "/var/run/docker.sock",
"target": "/var/run/docker.sock"
}
],
"containerEnv": {
"CHROME_DISABLE_DEV_SHM": "true",
"GITLAB_QA_ADMIN_ACCESS_TOKEN": "ypCa3Dzb23o5nvsixwPA",
"QA_GITLAB_URL": "${localEnv:QA_GITLAB_URL}",
"QA_SMOCKER_HOST": "smocker"
},
"onCreateCommand": "echo \"export QA_GITLAB_URL=${QA_GITLAB_URL:-http://$(docker ps | grep -m 1 'gitlab' | awk '{ print $NF }').test}\" >> /root/.bashrc",
"updateContentCommand": "cp .solargraph.yml.example .solargraph.yml",
"customizations": {
"vscode": {
"extensions": [
"castwide.solargraph"
],
"settings": {
"solargraph.diagnostics": true,
"solargraph.formatting": true
}
}
}
}

17
qa/.solargraph.yml.example Normal file
View File

@ -0,0 +1,17 @@
---
include:
- "**/*.rb"
exclude:
- "spec/**/*"
- qa/specs/features/**/*
- ".bundle/**/*"
require: []
domains: []
reporters:
- rubocop # diagnostics
- require_not_found
formatter:
rubocop: # formatting
require_paths: []
plugins: []
max_files: 20000

11
qa/Dockerfile
View File

@ -1,7 +1,7 @@
ARG DOCKER_VERSION=20.10.14
ARG CHROME_VERSION=106
ARG QA_BUILD_TARGET=ee
ARG DOCKER_VERSION=24.0.5
ARG CHROME_VERSION=113
ARG RUBY_VERSION=3.0
ARG QA_BUILD_TARGET=ee
FROM registry.gitlab.com/gitlab-org/gitlab-build-images/debian-bullseye-ruby-${RUBY_VERSION}:bundler-2.3-git-2.36-lfs-2.9-chrome-${CHROME_VERSION}-docker-${DOCKER_VERSION}-gcloud-383-kubectl-1.23 AS foss
LABEL maintainer="GitLab Quality Department <quality@gitlab.com>"
@ -70,4 +70,9 @@ ONBUILD COPY ./jh/qa /home/gitlab/jh/qa
ONBUILD COPY ./jh/lib /home/gitlab/jh/lib
ONBUILD COPY ./jh/config/feature_flags /home/gitlab/jh/config/feature_flags
# Add solargraph gem for devcontainer
# Solargraph is only present in parent Gemfile so we just install it manually
FROM ee as dev
RUN gem install solargraph --force
FROM $QA_BUILD_TARGET

11
qa/README.md
View File

@ -104,11 +104,22 @@ bundle exec rspec <path/to/spec.rb>
```
Note:
- If you want to run tests requiring SSH against GDK, you will need to [modify your GDK setup](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/run_qa_against_gdk.md).
- If this is your first time running GDK, you can use the password pre-set for `root`. [See supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). If you have changed your `root` password, export the password as `GITLAB_INITIAL_ROOT_PASSWORD`.
- By default the tests will run in a headless browser. If you'd like to watch the test execution, you can export `WEBDRIVER_HEADLESS=false`.
- Tests that are tagged `:orchestrated` require special setup (e.g., custom GitLab configuration, or additional services such as LDAP). All [orchestrated tests can be run via `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md). There are also [setup instructions](https://docs.gitlab.com/ee/development/testing_guide/end_to_end/running_tests_that_require_special_setup.html) for running some of those tests against GDK or another local GitLab instance.
#### Remote development
For [VSCode](https://code.visualstudio.com/) user, [.devcontainer](.devcontainer/devcontainer.json) defines configuration to develop E2E tests inside a Docker container which by default is attached to the same network as environments started by [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa) gem. For more information on how to use `dev containers`, see [tutorial](https://code.visualstudio.com/docs/devcontainers/tutorial).
This is useful when developing E2E tests that require GitLab instance with specific omnibus configuration. Typical workflow example:
- Start `GitLab` omnibus instance with specific configuration without running tests, for example: `gitlab-qa Test::Integration::Import EE --no-tests`. For available configurations, see [docs](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md)
- Start dev container from `VSCode` environment
- Develop and run tests from within the container which will automatically execute against started GitLab instance
#### Generic command for a typical GDK installation
The following is an example command you can use if you have configured GDK to run on a specific IP address and port, with a username, password, and personal access token that aren't the defaults, and you would like the test framework to show debug logs: