After upgrading to Ruby 2.5.3, we switched `URI.join` in favor of
`Gitlab::Utils.append_path`. However,
ActionController::Base.helpers.image_path can return a full URL if a CDN
host is present.
Rails provides a way to generate the full URL using the asset path, but
that doesn't appear to work because `request` is nil`.
Revert to the previous behavior to handle CDNs and relative URLs.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/56268
On GitLab.com, we saw numerous duplicate disk entry inserts because
the migration was not taking the routes table into account. We now
implement this in the migration to be consistent.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/56061
In the previous code, we locked the project during the migration
scheduling step, which works fine for small setups, but can be
problematic in really big installations.
We now moved the logic to inside the worker, so we minimize the time a
project will be read-only. We also make sure we only do that if
reference counter is `0` (no current operation is in progress).
As mentioned in
https://gitlab.com/gitlab-org/gitlab-ee/issues/9035#note_129093444,
Rails 5 switched ActionDispatch::Request so that it no longer inherits
Rack::Request directly. A middleware that uses Rack::Request to
read the environment may see stale request parameters if
another middleware modifies the environment via ActionDispatch::Request.
To be safe, we should be using ActionDispatch::Request everywhere.
Mixing and matching the use of Rack::Request and ActionDispatch::Request
in Rails 5 is bad, particularly if you have middleware that
manipulates or accesses environment variables.
`Gitlab::Middleware::Multipart` attempts to rewrite request parameters
to the proper values (e.g. replacing `data_file` with
`UploadedFile`). It does this by calling `Rack::Request#update_params`,
which essentially updates `env['rack.request.form_hash']`.
By changing to `ActionDispatch::Request`, the Go middleware was causing
the request parameters to be stored inside
`env['action_dispatch.request.request_parameters']`. Later calls to
`Rack::Request#update_params` would not have any effect because it would
attempt to update `env['rack.request.form_has']` instead of
`env['action_dispatch.request.request_parameters']`. As a result, the
controller still saw the old parameters.
Since the Go middleware appears to be using `ActionDispatch::Request`
for authorization methods, we can switch the multipart middleware to
use it too.
Closes https://gitlab.com/gitlab-org/gitlab-ee/issues/9035
These are data columns that store runtime configuration
of build needed to execute it on runner and within pipeline.
The definition of this data is that once used, and when no longer
needed (due to retry capability) they can be freely removed.
They use `jsonb` on PostgreSQL, and `text` on MySQL (due to lacking
support for json datatype on old enough version).
When the `changes` passed to `GitAccess` are the literal string `_any`,
which indicates that this is a pre-authorization check, we now check
whether the user can push to any branch in the project in question,
instead of running the per-change check with `oldrev` `_any`, `newrev`
`nil`, and `ref` `nil`.
This commit, introduced in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23812,
fixes a problem creating a displaying image diff notes when the image
is stored in LFS. The main problem was that `Gitlab::Diff::File` was
returning an invalid valid in `text?` for this kind of files.
It also fixes a rendering problem with other LFS files, like text
ones. They LFS pointer shouldn't be shown when LFS is enabled
for the project, but they were.
Calling `Fog::Storage::AWS::Directories#get` requires the
ListAllMyBuckets permission, but we can avoid that extra query and
permission by initializing the directory with a specific bucket:
https://stackoverflow.com/a/12288581/1992201
* upstream/master: (115 commits)
[CE] Speed up login page usage
Add new line and comments
Fix the seeder 24_forks.rb cannot find public project
Milestones on community contribution issues
Removed Gitlab Upgrader found in /lib/gitlab/upgrader.rb
Fix and move specs into admin_disables_git_access_protocol_spec.rb
Fix HTTP/SSH clone panel for mobile
Add spec for HTTP/SSH clone panel
Fix missing Git clone button when protocol restriction setting enabled
Fix deprecation: Using positional arguments in integration tests
Extend override check to also check arity
Update tm cli version
Bump Gitaly version to v1.12.0
Add @dbalexandre to CODEOWNERS
Update verbiage for clarity
Change group-cluster beta to regular note
Change alpha states to use note instead of warning
Update registry section. Update serverless.yaml formatting
Clarify obtaining application URL
Add @godfat to CODEOWNERS
...
This commit handles note diffs caching, which considerably improves
the performance on merge requests with lots of comments.
Important to note that the caching approach taken here is different
from `Gitlab::Diff::HighlightCache`. We do not reset the whole cache
when a new push is sent or anything else. That's because discussions
diffs are persisted and do not change.
Correct the ordering of metrics on performance dashboard. Before common
metrics were moved into the DB, metric groups were ordered by the
priority defined in the common_metrics.yml file.
This commit adds a priority to each metric group in the PrometheusMetric
model.
It also combines title, priority and required_metrics into one frozen
GROUP_DETAILS hash so that the code is clearer.
This can be done since there is a fixed set of groups which are not
configurable.
This replaces the use of fake_application_settings with
`::ApplicationSetting.build`_from_defaults. The reason is that
`fake_application_settings` doesn't have the custom accessors that
`ApplicationSetting` has, e.g. `#commit_email_hostname`, thus this
can lead to unexpected `nil` values which comes from the database
column instead of `.default_commit_email_hostname` returned by
`ApplicationSetting#commit_email_hostname`.
Using `::ApplicationSetting.build_from_defaults` should be safe as it
doesn't try to `INSERT` a DB record, in contrary to
`::ApplicationSetting.create_from_defaults` which we used to use, and
which created issues that the introduction of
`fake_application_settings` tried to resolve (575dced5).
Signed-off-by: Rémy Coutable <remy@rymai.me>
This change the write to accept an options hash to make
it compatible with ActiveSupport::Cache::Store#write method.
The options hash are not passed to the underlying cache
implementation because RequestStore#write accepts only
a key, and value params.
When the BFG object map file is in object storage (i.e., uploads in
general are placed into object storage), we get an instance of the
Gitlab::HttpIO class. This doesn't behave as expected when you try to
read past EOF, so we need to explicitly check for this condition to
avoid ending up in a tight loop around io.read
This commit adds a name to each release, defaulting it to tag name,
keeps track of the SHA when a new release is created and tracks the
current user as release author.
spec/lib/gitlab/background_migration/migrate_stage_status_spec.rb:9: warning: already initialized constant STATUSES
spec/lib/gitlab/background_migration/migrate_build_stage_spec.rb:9: warning: previous definition of STATUSES was here
It adds a base class for CompareTestReportsService
containing common code with CompareLicenseManagementReportsService
which is present in GitLab Enterprise Edition.
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.
Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.
When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.
These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.
Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.
To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.
Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)
In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
If a commit has two parents, one is direct ancestor, and one is not,
and the order of `commits` is in such fashion that the non-ancestor
side is visited first, the commit would be determined as non-ancestor,
when in fact it can be.
Therefore we should first determine all direct ancestors
prior to analyzing.
Analyze new commits graph to determine each commit's merge commit.
Fix "merged with [commit]" info for merge requests being merged
automatically by other actions.
Allow analyzing upto the relevant commit
It gathers list of file paths to delete before destroying
the parent object. Then after the parent_object is destroyed
these paths are scheduled for deletion asynchronously.
Carrierwave needed associated model for deleting upload file.
To avoid this requirement, simple Fog/File layer is used directly
for file deletion, this allows us to use just a simple list of paths.
The Correlation ID is taken or generated from received X-Request-ID.
Then it is being passed to all executed services (sidekiq workers
or gitaly calls).
The Correlation ID is logged in all structured logs as `correlation_id`.
* removed 100 limit on file search results because we
load all results anyway
* expensive processing (parsing match content, utf encoding)
is done only for selected page in paginated output
The API permits path traversal characters like '../' to be passed down
to the template finder. Detect these requests and cause them to fail
with a 500 response code.
- Rename ordered_group_clusters_for_project ->
ancestor_clusters_for_clusterable
- Improve name of order option. It makes much more sense to have `hierarchy_order: :asc`
and `hierarchy_order: :desc`
- Allow ancestor_clusters_for_clusterable for group
- Re-use code already present in Project
Look for matching clusters starting from the closest ancestor, then go
up the ancestor tree.
Then use Ruby to get clusters for each group in order. Not that
efficient, considering we will doing up to `NUMBER_OF_ANCESTORS_ALLOWED`
number of queries, but it's a finite number
Explicitly order query by depth
This allows us to control ordering explicitly and also to reverse the
order which is useful to allow us to be consistent with
Clusters::Cluster.on_environment (EE) which does reverse ordering.
Puts querying group clusters behind Feature Flag. Just in case we have
issues with performance, we can easily disable this
If the service fails mid-point, then we should be able to re-run this
service. So, detect presence of any previously created Kubernetes
resource and update or create accordingly.
Fix specs accordingly. In the case of finalize_creation_service_spec.rb,
I decided to stub out the async worker rather than maintaining
individual stubs for various kubeclient calls for that worker.
Currently, if a job does not have only/except policies, the policy is considered as an unspecified state, and therefore the job is executed regardless of how it's executed and which branch/tags are targetted.
Ideally, this should be specified as only: ['branches', 'tags'], as it indicates that unspecified policy jobs are meant to run on any git references.
If the service fails mid-point, then we should be able to re-run this
service. So, detect presence of any previously created Kubernetes
resource and update or create accordingly.
Fix specs accordingly. In the case of finalize_creation_service_spec.rb,
I decided to stub out the async worker rather than maintaining
individual stubs for various kubeclient calls for that worker.
Also add test cases for group clusters
A tablesample count executes in two phases:
* Estimate table sizes based on reltuples.
* Based on the estimate:
* If the table is considered 'small', execute an exact relation count.
* Otherwise, count on a sample of the table using TABLESAMPLE.
The size of the sample is chosen in a way that we always roughly scan
the same amount of rows (see TABLESAMPLE_ROW_TARGET).
Having an invalid KUBECONFIG without a token in it is not helpful. This
only became possible recently now that we are creating a separate
namespace and service account (and hence token) to send to the runners.
This led to somewhat surprising results when troubleshooting
https://gitlab.com/gitlab-org/gitlab-ce/issues/53879 as I found that the
KUBECONFIG was still being passed but KUBE_TOKEN was not. These things
really should have been linked.
Furthermore now that we are also using the [presence of KUBECONFIG to
decide whether or not to run build steps in Auto
DevOps](294d15be3e/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml (L164))
I think it makes even more sense to ensure that KUBECONFIG is a complete
config if passed to a job.
Adds gitlab.impersonation_enabled config option defaulting to true to
keep the current default behaviour.
Only the act of impersonation is modified, impersonation token
management is not affected.
[master] Resolve "Personal access token with only `read_user` scope can be used to authenticate any web request"
See merge request gitlab/gitlabhq!2583
If the EncryptColumns background migration runs in a sidekiq with a
stale view of the database schema, or when the purported destination
columns don't actually exist, data loss can result. Attempt to work
around these issues by reloading schema information before running
the migration, and raising errors if the model reports that any of its
source or destination columns are missing.
This suggests possibly related issues when the user types a title.
This uses GraphQL to allow the frontend to request the exact
data that is requires. We also get free caching through the Vue Apollo
plugin.
With this we can include the ability to import .graphql files in JS
and Vue files.
Also we now have the Vue test utils library to make testing
Vue components easier.
Closes#22071
We want to keep failed install pods around so that it is easier to debug
why a failure occured. With this change we also need to ensure that we
remove a previous pod with the same name before installing so that
re-install does not fail.
Another change here is that we no longer need to catch errors from
delete_pod! in CheckInstallationProgressService as we now catch the
ResourceNotFoundError in Helm::Api. The catch statement in
CheckInstallationProgressService was also probably too broad before and
should have been narrowed down simply to ResourceNotFoundError.
`Rugged::Reference.valid_name?` used in
`Gitlab::GitRefValidator.validate` fails on strings containing null
bytes because it uses `StringValueCStr()`. Per
https://silverhammermba.github.io/emberb/c/:
Ruby’s String kinda corresponds to C’s char*. The simplest macro is
StringValueCStr() which returns a null-terminated char* for a
String. The problem here is that a Ruby String might contain nulls - in
which case StringValueCStr() will raise an ArgumentError!
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/54466
Use shelling out to git to write refs instead of rugged, hoping to
avoid creating invalid refs.
To update HEAD we switched to using `git symbolic-ref`.
Previously, we used a personal access token. This had a couple of
problems:
1. If the user didn't have a PAT, we couldn't impersonate them.
2. It depended on reading the raw PAT from the database.
Instead, we can monkey-patch the authentication methods on
ApplicationController (overriding the Devise ones), and remove them once
we're done. This does mean that profiles will not profile auth
correctly, so for that, use a PAT directly.
By specifying `key`, we get a different lazy batch loader for each
repository, which means that accessing a lazy object from one repository
will only result in that repository's objects being fetched, not those
of other repositories, saving us some unnecessary Gitaly lookups.
Private commit emails were introduced in !22560, but some parts of
GitLab were not updated to take account of them. This commit adds
support in places that were missed.
Stan Hu
Toon Claes
Nick Thomas
Matija Čupić
Robert Speicher
Douglas Barbosa Alexandre
Oswaldo Ferreira
Mark Chao
Gabriel Mazetto
Reuben Pereira
Grzegorz Bizon
Kamil Trzciński
Shinya Maeda
Sean McGivern
Brett Walker
mortyccp
Douwe Maan
John Jarvis
Takuya Noguchi
Francisco Javier López
Douwe Maan
Jan Provaznik
Lin Jen-Shin
Jacob Wolen
Jacopo
Ahmad Hassan
Dmitriy Zaporozhets
Mike Greiling
Jarka Košanová
Rémy Coutable
blackst0ne
James Lopez
Yorick Peterse
Zeger-Jan van de Weg
Ash McKenzie
Jasper Maes
Alessio Caiazza
Sean Nichols
Semyon Pupkov
Gilbert Roulot
Clement Ho
Cindy Pallares
James Edwards-Jones
Dylan Griffith
🤖 GitLab Bot 🤖
Thong Kuah
Grzegorz Bizon
Andreas Brandl
Imre Farkas
Steve Azzopardi
Filipa Lacerda
Phil Hughes
Tiago Botelho
Bob Van Landuyt
Heinrich Lee Yu
Peter Leitzen
Amit Rathi
Steve Azzopardi
Fatih Acet
Kirill Zaitsev