http_max_redirects was introduced in 4.2.2, so upgrade kubeclient.
The monkey-patch was global so we will have to check that all instances
of Kubeclient::Client are handled.
Spec all methods of KubeClient
This should provide better confidence that we are indeed disallowing
redirection in all cases
Rename
Introduce Destroy expired job artifacts service
Revert a bit
Add changelog
Use expired
Improve
Fix spec
Fix spec
Use bang for destroy
Introduce iteration limit
Update comment
Simplify more
Refacor
Remove unnecessary thing
Fix comments
Fix coding offence
Make loop helper exception free
This change will publish distributed tracing spans related to
SQL ActiveRecord queries. These queries are unbound, so will still
contain parameter substitution strings in place of actual parameter
values.
This ensures that the SQL strings do not need to be scrubbed, since the
values are not included in the SQL.
More details of rails instrumentation can be found here:
https://guides.rubyonrails.org/active_support_instrumentation.html
This change allows the GitLab rails and sidekiq components to receive
tracing spans from upstream services such as Workhorse and pass these
spans on to downstream services including Gitaly and Sidekiq.
This change will also emit traces for incoming and outgoing requests
using the propagated trace information. This will allow operators and
engineers to view traces across the Workhorse, GitLab Rails, Sidekiq and
Gitaly components.
Additional intra-service instrumentation will be added in future
changes.
'Populate cluster kubernetes namespace' was using factories for their
specs. According to our documentation (see spec/migrations/readme.md),
we should use table helper to create a temproary ActiveRecord::Base
derived model for a table.
When logging arguments from Sidekiq to JSON, restrict the size of `args`
to 10 KB (when converted to JSON). This is to avoid blowing up with
excessively large job payloads.
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20724 removed
`Gitlab::Git::RepositoryMirroring::RemoteError`, but this was still
being handled in the Bitbucket Server importer and only encountered if
some exception other than `Gitlab::Shell::Error` were thrown.
During the review process for adding opentracing factories, a bug was
introduced which caused Jaeger to initialize an invalid tracer. The
bug was due to use sending nil through as a kwarg when the Jaeger
initializer used a non-nil default value.
This is fairly insidious as, the tracer looks like a tracer, but, when
methods are invoked, it throws `NoMethodError` errors. To ensure that
this issue does not happen in future, the tests have been changed to
ensure that the tracer works as expected. This could avoid problems
in future when upgrading to newer versions of Jaeger.
This change will instantiate an OpenTracing tracer and configure it
as the global tracer when the GITLAB_TRACING environment variable is
configured. GITLAB_TRACING takes a "connection string"-like value,
encapsulating the driver (eg jaeger, etc) and options for the driver.
Since each service, whether it's written in Ruby or Golang, uses the
same connection-string, it should be very easy to configure all
services in a cluster, or even a single development machine to be
setup to use tracing.
Note that this change does not include instrumentation or propagation
changes as this is a way of breaking a previous larger change into
components. The instrumentation and propagation changes will follow
in separate changes.
This refactors some of the logic used for protecting default branches,
in particular Project#after_create_default_branch. The logic for this
method is moved into a separate service class. Ideally we'd get rid of
Project#after_create_default_branch entirely, but unfortunately
Project#after_import depends on it. This means it has to stick around
until we also refactor Project#after_import.
For branch protection levels we introduce
Gitlab::Access::BranchProtection, which provides a small wrapper around
Integer based branch protection levels. Using this class removes the
need for having to constantly refer to Gitlab::Access::PROTECTION_*
constants.
Introduce `default:` configuration entry setting that makes it possible
to configure a default value of an entry, what overrides class-level
`def self.default` value.
After upgrading to Ruby 2.5.3, we switched `URI.join` in favor of
`Gitlab::Utils.append_path`. However,
ActionController::Base.helpers.image_path can return a full URL if a CDN
host is present.
Rails provides a way to generate the full URL using the asset path, but
that doesn't appear to work because `request` is nil`.
Revert to the previous behavior to handle CDNs and relative URLs.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/56268
On GitLab.com, we saw numerous duplicate disk entry inserts because
the migration was not taking the routes table into account. We now
implement this in the migration to be consistent.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/56061
In the previous code, we locked the project during the migration
scheduling step, which works fine for small setups, but can be
problematic in really big installations.
We now moved the logic to inside the worker, so we minimize the time a
project will be read-only. We also make sure we only do that if
reference counter is `0` (no current operation is in progress).
As mentioned in
https://gitlab.com/gitlab-org/gitlab-ee/issues/9035#note_129093444,
Rails 5 switched ActionDispatch::Request so that it no longer inherits
Rack::Request directly. A middleware that uses Rack::Request to
read the environment may see stale request parameters if
another middleware modifies the environment via ActionDispatch::Request.
To be safe, we should be using ActionDispatch::Request everywhere.
Mixing and matching the use of Rack::Request and ActionDispatch::Request
in Rails 5 is bad, particularly if you have middleware that
manipulates or accesses environment variables.
`Gitlab::Middleware::Multipart` attempts to rewrite request parameters
to the proper values (e.g. replacing `data_file` with
`UploadedFile`). It does this by calling `Rack::Request#update_params`,
which essentially updates `env['rack.request.form_hash']`.
By changing to `ActionDispatch::Request`, the Go middleware was causing
the request parameters to be stored inside
`env['action_dispatch.request.request_parameters']`. Later calls to
`Rack::Request#update_params` would not have any effect because it would
attempt to update `env['rack.request.form_has']` instead of
`env['action_dispatch.request.request_parameters']`. As a result, the
controller still saw the old parameters.
Since the Go middleware appears to be using `ActionDispatch::Request`
for authorization methods, we can switch the multipart middleware to
use it too.
Closes https://gitlab.com/gitlab-org/gitlab-ee/issues/9035
These are data columns that store runtime configuration
of build needed to execute it on runner and within pipeline.
The definition of this data is that once used, and when no longer
needed (due to retry capability) they can be freely removed.
They use `jsonb` on PostgreSQL, and `text` on MySQL (due to lacking
support for json datatype on old enough version).
When the `changes` passed to `GitAccess` are the literal string `_any`,
which indicates that this is a pre-authorization check, we now check
whether the user can push to any branch in the project in question,
instead of running the per-change check with `oldrev` `_any`, `newrev`
`nil`, and `ref` `nil`.
This commit, introduced in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23812,
fixes a problem creating a displaying image diff notes when the image
is stored in LFS. The main problem was that `Gitlab::Diff::File` was
returning an invalid valid in `text?` for this kind of files.
It also fixes a rendering problem with other LFS files, like text
ones. They LFS pointer shouldn't be shown when LFS is enabled
for the project, but they were.
Calling `Fog::Storage::AWS::Directories#get` requires the
ListAllMyBuckets permission, but we can avoid that extra query and
permission by initializing the directory with a specific bucket:
https://stackoverflow.com/a/12288581/1992201
* upstream/master: (115 commits)
[CE] Speed up login page usage
Add new line and comments
Fix the seeder 24_forks.rb cannot find public project
Milestones on community contribution issues
Removed Gitlab Upgrader found in /lib/gitlab/upgrader.rb
Fix and move specs into admin_disables_git_access_protocol_spec.rb
Fix HTTP/SSH clone panel for mobile
Add spec for HTTP/SSH clone panel
Fix missing Git clone button when protocol restriction setting enabled
Fix deprecation: Using positional arguments in integration tests
Extend override check to also check arity
Update tm cli version
Bump Gitaly version to v1.12.0
Add @dbalexandre to CODEOWNERS
Update verbiage for clarity
Change group-cluster beta to regular note
Change alpha states to use note instead of warning
Update registry section. Update serverless.yaml formatting
Clarify obtaining application URL
Add @godfat to CODEOWNERS
...
This commit handles note diffs caching, which considerably improves
the performance on merge requests with lots of comments.
Important to note that the caching approach taken here is different
from `Gitlab::Diff::HighlightCache`. We do not reset the whole cache
when a new push is sent or anything else. That's because discussions
diffs are persisted and do not change.
Correct the ordering of metrics on performance dashboard. Before common
metrics were moved into the DB, metric groups were ordered by the
priority defined in the common_metrics.yml file.
This commit adds a priority to each metric group in the PrometheusMetric
model.
It also combines title, priority and required_metrics into one frozen
GROUP_DETAILS hash so that the code is clearer.
This can be done since there is a fixed set of groups which are not
configurable.
This replaces the use of fake_application_settings with
`::ApplicationSetting.build`_from_defaults. The reason is that
`fake_application_settings` doesn't have the custom accessors that
`ApplicationSetting` has, e.g. `#commit_email_hostname`, thus this
can lead to unexpected `nil` values which comes from the database
column instead of `.default_commit_email_hostname` returned by
`ApplicationSetting#commit_email_hostname`.
Using `::ApplicationSetting.build_from_defaults` should be safe as it
doesn't try to `INSERT` a DB record, in contrary to
`::ApplicationSetting.create_from_defaults` which we used to use, and
which created issues that the introduction of
`fake_application_settings` tried to resolve (575dced5).
Signed-off-by: Rémy Coutable <remy@rymai.me>
This change the write to accept an options hash to make
it compatible with ActiveSupport::Cache::Store#write method.
The options hash are not passed to the underlying cache
implementation because RequestStore#write accepts only
a key, and value params.
When the BFG object map file is in object storage (i.e., uploads in
general are placed into object storage), we get an instance of the
Gitlab::HttpIO class. This doesn't behave as expected when you try to
read past EOF, so we need to explicitly check for this condition to
avoid ending up in a tight loop around io.read
This commit adds a name to each release, defaulting it to tag name,
keeps track of the SHA when a new release is created and tracks the
current user as release author.
spec/lib/gitlab/background_migration/migrate_stage_status_spec.rb:9: warning: already initialized constant STATUSES
spec/lib/gitlab/background_migration/migrate_build_stage_spec.rb:9: warning: previous definition of STATUSES was here
It adds a base class for CompareTestReportsService
containing common code with CompareLicenseManagementReportsService
which is present in GitLab Enterprise Edition.
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.
Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.
When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.
These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.
Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.
To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.
Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)
In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
If a commit has two parents, one is direct ancestor, and one is not,
and the order of `commits` is in such fashion that the non-ancestor
side is visited first, the commit would be determined as non-ancestor,
when in fact it can be.
Therefore we should first determine all direct ancestors
prior to analyzing.
Analyze new commits graph to determine each commit's merge commit.
Fix "merged with [commit]" info for merge requests being merged
automatically by other actions.
Allow analyzing upto the relevant commit
It gathers list of file paths to delete before destroying
the parent object. Then after the parent_object is destroyed
these paths are scheduled for deletion asynchronously.
Carrierwave needed associated model for deleting upload file.
To avoid this requirement, simple Fog/File layer is used directly
for file deletion, this allows us to use just a simple list of paths.
The Correlation ID is taken or generated from received X-Request-ID.
Then it is being passed to all executed services (sidekiq workers
or gitaly calls).
The Correlation ID is logged in all structured logs as `correlation_id`.
* removed 100 limit on file search results because we
load all results anyway
* expensive processing (parsing match content, utf encoding)
is done only for selected page in paginated output
The API permits path traversal characters like '../' to be passed down
to the template finder. Detect these requests and cause them to fail
with a 500 response code.
- Rename ordered_group_clusters_for_project ->
ancestor_clusters_for_clusterable
- Improve name of order option. It makes much more sense to have `hierarchy_order: :asc`
and `hierarchy_order: :desc`
- Allow ancestor_clusters_for_clusterable for group
- Re-use code already present in Project
Look for matching clusters starting from the closest ancestor, then go
up the ancestor tree.
Then use Ruby to get clusters for each group in order. Not that
efficient, considering we will doing up to `NUMBER_OF_ANCESTORS_ALLOWED`
number of queries, but it's a finite number
Explicitly order query by depth
This allows us to control ordering explicitly and also to reverse the
order which is useful to allow us to be consistent with
Clusters::Cluster.on_environment (EE) which does reverse ordering.
Puts querying group clusters behind Feature Flag. Just in case we have
issues with performance, we can easily disable this
If the service fails mid-point, then we should be able to re-run this
service. So, detect presence of any previously created Kubernetes
resource and update or create accordingly.
Fix specs accordingly. In the case of finalize_creation_service_spec.rb,
I decided to stub out the async worker rather than maintaining
individual stubs for various kubeclient calls for that worker.
Currently, if a job does not have only/except policies, the policy is considered as an unspecified state, and therefore the job is executed regardless of how it's executed and which branch/tags are targetted.
Ideally, this should be specified as only: ['branches', 'tags'], as it indicates that unspecified policy jobs are meant to run on any git references.
If the service fails mid-point, then we should be able to re-run this
service. So, detect presence of any previously created Kubernetes
resource and update or create accordingly.
Fix specs accordingly. In the case of finalize_creation_service_spec.rb,
I decided to stub out the async worker rather than maintaining
individual stubs for various kubeclient calls for that worker.
Also add test cases for group clusters
Gabriel Mazetto
Nick Thomas
Douwe Maan
Valery Sizov
Kamil Trzciński
Grzegorz Bizon
Grzegorz Bizon
Thong Kuah
Lin Jen-Shin
Rémy Coutable
Shinya Maeda
Andrew Newdigate
Robert Speicher
Mayra Cabrera
Vladimir Shushlin
Sean McGivern
Matija Čupić
Stan Hu
Takuya Noguchi
Yorick Peterse
Bob Van Landuyt
Ash McKenzie
Brandon Labuschagne
Toon Claes
Douglas Barbosa Alexandre
Oswaldo Ferreira
Mark Chao
Reuben Pereira
Brett Walker
mortyccp
Douwe Maan
John Jarvis
Francisco Javier López
Jan Provaznik
Jacob Wolen
Jacopo
Ahmad Hassan
Dmitriy Zaporozhets
Mike Greiling
Jarka Košanová
blackst0ne
James Lopez
Zeger-Jan van de Weg
Jasper Maes
Alessio Caiazza
Sean Nichols
Semyon Pupkov
Gilbert Roulot
Clement Ho
Cindy Pallares
James Edwards-Jones
Dylan Griffith
🤖 GitLab Bot 🤖