Previously imports would fail if a merge request note included a
suggestion with an
`ActiveRecord::HasManyThroughCantAssociateThroughHasOneOrManyReflection`
exception.
This was happening because suggestions were listed as a descendant of
merge requests, but this doesn't work because suggestions are directly
associated with notes, not merge requests, and that association is lost.
Rails also disallows creating intializing a has_many association through
a different object.
We fix this by making `suggestions` a child of `notes` within a merge
request. This doesn't fix previously broken exported project exports,
but new exports will work.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65880
Splits auto-refreshing of MR widget into 2 requests:
- the one which uses etag-caching and invalidates the fields on change
- the one without caching
The idea is to gradually move all the fields to etag-cached endpoint
Whenever `PostReceive` is enqueued, `UpdateMergeRequestsWorker`
is enqueued and `MergeRequests::RefreshService` is called, it'll
check if the source branch of each MR asssociated to the push exists
or not via `MergeRequest#source_branch_exists?`. The said method will
call `Repository#branch_exists?` which is cached in `Rails.cache`.
When the cache contains outdated data and the source branch actually
exists, the `MergeRequests#RefreshService` job will close associated
MRs which is not correct.
The fix is to expire the branches cache of the project so we have
updated data during the post receive hook which will help in the
accuracy of the check if we need to close associated MRs or not.
This will help identify Sidekiq jobs that invoke excessive number of
filesystem access.
The timing data is stored in `RequestStore`, but this is only active
within the middleware and is not directly accessible to the Sidekiq
logger. However, it is possible for the middleware to modify the job
hash to pass this data along to the logger.
These counts significantly increase the load time for these
requests. Users can now opt in to receiving the counts by setting
`with_counts=true` in requests. This is a breaking change, but hopefully
a fairly minor one.
EE added a single line to this file that can easily be backported to CE,
but it appears this was overlooked. The method called is a noop in CE,
so no additional changes are necessary.
Add possibilty to view starrers ("stargazers") of a repository & any user's starred repositories
Closes#20137
See merge request gitlab-org/gitlab-ce!24690
https://gitlab.com/gitlab-org/gitlab-ce/issues/62971
Adds support to EnvironmentsController#metrics_dashboard
for the following params: group, title, y_label
These params are used to uniquely identify a panel on
the metrics dashboard.
Metrics are stored in several places, so this adds
utilities to find a specific panel from the database
or filesystem depending on the metric specified.
Also moves some shared utilities into separate classes,
notably default values and errors.
Kubernetes deployments on new clusters will now have
a separate namespace per project environment, instead
of sharing a single namespace for the project.
Behaviour of existing clusters is unchanged.
All new functionality is controlled by the
:kubernetes_namespace_per_environment feature flag,
which is safe to enable/disable at any time.
- Closes#60024
- Change PrometheusClient.new to accept a base url instead of an
already created RestClient
- Use Gitlab::HTTP in PrometheusClient instead of creating RestClient
in PrometheusService
- Move http_options from PrometheusService to
PrometheusClient (follow_redirects: false)
- ensure that base urls don't have the trailing slash
- Created a `PrometheusClient#url` method that might not be strictly
required
- Change rescued exceptions from RestClient::* to
HTTParty::ResponseError where possible and StandardError for the
rest
A nonce-based Content-Security-Policy thwarts XSS attacks by allowing
inline JavaScript to execute if the script nonce matches the header
value. Rails 5.2 supports nonce-based Content-Security-Policy headers,
so provide configuration to enable this and make it work.
To support this, we need to change all `:javascript` HAML filters to the
following form:
```
= javascript_tag nonce: true do
:plain
...
```
We use `%script` throughout our HAML to store JSON and other text, but
since this doesn't execute, browsers don't appear to block this content
from being used and require the nonce value to be present.
Instead of highlighting all lines when not all of them are
needed, only highlight specific lines.
The `BlobPresenter#highlight` method has been updated to
support `since` and `to` params. These params will be used to
limit the content to be highlighted.
Modify `Gitlab::Highlight` to support `since` param which will
then be used to determine the starting line number.
API endpoints for requesting container repositories
and container repositories with their tag information
are enabled for users that want to specify the group
containing the repository rather than the specific project.
- ci_builds.artifacts_expire_at are copied to
ci_job_artifacts.expire_at with incorrect timestamps when the
database timezone is NOT utc
- ci_builds.artifacts_expire_at is `timestamp without time zone` and
ci_job_artifacts.expire_at is `timestamp with time zone` on
postgresql
- Tests fail locally for `rspec
./spec/lib/gitlab/import_export/import_export_spec.rb` without this
change
In dev environments, Sidekiq was encountering the message:
Circular dependency detected while autoloading constant Gitlab::Profiler
This saves some overhead during normal usage.
- Added Importable to models/list.rb
- Did unless: :importable? on board validation
- Created changelog
- Modified haml to show issue boards are importable
- Added needed spec tests
- Modified project.json to include board information
- Added relevant models to all_models
- Added relevant models to import_export
- Added relevant models to safe_model_attributes
Also creates specs
Only allow Helm to be uninstalled if it's the only app
- Remove Tiller leftovers after reser command
- Fixes specs and offenses
Adds changelog file
Fix reset_command specs
This changes `needs:` from weak reference
to have a strong reference.
This means that job will not be created
unless all needs are present as part of
a pipeline.
Simplify SystemHookUrlValidator to inherit from PublicUrlValidator
Refactor specs to move out shared examples to be used in both
system hooks and public url validators.
This MR adds new application setting to network section
`allow_local_requests_from_system_hooks`. Prior to this change
system hooks were allowed to do local network requests by default
and we are adding an ability for admins to control it.
Currently we write out empty CSS classes (`class=""`) every time we
create a new tag. This adds 9 unnecessary bytes per span element. In a
recent trace, I have counted 11950 span elements. So we transported 105
unnecessary kilobytes!
Previously, both InfluxSampler and RubySampler were relying on the
`GC::Profiler.total_time` data which is the sum over the list
of captured GC events. Also, both samplers asynchronously called
`GC::Profiler.clear` which led to incorrect metric data because
each sampler has the wrong assumption it is the only object who calls
`GC::Profiler.clear` and thus could rely on the gathered results between
such calls.
We should ensure that `GC::Profiler.total_time` is called only in one
place making it possible to rely on accumulated data between such wipes.
Also, we need to track the amount of profiler reports we lost.
Remove project from NotesFinder constructor
Add project parameter to specs
Also look for methods in private scope
Fix specs to match new NotesFinder constructor
Concurrent calls to UserMergeToRef RPC updating a single ref
can lead to an opaque fail that is being rescued at Gitaly.
So this commit adds an exclusive lease to the mergeability
check process with the key as the current MR ID.
Peek attempts to serialize results with `to_json`, which calls
`ActiveSupport::JSON`. If an object is passed to `to_json` that contains
instance variables, `ActiveSupport` will attempt to recursively traverse
all variables.
The problem is that we can get into an infinite loop if the instance
references to an instance that references to something else that points
back to the same instance.
To avoid this mess, we just call `to_s` on the object. It appears only
`Gitlab::Git::Repository` and `::Repository` are the culprits here.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65404
- After uninstalling the knative helm chart it's necessary to also
remove some leftover resources to allow the cluster to be clean
and knative to be reinstalleable.
- Adds knative uninstall disclaimer
- Uninstall ksvc before uninstalling knative
Make list of Knative and Ingres resources explicit
- To avoid deleting unwanted resources we are listing exact
which resources will be deleted rather than simply deleting any
resource that contains istio or knative words.
Add method to store session ids by ip
Add new specs for storing session ids
Add cleaning up records after login
Add retrieving anonymous sessions
Add login recaptcha setting
Add new setting to sessions controller
Add conditions for showing captcha
Add sessions controller specs
Add admin settings specs for login protection
Add new settings to api
Add stub to devise spec
Add new translation key
Add cr remarks
Rename class call
Add cr remarks
Change if-clause for consistency
Add cr remarks
Add code review remarks
Refactor AnonymousSession class
Add changelog entry
Move AnonymousSession class to lib
Move store unauthenticated sessions to sessions controller
Move link to recaptcha info
Regenerate text file
Improve copy on the spam page
Change action filter for storing anonymous sessions
Fix rubocop offences
Add code review remarks
Since now the role name can be:
"<a href=\"/job-families/engineering/test-automation-engineer/\">Senior Test Automation Engineer</a>, Create:Source Code"
We need to cope with </a> in the middle.
This rake task had been broken for a while. This fixes the breakages,
adds a test to help avoid future breakages, and adds a few ergonomic
improvements to the task itself.
1. The output isn't great. It can be hard to find hotspots and, even
when you do find them, to find why those are hotspots.
2. It uses some jQuery-specific frontend code which we can remove now
that we don't have this any more.
3. It's only possible to profile the initial request, not any subsequent
AJAX requests.
In preparation for embedding specific metrics in issues
https://gitlab.com/gitlab-org/gitlab-ce/issues/62971,
this commit moves the BaseService for metrics dashboards
to a new services subdirectory. This is purely for the sake
of organization and maintainability.
This backport changes made by EE to the files in the scripts/ directory.
This comes with a few changes to some scripts to make them work in the
single codebase setup.
The SanitizationFilter was running before the WikiFilter. Since
WikiFilter can modify links, we could see links that _should_ be stopped
by SanatizationFilter being rendered on the page. I (kerrizor) had
previously addressed the bug in: 7bc971915b
However, an additional exploit was discovered after that was merged.
Working through the issue, we couldn't simply shuffle the order of
filters, due to some implicit assumptions about the order of filters, so
instead we've extracted the logic that sanitizes a Nokogiri-generated
Node object, and applied it to the WikiLinkFilter as well.
On moving filters around:
Once we start moving around filters, we get cascading failures; fix one,
another one crops up. Many of the existing filters in the WikiPipeline
chain seem to assume that other filters have already done their work,
and thus operate on a "transform anything that's left" basis;
WikiFilter, for instance, assumes any link it finds in the markdown
should be prepended with the wiki_base_path.. but if it does that, it
also turns `href="@user"` into `href="/path/to/wiki/@user"`, which the
UserReferenceFilter doesn't see as a user reference it needs to
transform into a user profile link. This is true for all the reference
filters in the WikiPipeline.
This uses an ActiveRecord subscriber to get queries and calculate the
total query time from that. This means that the total will always be
consistent with the queries in the table. It does however mean that we
could potentially miss some queries that don't go through ActiveRecord.
Making this change also allows us to unify the response JSON a little
bit, making the frontend slightly simpler as a result.
* Reword Project#latest_successful_build_for to
Project#latest_successful_build_for_ref
* Reword Ci::Pipeline#latest_successful_for to
Ci::Pipeline#latest_successful_build_for_ref
This was shown in specs but surely this will be happening in application
code as well if this method is passes a frozen string.
We were also trying to force_encode a OmniAuth::AuthHash which had the
very confusing behaviour of returning nil when it was sent a method that
it did not define. Fix that by only force_encoding a String.
Docker Distribution v2.7.0 shipped with OCI support, but our container
registry client was not updated to handle the manifest format in the
HTTP `Accept` header. As a result, API calls to retrieve a manifest
would return with an error, "OCI manifest found, but accept header does
not support OCI manifests". This would result in blank fields in the
container registry page and prevent tags from being deleted.
To fix this, we just need to add
`application/vnd.oci.image.manifest.v1+json` to the `Accept` header and
configure Faraday to parse the response as JSON. The response structure
is the same as the standard Docker Distribution V2 manifest.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/58685
Closes https://gitlab.com/gitlab-org/gitlab-ee/issues/12877
`Array.reverse_each` is faster than `Array.reverse.each` because:
* reverse.each creates a new array then loops each element
* reverse_each loops in reverse order (no intermediate array created)
* Limits raw requests to 300 per minute and per raw path.
* Add a new attribute to ApplicationSettings so user can change this
value on their instance.
* Uses Gitlab::ActionRateLimiter to limit the raw requests.
* Add a new method into ActionRateLimiter to log the event into auth.log
Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/48717
- Fix `O(n)` complexity of `append_or_update_attribute`,
we append objects to an array and re-save project
- Remove the usage of `keys.include?` as it performs `O(n)`
search, instead use `.has_key?`
- Remove the usage of `.keys.first` as it performs a copy
of all keys, instead use `.first.first`
A project can have the same `LfsObject` linked with up to three
`LfsObjectsProject` records. Each of these records would be for a
different repository, recorded in the `repository_type` property. The
different repositories at time of writing are "project", "wiki", and
"design". See https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13894
This change exports the list of `repository_type`s as a JSON mapping of
oid => repository_types, which are imported to recreate the correct
`LfsObjectsProject` records.
https://gitlab.com/gitlab-org/gitlab-ee/issues/11090
MergeRequests::PushOptionsHandlerService has been updated to allow
creating and updating merge requests with the `title` and
`description` set using git push options.
To create a new merge request and set its title and description:
git push -u origin -o merge_request.create \
-o merge_request.title="My title" \
-o merge_request.description="My description"
To update an existing merge request and set its title and
description:
git push -u origin -o merge_request.title="My title" \
-o merge_request.description="My description"
Issue https://gitlab.com/gitlab-org/gitlab-ce/issues/64320
This will help diagnose the source of excessive I/O from Rugged
calls. To implement this, we need to obtain the full list of arguments
sent to each request method.
To allow project filtering
Prepare summary for accepting multiple groups
Modify deploys group summary class
Add filtering by project name in issues summary
Fix rubocop offences
Add changelog entry
Change name to id in project filtering
Fix rebase problem
Add project extraction
It seems that we missed the backward compatibility support
for profiles in the existing folder.
This commit also fixes some specs to be idempotent
and work in a temporary directory which not always
seems to be the case.
This commit also brings the profile_spec.rb which seems
to be missing.
Previously, in Omnibus, Workhorse expected to listen via the Redis
shared state cluster for the `workhorse:notifications` publish/subscribe
channel, but the Rails code was using the Sidekiq queue cluster for
this. To fix this inconsistency, we make the Rails code use the
persistent cluster, since we don't want Workhorse to be looking at
anything Sidekiq-related.
In some cases ActiveSession.cleanup was not called after authentication,
so for some user ActiveSession lookup keys grew without ever cleaning
up. This Rake task manually iterates over the lookup keys and removes
ones without existing ActiveSession.
This adds a counter to count page creation, which is reflected in the
usage-data we collect.
The number created is stored in Redis, avoiding DB access.
If `GitalyClient#can_use_disk?` returned `false`, it was never cached
properly and led to excessive number of Gitaly calls. Instead of using
`cached_value.present?`, we need to check `cached_value.nil?`.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/64802
* Remove `controller` and `action` labels from duration histogram.
* Create a new simple counter for `controller` and `action`.
* Adjust histogram buckets to observe smaller response times.
If an environment slug is predictable given only the environment
name, we can use the environment slug earlier in the CI variable
evaluation process as we don't have to wait for the environment
record itself to be persisted.
This improves the `add_timestamps_with_timezone` helper by allowing the
column names to be configured. This has the advantage that unnecessary
columns can be avoided, saving space.
A helper for removing the columns is also provided, to be used in the
`down` method of migrations.
Use relative worker identifier for metrics (instead of Process.pid) and
identify when Unicorn/Puma/Sidekiq is used.
Previously, it was assumed that all metrics are gathered from Unicorn
due to hardcoded implementation which was incorrect.
This adds `rugged_duration_ms` and `rugged_calls` fields to
`api_json.log` and `production_json.log`. This will make it easier to
identify performance issues caused by excessive I/O.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/64676
Extend the support for "X-Profile-Token: <token>" to have an additional
header that defines type of profile we are looking for, like:
X-Profile-Mode: execution
X-Profile-Mode: memory
When passing start_branch on committing from the WebIDE, it's possible
that the branch has changed since editing started, which results in the
change being applied on top of the latest commit in the branch and
overwriting the new changes.
By passing the start_sha instead we can make sure that the change is
applied on top of the commit which the user started editing from.
peek-redis resets its counters at the start of an ActionController
notification (`start_processing.action_controller`), which causes it to
miss some Redis queries that precede it, such as the database load
balancer and Rack Attack queries. This produces inconsistencies in the
performance bar between the number of calls and their durations with the
actual calls in the detailed view.
We fix this by getting rid of peek-redis in favor of consolidating all
logic into the `RedisDetailed` view, which tracks Redis queries using
`RequestStore`. This has the nice property of removing thread-specific
counters as well.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/64707
This allows user to specify the pipeline ID
when several pipelines has been triggered
on the same branch and commit.
Signed-off-by: Gaetan Semet <gaetan.semet@renault.com>
MergeRequests::PushOptionsHandlerService has been updated to allow
creating and updating merge requests with the
`remove_source_branch` set using git push options.
To create a new merge request and set it to remove the source branch
when it is merged:
git push -u origin -o merge_request.create \
-o merge_request.remove_source_branch
To update an existing merge request and set it to remove the source
branch when it is merged:
git push -u origin -o merge_request.remove_source_branch
Issue https://gitlab.com/gitlab-org/gitlab-ce/issues/64320
George Koltsov
Thong Kuah
Aleksei Lipniagov
Stan Hu
Armin Hohenegger
Igor
Robert Speicher
Tristan Read
Bob Van Landuyt
Rémy Coutable
Patrick Bajao
James Lopez
Cindy Pallares 🦉
Hordur Freyr Yngvason
Sean McGivern
Felipe Artur
Mark Chao
Yorick Peterse
Kushal Pandya
Grzegorz Bizon
Tiger
Mayra Cabrera
Philippe Lafoucrière
Camil Staps
Reuben Pereira
Sarah Yasonik
Filipa Lacerda
Kerri Miller
Zeger-Jan van de Weg
Ash McKenzie
David Wilkins
Kelly Hair
Gosia Ksionek
Steve Abrams
Douglas Barbosa Alexandre
Jason Colyer
Cameron Boulton
Patrick Derichs
Matija Čupić
David H. Wilkins
Zsolt Kovari
mo
Dylan Griffith
Kamil Trzciński
Lukas '+ alert('Eipi') + ' Eipert
Marcel Amirault
Heinrich Lee Yu
Douwe Maan
Oswaldo Ferreira
Lin Jen-Shin
Andrew Newdigate
drew
João Cunha
Shinya Maeda
Diego Louzán
Giorgenes Gelatti
🙈 jacopo beschi 🙉
Francisco Javier López
Jacopo
Nick Thomas
Ryan Cobb
Andreas Brandl
Oleg Zubchenko
GitLab Release Tools Bot
Peter Leitzen
Douwe Maan
Dmitriy Zaporozhets
Imre Farkas
manojmj
Luke Duncalfe
Luke Duncalfe
Christian Couder
Mohammad Alhashash
Jeremy Jackson
James Keogh
mo khan
Alex Kalderimis
Fabio Papa
Toon Claes
Phil Hughes
Ben Kochie
Mike Greiling
George Koltsov
Qingyu Zhao
Mathieu Parent
Markus Koller
Ash McKenzie
Vladimir Shushlin
Gaetan Semet
Rajendra kadam