When we un-escape HTML text to find references in it, we should then
re-escape the whole text again, not only found matches.
Because we replace matches with milestone/label links (which contain
HTML tags we don't want to escape again), we re-escape HTML text
with placeholders instead of these links and then replace placeholders
in the escaped text.
Increase sort options for issues list from updated_at and create_at,
to include more options close to what is required in actual issue list
UI.
This helps us to use REST API for issues list with sorting capabilities
https://gitlab.com/gitlab-org/gitlab-ce/issues/57402
Similar to https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31834,
we see that in https://gitlab.com/gitlab-org/gitlab-ce/issues/65957
there can be hundreds, even thousands, of Gitaly requests in the
`/api/:version/projects/:id/merge_requests/:noteable_id/notes` endpoint.
Previously, the API to retrieve notes generated hundreds of Gitaly calls
to determine whether a system note should be shown to the user. It did
this by:
1. Rendering the Markdown
2. Extracting cross-references from the Markdown
3. Issuing a Gitaly `FindCommit` RPC for every reference to validate
that the commit exists.
The last step is unnecessary because we don't need to display a commit
if the user doesn't have access to the project in the first place.
`RendersNotes#prepare_notes_for_rendering` is already used in
`MergeRequestsController`, which is why we don't see N+1 Gitaly calls
there. We use it here to optimize the note redaction process.
Root namespaces have storage statistics.
This commit allows namespace owners to get those stats via GraphQL
queries like the following one
{
namespace(fullPath: "a_namespace_path") {
rootStorageStatistics {
storageSize
repositorySize
lfsObjectsSize
buildArtifactsSize
packagesSize
wikiSize
}
}
}
Limiting the size of issuable description and comments to 1_000_000,
which is close to ~1MB of ASCII characters, which represents 99.9% of
all descriptions and comments we have in DB at the moment. This should
help prevent DoS attacks when comments contain refference strings.
Also this change updates regexp matching the namespaces paths by
limiting the namespaces paths to Namespace::NUMBER_OF_ANCESTORS_ALLOWED,
as we allow 20 levels deep groups.
see https://gitlab.com/gitlab-org/gitlab-ce/issues/61974#note_191274234
When post-processing relative links to absolute links
RelativeLinkFilter didn't take into consideration that
internal repository data could be exposed for users
that do not have repository access to the project.
This commit solves that by checking whether the user
can `download_code` at this repository, avoiding any
processing of this filter if the user can't.
Additionally, if we're processing for a group (
no project was given), we check if the user can
read it in order to expand the href as an extra.
That doesn't seem necessarily a breach now,
but an extra check doesn't hurt as after all
the user needs to be able to `read_group`.
If we process message that is not designated to us
previously we would fire a separate Thread for that.
We don't need to do it. We can cheaply check if thread
is available, if it is, we can perform expensive operation
then.
This makes:
- very shallow `Middleware::Monitor` to only request tracking
of sidekiq jobs,
- `SidekiqStatus::Monitor` to be responsible to maintain persistent
connection to receive messages,
- `SidekiqStatus::Monitor` to always use structured logging
and instance variables
This adds a middleware to track all threads
for running jobs.
This makes sidekiq to watch for redis-delivered notifications.
This makes be able to send notification to interrupt
running sidekiq jobs.
This does not take into account any native code,
as `Thread.raise` generates exception once the control gets
back to Ruby.
The separate measure should be taken to interrupt gRPC, shellouts,
or anything else that escapes Ruby.
Adding, destroying and toggling emoji previously lacked services and
instead were performed through methods called on Awardable models.
This led to inconsistencies where relevant todos would be marked as done
only when emoji were awarded through our controllers, but not through
the API. Todos could also be marked as done when an emoji was being
removed.
Behaviour changes
- Awarding emoji through the API will now mark a relevant Todo as done
- Toggling an emoji off (destroying it) through our controllers will no
longer mark a relevant Todo as done
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/63372
- Added Gitlab::Ci::Config::Entry::Rules and Gitlab::Ci::Config::Entry::Rules:Rule
to handle lists of Rule objects to be evalauted for job inclusion
- Added `if:` and `changes:` as available Rules::Rule::Clause classes
- Added Rules handling logic to Seed::Build#included? with extra specs
- Use DisallowedKeysValidator to mutually exclude rules: from only:/except: on job config
Current `auth.log` uses `fullpath` and `ip`, while `api_json.log` uses
`remote_ip` and `path` for the same fields. Let's standardize these
namings to make it easier for people working with the data.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/66167
- Add mail interceptor the signs outgoing email with SMIME
- Add lib and helpers to work with SMIME data
- New configuration params for setting up SMIME key and cert files
This functionality is available in the /autocomplete users pseudo-API.
We're attempting to replace that with the canonical API, so it needs
support for this parameter too.
In GitLab 9.0,
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9661 removed the
`subscribed` flag from the API when the user requested a list of issues
or merge requests since calculating this value triggers extensive
Markdown processing.
In GitLab 12.0 via a4fbf39e, we accidentally reintroduced this
performance regression by changing `IssueBasic` to `Issue` in
`entities.rb`. This showed up as a Gitaly N+1 issue since the Markdown
processing would attempt to extract a commit if it detected a regex that
matched a commit.
We restore the prior behavior by once again removing the `subscribed`
flag for the bulk list of issues and merge requests and add a test to
ensure they aren't reintroduced.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/66202
Previously `ProjectCacheWorker` would be scheduled once per ref, which
would generate unnecessary I/O and load on Sidekiq, especially if many
tags or branches were pushed at once. `ProjectCacheWorker` would expire
three items:
1. Repository size: This only needs to be updated once per push.
2. Commit count: This only needs to be updated if the default branch
is updated.
3. Project method caches: This only needs to be updated if the default
branch changes, but only if certain files change (e.g. README,
CHANGELOG, etc.).
Because the third item requires looking at the actual changes in the
commit deltas, we schedule one `ProjectCacheWorker` to handle the first
two cases, and schedule a separate `ProjectCacheWorker` for the third
case if it is needed. As a result, this brings down the number of
`ProjectCacheWorker` jobs from N to 2.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52046
This allows us to check inclusion for the *_exists? methods without
downloading the full list of branch names, which is over 100KiB in size
for gitlab-ce at the moment.
As per https://gitlab.com/gitlab-org/gitlab-ce/issues/46043, project
templates should be squashed before updating, so that repositories
created from these templates don't include the full history of the
backing repository.
Previously, the API to retrieve discussions from merge requests often
generated hundreds of Gitaly calls to determine whether a system note
should be shown to the user. It did this by:
1. Rendering the Markdown
2. Extracting cross-references from the Markdown
3. For cross-references that were commits, a Gitaly FindCommit RPC
would be issued to validate that the commit exists.
The last step is unnecessary because we don't need to display a commit
if the user doesn't have access to the project in the first place.
`RendersNotes#prepare_notes_for_rendering` is already used in
`MergeRequestsController`, which is why we don't see N+1 Gitaly calls
there. We use it here to optimize the note redaction process.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65957
Add Gitlab::VisibilityLevelChecker that verifies
selected project visibility level (or overridden param)
is not restricted when creating or importing a project
This makes us to expand variables only when needed,
instead of requesting all variables each time.
This specifically helps in situation when explicit name
of `environment: production` is used.
Previously each tag in a push would invoke the Gitaly `FindAllTags` RPC
since the tag cache would be invalidated with every tag.
We can eliminate those extraneous calls by expiring the tag cache once
in `PostReceive` and taking advantage of the cached tags.
Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/65795
This changes the `needs:` logic to require
that all jobs to be present. Instead of skipping
do fail the pipeline creation if `needs:` dependency
is not found.
This commit reduces I/O load and memory utilization during PostReceive
for the common case when no project hooks or services are set up.
We saw a Gitaly N+1 issue in `CommitDelta` when many tags or branches
are pushed. We can reduce this overhead in the common case because we
observe that most new projects do not have any Web hooks or services,
especially when they are first created. Previously, `BaseHooksService`
unconditionally iterated through the last 20 commits of each ref to
build the `push_data` structure. The `push_data` structured was used in
numerous places:
1. Building the push payload in `EventCreateService`
2. Creating a CI pipeline
3. Executing project Web or system hooks
4. Executing project services
5. As the return value of `BaseHooksService#execute`
6. `BranchHooksService#invalidated_file_types`
We only need to generate the full `push_data` for items 3, 4, and 6.
Item 1: `EventCreateService` only needs the last commit and doesn't
actually need the commit deltas.
Item 2: In addition, `Ci::CreatePipelineService` only needed a subset of
the parameters.
Item 5: The return value of `BaseHooksService#execute` also wasn't being
used anywhere.
Item 6: This is only used when pushing to the default branch, so if
many tags are pushed we can save significant I/O here.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65878
Fic
Querying all counts for the different search results in the same request
led to timeouts, so we now only calculate the count for the *current*
search results, and request the others in separate asynchronous calls.
Previously imports would fail if a merge request note included a
suggestion with an
`ActiveRecord::HasManyThroughCantAssociateThroughHasOneOrManyReflection`
exception.
This was happening because suggestions were listed as a descendant of
merge requests, but this doesn't work because suggestions are directly
associated with notes, not merge requests, and that association is lost.
Rails also disallows creating intializing a has_many association through
a different object.
We fix this by making `suggestions` a child of `notes` within a merge
request. This doesn't fix previously broken exported project exports,
but new exports will work.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65880
Splits auto-refreshing of MR widget into 2 requests:
- the one which uses etag-caching and invalidates the fields on change
- the one without caching
The idea is to gradually move all the fields to etag-cached endpoint
Whenever `PostReceive` is enqueued, `UpdateMergeRequestsWorker`
is enqueued and `MergeRequests::RefreshService` is called, it'll
check if the source branch of each MR asssociated to the push exists
or not via `MergeRequest#source_branch_exists?`. The said method will
call `Repository#branch_exists?` which is cached in `Rails.cache`.
When the cache contains outdated data and the source branch actually
exists, the `MergeRequests#RefreshService` job will close associated
MRs which is not correct.
The fix is to expire the branches cache of the project so we have
updated data during the post receive hook which will help in the
accuracy of the check if we need to close associated MRs or not.
This will help identify Sidekiq jobs that invoke excessive number of
filesystem access.
The timing data is stored in `RequestStore`, but this is only active
within the middleware and is not directly accessible to the Sidekiq
logger. However, it is possible for the middleware to modify the job
hash to pass this data along to the logger.
These counts significantly increase the load time for these
requests. Users can now opt in to receiving the counts by setting
`with_counts=true` in requests. This is a breaking change, but hopefully
a fairly minor one.
EE added a single line to this file that can easily be backported to CE,
but it appears this was overlooked. The method called is a noop in CE,
so no additional changes are necessary.
Add possibilty to view starrers ("stargazers") of a repository & any user's starred repositories
Closes#20137
See merge request gitlab-org/gitlab-ce!24690
https://gitlab.com/gitlab-org/gitlab-ce/issues/62971
Adds support to EnvironmentsController#metrics_dashboard
for the following params: group, title, y_label
These params are used to uniquely identify a panel on
the metrics dashboard.
Metrics are stored in several places, so this adds
utilities to find a specific panel from the database
or filesystem depending on the metric specified.
Also moves some shared utilities into separate classes,
notably default values and errors.
Stan Hu
John Cai
Patrick Derichs
Lin Jen-Shin
Kamil Trzciński
Jan Provaznik
Nick Thomas
Alexandru Croitor
Marius Bobin
Tiger
Thong Kuah
Alessio Caiazza
Balakumar
Sean McGivern
Kushal Pandya
Paul Okstad
Oswaldo Ferreira
Fabio Pitino
Luke Duncalfe
drew
Douwe Maan
Diego Louzán
Andrew Newdigate
Peter Leitzen
Cameron Swords
Alex Kalderimis
Robert Speicher
Felipe Artur
Aleksei Lipniagov
Michael Kozono
rossfuhrman
Bob Van Landuyt
Jarka Košanová
Sarah Yasonik
Hordur Freyr Yngvason
Mayra Cabrera
Brett Walker
Douglas Barbosa Alexandre
Alex Kalderimis
George Koltsov
João Cunha
Jeremy Jackson
Adam Hegyi
Grzegorz Bizon
Igor
Bob Van Landuyt
GitalyBot
Heinrich Lee Yu
Markus Koller
Annabel Dunstone Gray
Filipa Lacerda
Jacob Vosmaer
Armin Hohenegger
Tristan Read
Rémy Coutable
Patrick Bajao
James Lopez
Cindy Pallares 🦉
Mark Chao
Yorick Peterse
Philippe Lafoucrière
Camil Staps
Reuben Pereira
Kerri Miller
Zeger-Jan van de Weg