When processing push payloads we now schedule at most the 100 most
recent commits, instead of all commits that were in a payload. This
prevents one from overloading the system by pushing thousands if not
millions of commits in a single go.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/25827
This adds counters for build artifacts and LFS objects, and moves
the preexisting repository_size and commit_count from the projects
table into a new project_statistics table.
The counters are displayed in the administration area for projects
and groups, and also available through the API for admins (on */all)
and normal users (on */owned)
The statistics are updated through ProjectCacheWorker, which can now
do more granular updates with the new :statistics argument.
By passing commit data to this worker we remove the need for querying
the Git repository for every job. This in turn reduces the time spent
processing each job.
The migration included migrates jobs from the old format to the new
format. For this to work properly it requires downtime as otherwise
workers may start producing errors until they're using a newer version
of the worker code.
This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.
To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.
In this new setup caches are refreshed as follows:
1. After a commit (but before running ProjectCacheWorker) we expire some
basic caches such as the commit count and repository size.
2. ProjectCacheWorker will recalculate the commit count, repository
size, then refresh a specific set of caches based on the list of
files changed in a push payload.
This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.
With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.
This moves the code used for processing commits from GitPushService to
its own Sidekiq worker: ProcessCommitWorker.
Using a Sidekiq worker allows us to process multiple commits in
parallel. This in turn will lead to issues being closed faster and cross
references being created faster. Furthermore by isolating this code into
a separate class it's easier to test and maintain the code.
The new worker also ensures it can efficiently check which issues can be
closed, without having to run numerous SQL queries for every issue.
1. Change multiple updates to a single `update_all`
2. Use cascading deletes
3. Extract an average function for the database median.
4. Move database median to `lib/gitlab/database`
5. Use `delete_all` instead of `destroy_all`
6. Minor refactoring
All the code that pre-calculates metrics for use in the cycle analytics
page.
- Ci::Pipeline -> build start/finish
- Ci::Pipeline#merge_requests
- Issue -> record default metrics after save
- MergeRequest -> record default metrics after save
- Deployment -> Update "first_deployed_to_production_at" for MR metrics
- Git Push -> Update "first commit mention" for issue metrics
- Merge request create/update/refresh -> Update "merge requests closing issues"
A customer ran into an issue where a Sidekiq task retried over and over, leading to duplicate
master branches in their protected branch list.
Closes#22177
1. These changes bring down page load time for 100 issues from more than
a minute to about 1.5 seconds.
2. This entire commit is composed of these types of performance
enhancements:
- Cache relevant data in `IssueMetrics` wherever possible.
- Cache relevant data in `MergeRequestMetrics` wherever possible.
- Preload metrics
3. Given these improvements, we now only need to make 4 SQL calls:
- Load all issues
- Load all merge requests
- Load all metrics for the issues
- Load all metrics for the merge requests
4. A list of all the data points that are now being pre-calculated:
a. The first time an issue is mentioned in a commit
- In `GitPushService`, find all issues mentioned by the given commit
using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at`
flag for each of them.
- There seems to be a (pre-existing) bug here - files (and
therefore commits) created using the Web CI don't have
cross-references created, and issues are not closed even when
the commit title is "Fixes #xx".
b. The first time a merge request is deployed to production
When a `Deployment` is created, find all merge requests that
were merged in before the deployment, and set the
`first_deployed_to_production_at` flag for each of them.
c. The start / end time for a merge request pipeline
Hook into the `Pipeline` state machine. When the `status` moves to
`running`, find the merge requests whose tip commit matches the
pipeline, and record the `latest_build_started_at` time for each
of them. When the `status` moves to `success`, record the
`latest_build_finished_at` time.
d. The merge requests that close an issue
- This was a big cause of the performance problems we were having
with Cycle Analytics. We need to use `ReferenceExtractor` to make
this calculation, which is slow when we have to run it on a large
number of merge requests.
- When a merge request is created, updated, or refreshed, find the
issues it closes, and create an instance of
`MergeRequestsClosingIssues`, which acts as a join model between
merge requests and issues.
- If a `MergeRequestsClosingIssues` instance links a merge request
and an issue, that issue closes that merge request.
5. The `Queries` module was changed into a class, so we can cache the
results of `issues` and `merge_requests_closing_issues` across
various cycle analytics stages.
6. The code added in this commit is untested. Tests will be added in the
next commit.
1. `GitPushService` was still using `{merge,push}_access_level_attributes` instead
of `{merge,push}_access_levels_attributes`.
2. The branches API creates access levels regardless of the state of the
`developers_can_{push,merge}` parameters. This is in line with the UI,
where Master access is the default for a new protected branch.
3. Use `after(:build)` to create access levels in the
`protected_branches` factory, so that `factories_spec` passes. It
only builds records, so we need to create access levels on `build` as
well.
This change simplifies a Pipeline processing by introducing a special new status: created.
This status is used for all builds that are created for a pipeline.
We are then processing next stages and queueing some of the builds (created -> pending) or skipping them (created -> skipped).
This makes it possible to simplify and solve a few ordering problems with how previously builds were scheduled.
This also allows us to visualise a full pipeline (with created builds).
This also removes an after_touch used for updating a pipeline state parameters.
Right now in various places we explicitly call a reload_status! on pipeline to force it to be updated and saved.
1. It makes sense to reuse these constants since we had them duplicated
in the previous enum implementation. This also simplifies our
`check_access` implementation, because we can use
`project.team.max_member_access` directly.
2. Use `accepts_nested_attributes_for` to create push/merge access
levels. This was a bit fiddly to set up, but this simplifies our code
by quite a large amount. We can even get rid of
`ProtectedBranches::BaseService`.
3. Move API handling back into the API (previously in
`ProtectedBranches::BaseService#translate_api_params`.
4. The protected branch services now return a `ProtectedBranch` rather
than `true/false`.
5. Run `load_protected_branches` on-demand in the `create` action, to
prevent it being called unneccessarily.
6. "Masters" is pre-selected as the default option for "Allowed to Push"
and "Allowed to Merge".
7. These changes were based on a review from @rymai in !5081.
This is a combination of 3 commits.
- Update the bare repositories info/attributes if the default branch is updated
- Check the diff attributes of a file before showing a diff
- Update CHANGELOG
* master: (121 commits)
Dedupe labels in labels selector in Dashboard pages
Refactor colors and lists
Add a safeguard in MergeRequest#compute_diverged_commits_count
Fix an issue when the target branch of a MR had been deleted
Add avatar to issue and MR pages header
Cleanup somce css colors
Re-group scss variables
Refactor `Todo#target`
Fixes issue with filter label missing on labels & milestones
Rename `Todo#to_reference` to `Todo#target_reference`
Fixed failing tests
Updated controller with before_action Fixed other issues based on feedback
Fixes issue on dashboard issues
Full labels data in JSON
Fixed issue with labels dropdown getting wrong labels
Update CHANGELOG
Use `Note#for_project_snippet?` to skip notes on project snippet
Use `Commit#short_id` instead of `Commit.truncate_sha`
Reuse `for_commit?` on conditional validations
Update schema info comment on todo related files
...
Conflicts:
app/models/issue.rb
db/schema.rb
spec/models/issue_spec.rb
The avatar logic has been moved from Project to Repository as this makes
caching easier. The logic itself in turn has been changed so that the
logo file names are cached in Redis. This cache is flushed upon pushing
a commit but _only_ if:
1. The commit was pushed to the default branch
2. The commit actually changes any of the logo files
If no branch or commit is given the cache is flushed anyway, this
ensures that calling Repository#expire_cache without any arguments still
flushes the avatar cache (e.g. this is used when removing a project).
Fixesgitlab-org/gitlab-ce#14363
James Edwards-Jones
gpongelli
Sean McGivern
Yorick Peterse
Markus Koller
Grzegorz Bizon
Paco Guzman
Timothy Andrew
Stan Hu
Kamil Trzcinski
Lin Jen-Shin
Rémy Coutable
Robert Speicher
Mathias Vestergaard
Josh Frye
Matt Oakes
Gabriel Mazetto
Drew Blessing
connorshea
Jacob Vosmaer
ashleys
Dmitriy Zaporozhets
tiagonbotelho
Valery Sizov
James Lopez