When processing push payloads we now schedule at most the 100 most
recent commits, instead of all commits that were in a payload. This
prevents one from overloading the system by pushing thousands if not
millions of commits in a single go.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/25827
This adds counters for build artifacts and LFS objects, and moves
the preexisting repository_size and commit_count from the projects
table into a new project_statistics table.
The counters are displayed in the administration area for projects
and groups, and also available through the API for admins (on */all)
and normal users (on */owned)
The statistics are updated through ProjectCacheWorker, which can now
do more granular updates with the new :statistics argument.
By passing commit data to this worker we remove the need for querying
the Git repository for every job. This in turn reduces the time spent
processing each job.
The migration included migrates jobs from the old format to the new
format. For this to work properly it requires downtime as otherwise
workers may start producing errors until they're using a newer version
of the worker code.
This refactors repository caching so it's possible to selectively
refresh certain caches, instead of just expiring and refreshing
everything.
To allow this the various methods that were cached (e.g. "tag_count" and
"readme") use a similar pattern that makes expiring and refreshing
their data much easier.
In this new setup caches are refreshed as follows:
1. After a commit (but before running ProjectCacheWorker) we expire some
basic caches such as the commit count and repository size.
2. ProjectCacheWorker will recalculate the commit count, repository
size, then refresh a specific set of caches based on the list of
files changed in a push payload.
This requires a bunch of changes to the various methods that may be
cached. For one, data should not be cached if a branch used or the
entire repository does not exist. To prevent all these methods from
handling this manually this is taken care of in
Repository#cache_method_output. Some methods still manually check for
the existence of a repository but this result is also cached.
With selective flushing implemented ProjectCacheWorker no longer uses an
exclusive lease for all of its work. Instead this worker only uses a
lease to limit the number of times the repository size is updated as
this is a fairly expensive operation.
Use `Gitlab.config.gitlab.host` over `'localhost'`
Use `Gitlab.config.gitlab.host` over `'localhost'`
This would fix long standing failures running tests on
my development machine, which set `Gitlab.config.gitlab.host`
to another host because it's not my local computer. Now I
finally cannot withstand it and decided to fix them once and
for all.
See merge request !7562
This would fix long standing failures running tests on
my development machine, which set `Gitlab.config.gitlab.host`
to another host because it's not my local computer. Now I
finally cannot withstand it and decided to fix them once and
for all.
This moves the code used for processing commits from GitPushService to
its own Sidekiq worker: ProcessCommitWorker.
Using a Sidekiq worker allows us to process multiple commits in
parallel. This in turn will lead to issues being closed faster and cross
references being created faster. Furthermore by isolating this code into
a separate class it's easier to test and maintain the code.
The new worker also ensures it can efficiently check which issues can be
closed, without having to run numerous SQL queries for every issue.
These specs use raw Redis objects which can not use the memory based
caching mechanism used for tests. As such we have to explicitly flush
the data from Redis before/after each spec to ensure no data lingers on.
The amount of precision times have in databases is variable, so we need
tolerances when comparing in specs. It's better to have the tolerance defined
in one place than several.
This commit adds a number of _html columns and, with the exception of Note,
starts updating them whenever the content of their partner fields changes.
Note has a collision with the note_html attr_accessor; that will be fixed later
A background worker for clearing these cache columns is also introduced - use
`rake cache:clear` to set it off. You can clear the database or Redis caches
separately by running `rake cache:clear:db` or `rake cache:clear:redis`,
respectively.
All the code that pre-calculates metrics for use in the cycle analytics
page.
- Ci::Pipeline -> build start/finish
- Ci::Pipeline#merge_requests
- Issue -> record default metrics after save
- MergeRequest -> record default metrics after save
- Deployment -> Update "first_deployed_to_production_at" for MR metrics
- Git Push -> Update "first commit mention" for issue metrics
- Merge request create/update/refresh -> Update "merge requests closing issues"
A customer ran into an issue where a Sidekiq task retried over and over, leading to duplicate
master branches in their protected branch list.
Closes#22177
1. Change a few incorrect `access_level` to `access_levels.first` that
were missed in e805a64.
2. `API::Entities` can iterate over all access levels instead of just
the first one. This makes no difference to CE, and makes it more compatible
with EE.
!581 has a lot of changes that would cause merge conflicts if not
properly backported to CE. This commit/MR serves as a better
foundation for gitlab-org/gitlab-ee!581.
= Changes =
1. Move from `has_one {merge,push}_access_level` to `has_many`, with the
`length` of the association limited to `1`. This is _effectively_ a
`has_one` association, but should cause less conflicts with EE, which
is set to `has_many`. This has a number of related changes in the
views, specs, and factories.
2. Make `gon` variable loading more consistent (with EE!581) in the
`ProtectedBranchesController`. Also use `::` to prefix the
`ProtectedBranches` services, because this is required in EE.
3. Extract a `ProtectedBranchAccess` concern from the two access level
models. This concern only has a single `humanize` method here, but
will have more methods in EE.
4. Add `form_errors` to the protected branches creation form. This is
not strictly required for EE compatibility, but was an oversight
nonetheless.
1. It makes sense to reuse these constants since we had them duplicated
in the previous enum implementation. This also simplifies our
`check_access` implementation, because we can use
`project.team.max_member_access` directly.
2. Use `accepts_nested_attributes_for` to create push/merge access
levels. This was a bit fiddly to set up, but this simplifies our code
by quite a large amount. We can even get rid of
`ProtectedBranches::BaseService`.
3. Move API handling back into the API (previously in
`ProtectedBranches::BaseService#translate_api_params`.
4. The protected branch services now return a `ProtectedBranch` rather
than `true/false`.
5. Run `load_protected_branches` on-demand in the `create` action, to
prevent it being called unneccessarily.
6. "Masters" is pre-selected as the default option for "Allowed to Push"
and "Allowed to Merge".
7. These changes were based on a review from @rymai in !5081.
1. Caused by incorrect test setup. The user wasn't added to the project,
so protected branch creation failed authorization.
2. Change setup for a different test (`Event.last` to
`Event.find_by_action`) because our `project.team << ...` addition
was causing a conflict.
Handle external issues in IssueReferenceFilter
Rendering issue references such as `#1` was broken for projects using an external issues tracker.
See gitlab-org/gitlab-ce#19036
See merge request !4988
When pushing commits to existing branches we don’t
need to flush branch git data (branch names / counts)
When flushes the cache when pushing commits skip to
flush branch and tag git data (names / counts) because
those operations are managed explicitly in each case
Repopulated expired cache as soon as possible
This reduces the number of queries executed in IssueReferenceFilter by
retrieving the various projects/issues that may be referenced in batches
_before_ iterating over all the HTML nodes.
A chunk of the logic resides in AbstractReferenceFilter so it can be
re-used by other filters in the future.
This is a combination of 3 commits.
- Update the bare repositories info/attributes if the default branch is updated
- Check the diff attributes of a file before showing a diff
- Update CHANGELOG
The avatar logic has been moved from Project to Repository as this makes
caching easier. The logic itself in turn has been changed so that the
logo file names are cached in Redis. This cache is flushed upon pushing
a commit but _only_ if:
1. The commit was pushed to the default branch
2. The commit actually changes any of the logo files
If no branch or commit is given the cache is flushed anyway, this
ensures that calling Repository#expire_cache without any arguments still
flushes the avatar cache (e.g. this is used when removing a project).
Fixesgitlab-org/gitlab-ce#14363
Kamil Trzcinski
Jarka Kadlecova
Sean McGivern
Jacob Vosmaer
James Edwards-Jones
Robert Speicher
gpongelli
Yorick Peterse
Markus Koller
Rémy Coutable
Lin Jen-Shin
Felipe Artur
Nick Thomas
Paco Guzman
Timothy Andrew
Stan Hu
tiagonbotelho
Mathias Vestergaard
Grzegorz Bizon
Robert Speicher
Josh Frye
Matt Oakes
Drew Blessing
Douglas Barbosa Alexandre
Jacob Vosmaer