Allow certain Sidekiq jobs to be throttled
## What does this MR do?
Allows certain slow running Sidekiq jobs to be throttled. It is disabled by default and can be enabled via the Application Settings.
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG.md) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
## What are the relevant issue numbers?
Related to #23352
See merge request !7292
Respect project visibility settings in the contributions calendar
This MR fixes a number of bugs relating to access controls and date selection of events for the contributions calendar
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/23403
See merge request !2019
Signed-off-by: Rémy Coutable <remy@rymai.me>
It was previously possible for invalid credential errors to go unnoticed
in this task. Users would believe everything was configured correctly and
then sign in would fail with 'invalid credentials'. This adds a specific
bind check, plus catches errors connecting to the server. Also, specs :)
No matter which environment Gitlab was running as, the admin/logs view
always showed production.log. This commit selects the logfile based
on Rails.env.
- Rename ProductionLogger to EnvironmentLogger
- Make EnvironmentLogger logfile depend on env
- Update spinach test for log tabs
When we updated gitlab_git to 10.4.1, `tag.target` changed from pointing
to the sha of the tag to the sha of the commit the tag points to. The
problem is that only annotated tags have `object_sha`s, lightweight tags
don't (it's nil), so (only) in their case we still need to use
`tag.target`.
Pass user instance to Labels::FindOrCreateService or skip_authorization: true
## What does this MR do?
It fixes a bug described in #23694 when `project.owner` was passed to `Labels::FindOrCreateService`. `Labels::FindOrCreateService` expected a user instance and `project.owner` may return a group as well. This MR makes sure that we either pass a user instance or `skip_authorization: true`.
## Are there points in the code the reviewer needs to double check?
- places where we pass `skip_authorization: true`
## Does this MR meet the acceptance criteria?
- Tests
- [x] Added for this feature/bug
- [ ] All builds are passing
- [ ] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if it does - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
## What are the relevant issue numbers?
Fixes#23694
See merge request !7093
Use optimistic locking
## What does this MR do?
Removes the usage of pessimistic locking in favor of optimistic which is way cheaper and doesn't block database operation.
Since this is very simple change it should be safe. If we receive `StaleObjectError` message we will reload object a retry operations in lock.
However, I still believe that we need this one: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7005 as this will reduce a load on Database and FS.
This changes a behavior from:
### Pesimistic locking (previous behavior)
#### For updating
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. enqueue: (use: transition :created -> :pending)
5. [state_machine] we are in state created, we can go to pending
6. [state_machine] ci_pipeline.status = created
7. [state_machine] ci_pipeline.save
8. [state_machine] after_transition: (if for success): PipelineSuccessWorker on Sidekiq
9. release DB lock
#### If no update is required
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. we are in pending, we can't transition to pending, because it's forbidden
5. release DB lock
### Optimistic locking (implemented by this MR)
#### For updating
1. latest_build_status
2. enqueue: (use `transition :created -> :pending`)
3. [state_machine] we are in state created, we can go to pending
4. [state_machine] ci_pipeline.status = created
5. [state_machine] ci_pipeline.save
6. [state_machine] [save] where(lock_version: ci_pipeline.lock_version).update_all(status: :created, updated_at: Time.now)
7. [state_machine] [save] unless we_updated_row then raise ObjectInconsistentError
#### If no update is required
1. we update ci_pipeline
2. latest_build_status
3. we are in pending, we can't transition to pending, because it's forbidden
## Why was this MR needed?
We have been seeing a number of problems when we migrated Pipeline/Build processing to Sidekiq. Especially we started seeing a lot of blocking queries.
We used a pessimistic locking which doesn't seem to be required. This effectively allows us to fix our issues with blocked queries by using more efficient method of operation.
## What are the relevant issue numbers?
Issues: https://gitlab.com/gitlab-com/infrastructure/issues/623 and https://gitlab.com/gitlab-com/infrastructure/issues/584, but also there's a bunch of Merge Requests that try to improve behavior of scheduled jobs.
cc @pcarranza @yorickpeterse @stanhu
See merge request !7040
This changes ProjectCacheWorker.perform_async so it only schedules a job
when no lease for the given project is present. This ensures we don't
end up scheduling hundreds of jobs when they won't be executed anyway.
Douwe Maan
Patricio Cano
Sean McGivern
Adam Niedzielski
Rémy Coutable
Robert Speicher
Drew Blessing
Valery Sizov
Dmitriy Zaporozhets
tiagonbotelho
Douwe Maan
Ahmad Sherif
Jacob Vosmaer
Lukas Erlacher
Felipe Artur
Alejandro Rodríguez
Stan Hu
Kamil Trzcinski
Gabriel Mazetto
Yorick Peterse