* upstream/master: (292 commits)
Deletes extra empty line breaking the build
Optimize the `award_user_list` helper spec
Fix typo and add he MWBS accronym for "Merge When Build Succeeds"
Added missing content and improved layout
ExpireBuildArtifactsWorker query builds table without ordering enqueuing one job per build to cleanup
Improve the contribution and MR review guide
Updates test in order to look for link
Make projects API docs match parameter style
Fix Event#reset_project_activity updates
Update user whitelist reject message
Call ensure_secret_token! in secret token test's before block since it would be called in an initializer.
Add a CHANGELOG for CacheMarkdownField
Enable CacheMarkdownField for the remaining models
Make search results use the markdown cache columns, treating them consistently
Use CacheMarkdownField for notes
Add markdown cache columns to the database, but don't use them yet
Update issue board spec
Link to Registry docs from project settings
Truncate long labels with ellipsis in labels page
Improve issue load time performance by avoiding ORDER BY in find_by call
...
Memoize Github::Shell's secret token
## What does this MR do?
`API::Helpers#secret_token` was reading the secret file on every invocation. This MR reads the file in the `gitlab_shell_secret_token.rb` initializer and saves it as a class variable at `Gitlab::Shell.secret_token`
## Are there points in the code the reviewer needs to double check?
- I'm not sure if the use of `cattr_accessor` is the best approach, or if should be moved into the `class << self` block?
- Should `API::Helpers#secret_token` be removed in favor of using `Gitlab::Shell.secret_token`?
## Why was this MR needed?
Performance optimization.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/22510
See merge request !6599
This commit adds a number of _html columns and, with the exception of Note,
starts updating them whenever the content of their partner fields changes.
Note has a collision with the note_html attr_accessor; that will be fixed later
A background worker for clearing these cache columns is also introduced - use
`rake cache:clear` to set it off. You can clear the database or Redis caches
separately by running `rake cache:clear:db` or `rake cache:clear:redis`,
respectively.
Improve asynchronous pipeline processing
## What does this MR do?
This MR improves asynchronous processing of pipeline.
## Why was this MR needed?
It eliminates some race conditions and improves performance.
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
## What are the relevant issue / merge request numbers?
Related merge request: !6410
Extracted from !6411
See merge request !6650
Refactor Gitlab::Identifier
## What does this MR do?
This refactors `Gitlab::Identifier` so that it:
1. Has tests
2. Caches output in an instance variable to reduce queries
3. Uses only a single query to find a user by an SSH key, instead of 2
## Why was this MR needed?
This code was untested and would execute more SQL queries than needed.
See merge request !6680
This refactors Gitlab::Identifier so it uses fewer queries and is
actually tested. Queries are reduced by caching the output as well as
using 1 query (instead of 2) to find a user using an SSH key.
* upstream/master: (372 commits)
Enable Lint/StringConversionInInterpolation cop and autocorrect offenses
resolve duplicated changelog entry
credit myself 😄
change determine conditions
override subject method in devise mailer
follow the styleguide: Don't use parentheses around a literal
wrap subject with method subject
move spec back into shared example `an email sent from GitLab`
stub config settings in spec
remove empty line at block body end
remove extra entry
create new test in `spec/mailers/notify_spec.rb`
move changelog to 8.13
add configurable email subject suffix
Fixes sidebar navigation.
Convert "SSH Keys" Spinach features to RSpec
Enable import/export back for non-admins
Update gitlab-shell to 3.6.3
Updated artwork of empty group state.
Better empty state for Groups view.
...
Resolve "`Member.add_user`doesn't detect existing members that have requested access"
## What does this MR do?
This merge request handle the case when an access requester is added to a group or project (via the members page or the API).
In `Member.add_user`, if an access requester already exists, we simply accept their request (and set the `created_by`, `access_level` and `expires_at` attributes if given).
## Are there points in the code the reviewer needs to double check?
I've taken the opportunity to cleanup the whole `{Group,Project}Member.add_user*` methods since it was quite a mess.
## What are the relevant issue numbers?
Closes#21983
See merge request !6393
Prevent claiming associated model IDs via import
On the import side, we should be careful not to use any IDs as part of the JSON file that could have been manipulated.
Part of https://gitlab.com/gitlab-org/gitlab-ce/issues/20821
Things we already do (__before__ this fix):
1. Remove all primary keys
1. **Always** reassign some of the foreign keys, such as ALL project IDs and user IDs (so it would be difficult to impersonate or try to gain access to another project)
1. Ignore/reject attributes that do not exist in the model
1. If someone reassigns a foreign key `submodel_id`, and that object has another json as the submodel, the new submodel will reassign the `submodel_id` to the newly created submodel ID.
Things we should do:
1. Remove/nullify any other foreign keys that we don't reassign (checked this, and there aren't many, fortunately. In fact, I don't think much harm can be done at all - at the moment).
See merge request !1985
Do not regenerate the `lfs_token` every time `git-lfs-authenticate` is called
## What does this MR do?
Do not regenerate the `lfs_token` every time `git-lfs-authenticate` is called, instead return the saved token if one is present.
This was causing a lot of 401s, leading to 403s, as state in #22527
As it turns out, when pushing a lot of LFS objects, the LFS client was calling `git-lfs-authenticate` in the middle of the request again. This caused the `lfs_token` to be regenerated. The problem lies in that the LFS client was not aware of this change, and was still using the old token. This caused all subsequent requests to fail with a 401 error.
Since HTTP Auth is protected by Rack Attack, this 401s where immediately flagged and resulted in the IP of the user being banned.
With this change, GitLab returns the value stored in Redis, if one is present, thus if the LFS client calls `git-lfs-authenticate` again during the request, the auth header will remain unchanged, allowing all subsequent requests to continue without issues.
## What are the relevant issue numbers?
Fixes#22527
cc @SeanPackham @jacobvosmaer-gitlab
See merge request !6551
Changes include:
- Ensure Member.add_user is not called directly when not necessary
- New GroupMember.add_users_to_group to have the same abstraction level as for Project
- Refactor Member.add_user to take a source instead of an array of members
- Fix Rubocop offenses
- Always use Project#add_user instead of project.team.add_user
- Factorize users addition as members in Member.add_users_to_source
- Make access_level a keyword argument in GroupMember.add_users_to_group and ProjectMember.add_users_to_projects
- Destroy any requester before adding them as a member
- Improve the way we handle access requesters in Member.add_user
Instead of removing the requester and creating a new member,
we now simply accepts their access request. This way, they will
receive a "access request granted" email.
- Fix error that was previously silently ignored
- Stop raising when access level is invalid in Member, let Rails validation do their work
Signed-off-by: Rémy Coutable <remy@rymai.me>
* No need to re-fetch issues from GH to read their labels, the labels
are already there from the index request.
* No need to look up labels on the database for every application, so we
cache them.
Use base SHA for patches and diffs
## What does this MR do?
Switch from using 'start SHA' to 'base SHA' for patches and diffs
## Are there points in the code the reviewer needs to double check?
## Why was this MR needed?
Makes the downloaded patches and diffs on the merge request page match the frontend-rendered "changes" in these scenarios:
* Unpatched gitlab-workhorse, downloading patchsets of open MRs (https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/68)
* Unpatched gitlab-workhorse, downloading diffs of open and merged MRs
* Patched gitlab-workhorse, downloading patchsets of merged merge requests
## What are the relevant issue numbers?
Closes#22229
See merge request !6435
This commit changes the revisions used for diffs. The current behaviour is
to show all changes between current tip of master and tip of the MR, rather
than matching the output of the web frontend (which just shows the changes
in the MR). Switching from start_sha to base_sha fixes this.
* upstream/master: (206 commits)
Implement fourth round of comments from @DouweM.
Fix `CreateDeploymentService` spec.
Reload issues in spec to ensure label<->issue mapping properly loaded
Fix build.
Remove unnecessary #{} in cycle analytics template.
Update cycle analytics icon and fix color of the dismiss button.
Use triple dashes for the empty value in cycle analytics.
Fix typo on cycle analytics copy.
Add page title and fix sub menu width in Cycle Analytics page.
Update Cycle Analytics Read more link URL.
Display the cycle analytics navbar based on the `:read_cycle_analytics` ability.
Improve indentation in `Gitlab::Database::Median`
Add a spec for merge request metric caching while refreshing a merge request from a forked project.
Use the `IssuableBaseService` lifecycle hooks to cache `MergeRequestsClosingIssues`
Implement a second round of review comments from @DouweM.
Add docs on Cycle Analytics
Test if issue authors can access private projects
Update .pkgr.yml with Ubuntu 16.04 dependencies
fix issues mr counter
Move JSON generation (cycle analytics) into a controller method.
...
Better import of labels, milestones and protected branches. Updated relevant specs.
Loose pipeline validation on importing, so it does not fail when there are missing fields, which are not validated at DB level. Also, updated spec with relevant test.
Only add original author tag line when importing from GitHub if there isn't a linked GitLab account
## What does this MR do?
If there we've found a linked GitLab user for a creator of an issue or comment don't add the 'Created By:' line.
## What are the relevant issue numbers?
Closes#21569
See merge request !6081
Restrict ldap return attributes
## What does this MR do?
Fixes the CE part of #13821. We really only ever need uid, dn, cn, and mail attributes, and in some cases, even less. This merge request strips the request down to those four attributes by default, and allows the caller to specify others, if needed.
## Why was this MR needed?
This will improve performance especially in cases where the connection is slow between GitLab and LDAP, or when the LDAP object has lots of attributes we don't care about.
See merge request !6187
Security and safety improvements for gitlab-workhorse integration
Companion to https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/60
- Use a custom content type when sending data to gitlab-workhorse
- Verify (using JWT and a shared secret on disk) that internal API requests came from gitlab-workhorse
This will allow us to build features in gitlab-workhorse that require
more trust, and protect us against programming mistakes in the future.
This is designed so that no action is required for installations from
source. For omnibus-gitlab we need to add code that manages the shared
secret.
See merge request !5907
Smartly calculate real running time and pending time
## What does this MR do?
Try to smartly calculate the running time and pending time for pipelines, instead of just use wall clock time from start to end. The algorithm is based on:
> Suppose we have A, B, and C jobs:
> * A: from 1 to 3
> * B: from 2 to 4
> * C: from 6 to 7
> The processing time should be accumulated from 1 to 4, and 6 to 7, totally 4, excluding retires, and calculate on `%w[success failed running canceled]` jobs (if a job is not finished yet, assume it's `Time.now`)
## Are there points in the code the reviewer needs to double check?
I would actually like to test `Gitlab::Ci::PipelineDuration#process_segments`, but it's a private method right now and it's not very convenient to test it. Is there a way to test it without changing the original code too much? Note that I would like to avoid saving merged segments because it's not used and should be garbage collected.
## Screenshots:
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- Tests
- [x] Added for this feature/bug
## What are the relevant issue numbers?
Closes#18260, #19804
See merge request !6084
Pass dependencies to CI configuration nodes
## What does this MR do?
This MR makes it possible to pass dependencies to CI configuration nodes.
## What are the relevant issue numbers?
See #15060
## Does this MR meet the acceptance criteria?
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
See merge request !6009
Sean McGivern
tiagonbotelho
Douglas Barbosa Alexandre
Lin Jen-Shin
Rémy Coutable
Justin DiPierro
Nick Thomas
Paco Guzman
Kamil Trzciński
Yorick Peterse
Grzegorz Bizon
Robert Speicher
Robert Speicher
Douwe Maan
Patricio Cano
Markus Koller
Ahmad Sherif
James Lopez
Jacob Vosmaer
Dmitriy Zaporozhets
Dan Dunckel
Drew Blessing
Andrew Smith
Sean McGivern