Fix broken handling of certain calls in GitHub importer client
## What does this MR do?
It changes/fixes the behavior of request handling in GH client. Now it returns the response directly if it's not a collection of resources. Otherwise, it checks for a passed block, if true, then it yield each page to said block, if not, it collects all response in a single array then returns it.
Closes#22998
See merge request !6703
Refactor Gitlab::Identifier
## What does this MR do?
This refactors `Gitlab::Identifier` so that it:
1. Has tests
2. Caches output in an instance variable to reduce queries
3. Uses only a single query to find a user by an SSH key, instead of 2
## Why was this MR needed?
This code was untested and would execute more SQL queries than needed.
See merge request !6680
This refactors Gitlab::Identifier so it uses fewer queries and is
actually tested. Queries are reduced by caching the output as well as
using 1 query (instead of 2) to find a user using an SSH key.
Log sidekiq arguments as json
Logging the sidekiq job arguments as a ruby literal structure
is akward. Specially when parsing the logs.
JSON is a standar format.
See merge request !3735
Resolve "`Member.add_user`doesn't detect existing members that have requested access"
## What does this MR do?
This merge request handle the case when an access requester is added to a group or project (via the members page or the API).
In `Member.add_user`, if an access requester already exists, we simply accept their request (and set the `created_by`, `access_level` and `expires_at` attributes if given).
## Are there points in the code the reviewer needs to double check?
I've taken the opportunity to cleanup the whole `{Group,Project}Member.add_user*` methods since it was quite a mess.
## What are the relevant issue numbers?
Closes#21983
See merge request !6393
Prevent claiming associated model IDs via import
On the import side, we should be careful not to use any IDs as part of the JSON file that could have been manipulated.
Part of https://gitlab.com/gitlab-org/gitlab-ce/issues/20821
Things we already do (__before__ this fix):
1. Remove all primary keys
1. **Always** reassign some of the foreign keys, such as ALL project IDs and user IDs (so it would be difficult to impersonate or try to gain access to another project)
1. Ignore/reject attributes that do not exist in the model
1. If someone reassigns a foreign key `submodel_id`, and that object has another json as the submodel, the new submodel will reassign the `submodel_id` to the newly created submodel ID.
Things we should do:
1. Remove/nullify any other foreign keys that we don't reassign (checked this, and there aren't many, fortunately. In fact, I don't think much harm can be done at all - at the moment).
See merge request !1985
Do not regenerate the `lfs_token` every time `git-lfs-authenticate` is called
## What does this MR do?
Do not regenerate the `lfs_token` every time `git-lfs-authenticate` is called, instead return the saved token if one is present.
This was causing a lot of 401s, leading to 403s, as state in #22527
As it turns out, when pushing a lot of LFS objects, the LFS client was calling `git-lfs-authenticate` in the middle of the request again. This caused the `lfs_token` to be regenerated. The problem lies in that the LFS client was not aware of this change, and was still using the old token. This caused all subsequent requests to fail with a 401 error.
Since HTTP Auth is protected by Rack Attack, this 401s where immediately flagged and resulted in the IP of the user being banned.
With this change, GitLab returns the value stored in Redis, if one is present, thus if the LFS client calls `git-lfs-authenticate` again during the request, the auth header will remain unchanged, allowing all subsequent requests to continue without issues.
## What are the relevant issue numbers?
Fixes#22527
cc @SeanPackham @jacobvosmaer-gitlab
See merge request !6551
Changes include:
- Ensure Member.add_user is not called directly when not necessary
- New GroupMember.add_users_to_group to have the same abstraction level as for Project
- Refactor Member.add_user to take a source instead of an array of members
- Fix Rubocop offenses
- Always use Project#add_user instead of project.team.add_user
- Factorize users addition as members in Member.add_users_to_source
- Make access_level a keyword argument in GroupMember.add_users_to_group and ProjectMember.add_users_to_projects
- Destroy any requester before adding them as a member
- Improve the way we handle access requesters in Member.add_user
Instead of removing the requester and creating a new member,
we now simply accepts their access request. This way, they will
receive a "access request granted" email.
- Fix error that was previously silently ignored
- Stop raising when access level is invalid in Member, let Rails validation do their work
Signed-off-by: Rémy Coutable <remy@rymai.me>
* No need to re-fetch issues from GH to read their labels, the labels
are already there from the index request.
* No need to look up labels on the database for every application, so we
cache them.
Use base SHA for patches and diffs
## What does this MR do?
Switch from using 'start SHA' to 'base SHA' for patches and diffs
## Are there points in the code the reviewer needs to double check?
## Why was this MR needed?
Makes the downloaded patches and diffs on the merge request page match the frontend-rendered "changes" in these scenarios:
* Unpatched gitlab-workhorse, downloading patchsets of open MRs (https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/68)
* Unpatched gitlab-workhorse, downloading diffs of open and merged MRs
* Patched gitlab-workhorse, downloading patchsets of merged merge requests
## What are the relevant issue numbers?
Closes#22229
See merge request !6435
Fix database seeds for development environment
## What does this MR do?
This MR fixes database seeds for development environment and adds CI test for it.
## Why was this MR needed?
Database seeds for development environment are often broken, and we are not able to catch that when someone modified `db/fixtures` and forgets to reseed database.
Closes#22422
See merge request !6475
This commit changes the revisions used for diffs. The current behaviour is
to show all changes between current tip of master and tip of the MR, rather
than matching the output of the web frontend (which just shows the changes
in the MR). Switching from start_sha to base_sha fixes this.
Cycle Analytics: first iteration
## What does this MR do?
- Implement the first iteration of the "Cycle Analytics" feature.
## What are the relevant issue numbers?
- Closes#21170
## Screenshots
## Backend Tasks
- [x] Implementation
- [x] Phases
- [x] Issue (Tracker)
- [x] Plan (Board)
- [x] Code (IDE)
- [x] Test (CI)
- [x] Review (MR)
- [x] Staging (CD)
- [x] Production (Total)
- [x] Make heuristics more modular
- [x] Scope to project
- [x] Date range (30 days, 90 days)
- [x] Access restriction
- [x] Test
- [x] Find a better way to test these phases
- [x] Phases
- [x] Issue (Tracker)
- [x] Plan (Board)
- [x] Code (IDE)
- [x] Test (CI)
- [x] Review (MR)
- [x] Staging (CD)
- [x] Production (Total)
- [x] Test for "end case happens before start case"
- [x] Consolidate helper
- [x] Miniboss review
- [x] Performance testing with mock data
- [x] Improve performance
- [x] Pre-calculate "merge requests closing issues
- [x] Pre-calculate everything else
- [x] Test performance against 10k issues
- [x] Test all pre-calculation code
- [x] Ci::Pipeline -> build start/finish
- [x] Ci::Pipeline#merge_requests
- [x] Issue -> record default metrics after save
- [x] MergeRequest -> record default metrics after save
- [x] Deployment -> Update "first_deployed_to_production_at" for MR metrics
- [x] Git Push -> Update "first commit mention" for issue metrics
- [x] Merge request create/update/refresh -> Update "merge requests closing issues"
- [x] Remove `MergeRequestsClosingIssues` when necessary
- [x] Changes to unblock Fatih
- [x] Add summary data
- [x] `stats` should be array
- [x] Let `stats` be `null` if all `stats` are null
- [x] Indexes for "merge requests closing issues"
- [x] Test summary data
- [x] Scope everything to project
- [x] Find out why tests were passing
- [x] Filter should include issues/MRs which have made it to production within the range
- [x] Don't create duplicate `MergeRequestsClosingIssues`
- [x] Fix tests
- [x] MySQL median
- [x] Assign to Douwe for review
- [x] Fix conflicts
- [x] Implement suggestions from Yorick's review
- [x] Test on PG
- [x] Test on MySQL
- [x] Refactor
- [x] Cleanup
- [x] What happens if we have no data at all?
- [x] Extract common queries to methods / scopes
- [x] Remove unused queries
- [x] Downtime for foreign key migrations
- [x] Find a way around "if issue.metrics.present?" all over the place
- [x] Find a way around "if merge_request.metrics.present?" all over the place
- [x] Test migrations on a fresh database
- [x] MySQL
- [x] Pg
- [x] Access issues
- While the project is public and the visibility is set to "Everyone with access", you cannot visit the cycle analytics page when signed out.
- [x] CHANGELOG
- [x] Implement suggestions from Douwe's review
- [x] First set of comments
- [x] Second set of comments
- [x] Third set of comments
- [x] Fourth set of comments
- [x] Make sure build is green
- [ ] Make issue for "polish"
- [ ] EE MR
See merge request !5986
- Don't use `TableReferences` - using `.arel_table` is shorter!
- Move some database-related code to `Gitlab::Database`
- Remove the `MergeRequest#issues_closed` and
`Issue#closed_by_merge_requests` associations. They were either
shadowing or were too similar to existing methods. They are not being
used anywhere, so it's better to remove them to reduce confusion.
- Use Rails 3-style validations
- Index for `MergeRequest::Metrics#first_deployed_to_production_at`
- Only include `CycleAnalyticsHelpers::TestGeneration` for specs that
need it.
- Other minor refactorings.
Johan H
henrik
Robert Speicher
Will Starms
Justin DiPierro
Rémy Coutable
Ahmad Sherif
Paco Guzman
Yorick Peterse
Douglas Barbosa Alexandre
James Lopez
Jacob Vosmaer
Douwe Maan
Patricio Cano
Markus Koller
Grzegorz Bizon
Nick Thomas
Dmitriy Zaporozhets
Stan Hu
Timothy Andrew
Dan Dunckel