Having two states that essentially mean the same thing is very much like
having a boolean "true" and boolean "mostly-true": it's rather silly.
This commit merges the "reopened" state into the "opened" state while
taking care of system notes still showing messages along the lines of
"Alice reopened this issue".
A big benefit from having only two states (opened and closed) is that
indexing and querying becomes simpler and more performant. For example,
to get all the opened queries we no longer have to query both states:
SELECT *
FROM issues
WHERE project_id = 2
AND state IN ('opened', 'reopened');
Instead we can query a single state directly, which can be much faster:
SELECT *
FROM issues
WHERE project_id = 2
AND state = 'opened';
Further, only having two states makes indexing easier as we will only
ever filter (and thus scan an index) using a single value. Partial
indexes could help but aren't supported on MySQL, complicating the
development process and not being helpful for MySQL.
For merge requests created after 9.4, we have a `merge_request_diff_commits`
table we can get all the SHAs from very quickly. We just need to exclude these
when we load from the legacy format, by ignoring diffs with no serialised
commits.
Once these have been migrated in the background, every MR will see this
improvement.
This is an ID-less table with just three columns: an association to the merge
request diff the commit belongs to, the relative order of the commit within the
merge request diff, and the commit SHA itself.
Previously we stored much more information about the commits, so that we could
display them even when they were deleted from the repo. Since 8.0, we ensure
that those commits are kept around for as long as the target repo itself is, so
we don't need to duplicate that data in the database.
This is allowed for existing instances so we don't end up 76 offenses
right away, but for new code one should _only_ use this if they _have_
to remove non database data. Even then it's usually better to do this in
a service class as this gives you more control over how to remove the
data (e.g. in bulk).
This removes the need for relying on Rails' "dependent" option for data
removal, which is _incredibly_ slow (even when using :delete_all) when
deleting large amounts of data. This also ensures data consistency is
enforced on DB level and not on application level (something Rails is
really bad at).
This commit also includes various migrations to add foreign keys to
tables that eventually point to "projects" to ensure no rows get
orphaned upon removing a project.
Fix https://gitlab.com/gitlab-org/gitlab-ce/issues/27070
Deprecate "chat commands" in favor of "slash commands"
We looked for things like:
- `slash commmand`
- `slash_command`
- `slash-command`
- `SlashCommand`
I don't know why this happens exactly, but given an upstream and fork repository
from a customer, both of which required GC, resolving conflicts would corrupt
the fork so badly that it couldn't be cloned.
This isn't a perfect fix for that case, because the MR may still need to be
merged manually, but it does ensure that the repository is at least usable.
My best guess is that when we generate the index for the conflict
resolution (which we previously did in the target project), we obtain a
reference to an OID that doesn't exist in the source, even though we already
fetch the refs from the target into the source.
Explicitly setting the source project as the place to get the merge index from
seems to prevent repository corruption in this way.
The problem is that we often go via a diff object constructed from the diffs
stored in the DB. Those diffs, by definition, don't overflow, so we don't have
access to the 'correct' `real_size` - that is stored on the MR diff object
iself.
Rename column in the database
Rename fields related to import/export feature
Rename API endpoints
Rename documentation links
Rename the rest of occurrences in the code
Replace the images that contain the words "build succeeds" and docs referencing to them
Make sure pipeline is green and nothing is missing.
updated doc images
renamed only_allow_merge_if_build_succeeds in projects and fixed references
more updates
fix some spec failures
fix rubocop offences
fix v3 api spec
fix MR specs
fixed issues with partials
fix MR spec
fix alignment
add missing v3 to v4 doc
wip - refactor v3 endpoints
fix specs
fix a few typos
fix project specs
copy entities fully to V3
fix entity error
more fixes
fix failing specs
fixed missing entities in V3 API
remove comment
updated code based on feedback
typo
fix spec
Previously, we created an unmergeable todo when a merge request:
1. Had merge when pipeline succeeds set.
2. Became unmergeable.
However, when merge when pipeline succeeds fails due to unmergeability,
the flag isn't actually removed. And a merge request can become
unmergeable multiple times, as every time the target branch is updated
we need to re-check the mergeable status. This means that if the todo
was marked done, and the MR was checked again, a new todo would be
created for the same event.
Instead of checking this, we should create the todo from the service
responsible for merging when the pipeline succeeds. That way the todo is
guaranteed to only be created when we care about it.
This was wrong when there were over 100 files in the diff, because we
did not use the same diff options as subclasses of
`Gitlab::Diff::FileCollection::Base` when getting the raw diffs. (The
reason we don't use those classes directly is because they may perform
highlighting, which isn't needed for just counting the diffs.)
We don't need to create the intermediate FileCollection object with its
associated highlighting in order to count how many files changed. Both
the compare object and the MR diff object have a raw_diffs method that
is perfectly fine for this case.
Fix missing Note access checks in by moving Note#search to updated NoteFinder
Split from !2024 to partially solve https://gitlab.com/gitlab-org/gitlab-ce/issues/23867
## Which fixes are in this MR?
⚠️ - Potentially untested
💣 - No test coverage
🚥 - Test coverage of some sort exists (a test failed when error raised)
🚦 - Test coverage of return value (a test failed when nil used)
✅ - Permissions check tested
### Note lookup without access check
- [x] ✅ app/finders/notes_finder.rb:13 :download_code check
- [x] ✅ app/finders/notes_finder.rb:19 `SnippetsFinder`
- [x] ✅ app/models/note.rb:121 [`Issue#visible_to_user`]
- [x] ✅ lib/gitlab/project_search_results.rb:113
- This is the only use of `app/models/note.rb:121` above, but importantly has no access checks at all. This means it leaks MR comments and snippets when those features are `team-only` in addition to the issue comments which would be fixed by `app/models/note.rb:121`.
- It is only called from SearchController where `can?(current_user, :download_code, @project)` is checked, so commit comments are not leaked.
### Previous discussions
- [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#b915c5267a63628b0bafd23d37792ae73ceae272_13_13 `: download_code` check on commit
- [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#b915c5267a63628b0bafd23d37792ae73ceae272_19_19 `SnippetsFinder` should be used
- `SnippetsFinder` should check if the snippets feature is enabled -> https://gitlab.com/gitlab-org/gitlab-ce/issues/25223
### Acceptance criteria met?
- [x] Tests added for new code
- [x] TODO comments removed
- [x] Squashed and removed skipped tests
- [x] Changelog entry
- [ ] State Gitlab versions affected and issue severity in description
- [ ] Create technical debt issue for NotesFinder.
- Either split into `NotesFinder::ForTarget` and `NotesFinder::Search` or consider object per notable type such as `NotesFinder::OnIssue`. For the first option could create `NotesFinder::Base` which is either inherited from or which can be included in the other two.
- Avoid case statement anti-pattern in this finder with use of `NotesFinder::OnCommit` etc. Consider something on the finder for this? `Model.finder(user, project)`
- Move `inc_author` to the controller, and implement `related_notes` to replace `non_diff_notes`/`mr_and_commit_notes`
See merge request !2035
The target branch of a merge request has to be a branch in the project
for which the merge request is submitted. When a branch changes in a fork,
it does not make sense to reload diffs of merge requests in the upstream
project that use the same branch name as the target branch.
Please note that it does make sense to reload diffs when the source branch
changes.
When a merge request can only be merged when all discussions are
resolved. This feature allows to easily delegate those discussions to a
new issue, while marking them as resolved in the merge request.
The user is presented with a new issue, prepared with mentions of all
unresolved discussions, including the first unresolved note of the
discussion, time and link to the note.
When the issue is created, the discussions in the merge request will get
a system note directing the user to the newly created issue.
Flushing the events cache worked by updating a recent number of rows in
the "events" table. This has the result that on PostgreSQL a lot of dead
tuples are produced on a regular basis. This in turn means that
PostgreSQL will spend considerable amounts of time vacuuming this table.
This in turn can lead to an increase of database load.
For GitLab.com we measured the impact of not using events caching and
found no measurable increase in response timings. Meanwhile not flushing
the events cache lead to the "events" table having no more dead tuples
as now rows are only inserted into this table.
As a result of this we are hereby removing events caching as it does not
appear to help and only increases database load.
For more information see the following comment:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6578#note_18864037
Also allow merge request to be merged with skipped pipeline and the
"only allow merge when pipeline is green" feature enabled
Signed-off-by: Rémy Coutable <remy@rymai.me>
When reading conflicts:
1. Add a `type` field. `text` works as before, and has `sections`;
`text-editor` is a file with ambiguous conflict markers that can only
be resolved in an editor.
2. Add a `content_path` field pointing to a JSON representation of the
file's content for a single file.
3. Hitting `content_path` returns a similar datastructure to the `file`,
but without the `content_path` and `sections` fields, and with a
`content` field containing the full contents of the file (with
conflict markers).
When writing conflicts:
1. Instead of `sections` being at the top level, they are now in a
`files` array. This matches the read format better.
2. The `files` array contains file hashes, each of which must contain:
a. `new_path`
b. `old_path`
c. EITHER `sections` (which works as before) or `content` (with the
full content of the resolved file).
Now a merge request with a blank description will no longer produce a
merge commit message like this:
```
Merge branch 'foo' into 'master'
Bring the wonders of foo into the world
See merge request !7283
```
What an improvement! 🎉
Show all pipelines from all merge_request_diffs
This way we could also show pipelines from commits which
were discarded due to a force push.
Closes#21889
See merge request !6414
- Don't use `TableReferences` - using `.arel_table` is shorter!
- Move some database-related code to `Gitlab::Database`
- Remove the `MergeRequest#issues_closed` and
`Issue#closed_by_merge_requests` associations. They were either
shadowing or were too similar to existing methods. They are not being
used anywhere, so it's better to remove them to reduce confusion.
- Use Rails 3-style validations
- Index for `MergeRequest::Metrics#first_deployed_to_production_at`
- Only include `CycleAnalyticsHelpers::TestGeneration` for specs that
need it.
- Other minor refactorings.
- Move things common to `Issue` and `MergeRequest` into `Issuable`
- Move more database-specific functions into `Gitlab::Database`
- Indentation changes and other minor refactorings.
However, if MergeRequest#all_commits_sha would want to handle
non-persisted merge request, by judging its name, it should not
just give 1 SHA, but all of them.
But we don't really care all_commits_sha for non-persisted merge
request anyway. So I think we should just ignore that case.
Better to not implementing something than implementing it in a
wrong and confusing way.
1. Change multiple updates to a single `update_all`
2. Use cascading deletes
3. Extract an average function for the database median.
4. Move database median to `lib/gitlab/database`
5. Use `delete_all` instead of `destroy_all`
6. Minor refactoring
All the code that pre-calculates metrics for use in the cycle analytics
page.
- Ci::Pipeline -> build start/finish
- Ci::Pipeline#merge_requests
- Issue -> record default metrics after save
- MergeRequest -> record default metrics after save
- Deployment -> Update "first_deployed_to_production_at" for MR metrics
- Git Push -> Update "first commit mention" for issue metrics
- Merge request create/update/refresh -> Update "merge requests closing issues"
1. These changes bring down page load time for 100 issues from more than
a minute to about 1.5 seconds.
2. This entire commit is composed of these types of performance
enhancements:
- Cache relevant data in `IssueMetrics` wherever possible.
- Cache relevant data in `MergeRequestMetrics` wherever possible.
- Preload metrics
3. Given these improvements, we now only need to make 4 SQL calls:
- Load all issues
- Load all merge requests
- Load all metrics for the issues
- Load all metrics for the merge requests
4. A list of all the data points that are now being pre-calculated:
a. The first time an issue is mentioned in a commit
- In `GitPushService`, find all issues mentioned by the given commit
using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at`
flag for each of them.
- There seems to be a (pre-existing) bug here - files (and
therefore commits) created using the Web CI don't have
cross-references created, and issues are not closed even when
the commit title is "Fixes #xx".
b. The first time a merge request is deployed to production
When a `Deployment` is created, find all merge requests that
were merged in before the deployment, and set the
`first_deployed_to_production_at` flag for each of them.
c. The start / end time for a merge request pipeline
Hook into the `Pipeline` state machine. When the `status` moves to
`running`, find the merge requests whose tip commit matches the
pipeline, and record the `latest_build_started_at` time for each
of them. When the `status` moves to `success`, record the
`latest_build_finished_at` time.
d. The merge requests that close an issue
- This was a big cause of the performance problems we were having
with Cycle Analytics. We need to use `ReferenceExtractor` to make
this calculation, which is slow when we have to run it on a large
number of merge requests.
- When a merge request is created, updated, or refreshed, find the
issues it closes, and create an instance of
`MergeRequestsClosingIssues`, which acts as a join model between
merge requests and issues.
- If a `MergeRequestsClosingIssues` instance links a merge request
and an issue, that issue closes that merge request.
5. The `Queries` module was changed into a class, so we can cache the
results of `issues` and `merge_requests_closing_issues` across
various cycle analytics stages.
6. The code added in this commit is untested. Tests will be added in the
next commit.
Refactor ability.rb into Policies
## What does this MR do?
Factors out `ability.rb` into a new abstraction - the "policy" (stored in `app/policies`). A policy is a class named `#{class_name}Policy` (looked up automatically as needed) that implements `rules` as follows:
``` ruby
class ThingPolicy < BasePolicy
def rules
@user # this is a user to determine abilities for, optionally nil in the anonymous case
@subject # this is the subject of the ability, guaranteed to be an instance of `Thing`
can! :some_ability # grant the :some_ability permission
cannot! :some_ability # ensure that :some_ability is not allowed. this overrides any `can!` that is called before or after
delegate! @subject.other_thing # merge the abilities (can!) and prohibitions (cannot!) from `@subject.other_thing`
can? :some_ability # test whether, so far, :some_ability is allowed
end
def anonymous_rules
# optional. if not implemented `rules` is called where `@user` is nil. otherwise this method is called when `@user` is nil.
end
end
```
See merge request !5796
Add a download buttons for Build Artifacts
## What does this MR do?
This MR adds a download buttons for build artifacts of latest succesful pipeline in:
- dashboard of project,
- branches and tags views,
- and tree viewer
Implement #4255
## What are the relevant issue numbers?
Closes#4255, Closes#14419
## Screenshots
### Project main
### Branches
### Tags
### Source Tree
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] Download buttons
- [x] Models
- [x] Routes
- [x] Projects::ArtifactsController
- [x] API
- Tests
- Rails
- [x] Project#builds_for
- [x] branch name with slashes
- [x] only success builds
- [x] only latest builds
- [x] feature tests for download buttons
- API
- [x] branch name with slashes
- [x] only success builds
- [x] only latest builds
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !5142
Add test for closed MR without fork
Add view test visibility of Reopen and Close buttons
Fix controller tests and validation method
Fix missing space
Remove unused variables from test
closed_without_fork? method refactoring
Add information about missing fork
When closed MR without fork can't edit target branch
Tests for closed MR edit view
Fix indentation and rebase, refactoring
1. Record the `wip_flag_first_removed_at` and
`first_assigned_to_user_other_than_author` metrics for a merge
request. Use a `merge_request_metrics` table, similar to the one for
`issues`. Metrics are recorded `after_save`.
2. Move larger queries to a `CycleAnalytics::Queries` module.
* upstream/master: (107 commits)
Fix external issue tracker "Issues" link leading to 404s
Fix CHANGELOG entries related to 8.11 release.
Fix changelog
Reduce contributions calendar data payload
Add lock_version to merge_requests table
Add hover color to emoji icon
Fix wrong Koding link
Capitalize mentioned issue timeline notes
Fix groups sort dropdown alignment
Use icon helper
Fix inline emoji text alignment
Adds response mime type to transaction metric action when it's not HTML
Moved two 8.11 changelog entries to 8.12
Fix markdown link in doc_styleguide.md
Display project icon from default branch
Reduce number of database queries on builds tab
Update CHANGELOG
Update Issue board documentation
Label list shows all issues (opened or closed) with that label
Update CHANGELOG
...
* upstream/master: (195 commits)
Fix expansion of discussions in diff
Improve performance of MR show page
Fix jumping between discussions on changes tab
Update doorkeeper to 4.2.0
Fix MR note discussion ID
Handle legacy sort order values
Refactor `find_for_git_client` and its related methods.
Remove right margin on Jump button icon
Fix bug causing “Jump to discussion” button not to show
Small refactor and syntax fixes.
Removed unnecessary service for user retrieval and improved API error message.
Added documentation and CHANGELOG item
Added checks for 2FA to the API `/sessions` endpoint and the Resource Owner Password Credentials flow.
Fix behavior around commands with optional arguments
Fix behavior of label_ids and add/remove_label_ids
Remove unneeded aliases
Do not expose projects on deployments
Incorporate feedback
Docs for API endpoints
Expose project for environments
...
* upstream/master: (359 commits)
Add new image to show the 'Reset template' button
Refactor description templates documentation
Remove index from pipeline toggles
Hide `Create new list button` on Issues and MRs pages
Remove params from build; general refactor
Style build container box; add check mark to active build
Display jobs as scrolling list in sidebar
Move stages and jobs to build sidebar
Removed vue assets
Move skipped tests to end of array
Remove unused data attributes
Update changelog
Style build dropdowns
Populate dropdowns with current build on pageload
Selecting stage updates builds dropdown
Add data attributes to builds
Change active state of list items; style dropdown items
Hide dropdown if all tests fit on one line; add counter to dropdown
Add overflow tests to dropdown
Order by build status
...
* upstream/master: (123 commits)
Limit the size of SVGs when viewing them as blobs
Add a spec for ProjectsFinder project_ids_relation option
Fix ProjectsFinder spec
Pass project IDs relation to ProjectsFinder instead of using a block
Speed up todos queries by limiting the projects set we join with
Used phantomjs variable
Ability to specify branches for pivotal tracker integration
Fix a memory leak caused by Banzai::Filter::SanitizationFilter
Update phantomjs link
Remove sleeping and replace escaped text.
Filters test fix
Fixed filtering tests
Removed screenshot command 💩
Updated failing tests
Used mirrored version on GitLab
Updated tests
Fix `U2fSpec` for PhantomJS versions > 2.
Fix file downloading
Install latest stable phantomjs
Use new PhantomJS version
...
Show deployment status on a MR view
## What are the relevant issue numbers?
Resolves#19571, one in the list of #19992
## Screenshots (if relevant)
external_url = nil
external_url != nil
Note, the timings are weird between merging and deploying, that is because I did it in the wrong order.
## Does this MR meet the acceptance criteria?
- [X] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
See merge request !5622
An MR can only be resolved in the UI if:
- It has conflicts.
- It has valid diff_refs (in other words, it supports new diff notes).
- It has no conflicts with one side missing.
- It has no conflicts in binary files.
- It has no conflicts in files too large to display.
- It has no conflicts containing invalid conflict markers.
DRY code + fix rubocop
Add more test cases
Append to changelog
DRY changes list
find_url service for merge_requests
use GET for getting merge request links
remove files
rename to get_url_service
reduce loop
add test case for cross project
refactor tiny thing
update changelog
- Preloading noteable we share the same noteable instance when more than one
discussion refers to the same noteable.
- Any other call to that object that is cached in that object will be for any
discussion.
- In those cases where merge_request_diff has all the sha stored to build a diff_refs get that
diff_refs using directly those sha instead accessing to the git repository to first get the
commits and later the sha.
* Add merge_request_diff validation to merge_request model
* Improve initialize of merge_request_diff object
* Rename some merge_request_diff methods for clarity
Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>
When importing a pull request from GitHub, the old and new branches may
no longer actually exist by those names, but we need to recreate the
merge request diff with the right source and target shas.
We use these `target_branch_sha` and `source_branch_sha` attributes to
force these to the intended values. But the reader methods were always
looking up to the target/source branch head instead of check if these
values was previously set.
1. Don't use case statements for dispatch anymore. This leads to a lot
of duplication, and makes the logic harder to follow.
2. Remove duplicated logic.
- For example, the `can_push_to_branch?` exists, but we also have a
different way of checking the same condition within `change_access_check`.
- This kind of duplication is removed, and the `can_push_to_branch?`
method is used in both places.
3. Move checks returning true/false to `UserAccess`.
- All public methods in `GitAccess` now return an instance of
`GitAccessStatus`. Previously, some methods would return
true/false as well, which was confusing.
- It makes sense for these kinds of checks to be at the level of a
user, so the `UserAccess` class was repurposed for this. The prior
`UserAccess.allowed?` classmethod is converted into an instance
method.
- All external uses of these checks have been migrated to use the
`UserAccess` class
4. Move the "change_access_check" into a separate class.
- Create the `GitAccess::ChangeAccessCheck` class to run these
checks, which are quite substantial.
- `ChangeAccessCheck` returns an instance of `GitAccessStatus` as
well.
5. Break out the boolean logic in `ChangeAccessCheck` into `if/else`
chains - this seems more readable.
6. I can understand that this might look like overkill for !4892, but I
think this is a good opportunity to clean it up.
- http://martinfowler.com/bliki/OpportunisticRefactoring.html
The issue was that `MergeRequest#mergeable?` returns false when the CI
state is not success and project.only_allow_merge_if_build_succeeds is
true. In this case `Projects::MergeRequestsController#merge` would
return the `:failed` status when enabling `merge_when_build_succeeds`,
thus leading to a weird state and the MR never automatically merged.
The fix is to disable the CI state check in the controller safeguard
that early return the `:failed` status.
Signed-off-by: Rémy Coutable <remy@rymai.me>
When using #XYZ in Markdown text, if XYZ exceeds the maximum value of a signed 32-bit integer, we
get an exception when the Markdown render attempts to run `where(iids: XYZ)`. Introduce a method
that will throw out out-of-bounds values.
Closes#18777
Gabriel Mazetto
Yorick Peterse
Jarka Kadlecova
Douwe Maan
Stan Hu
Douwe Maan
James Lopez
Sean McGivern
Adam Niedzielski
Grzegorz Bizon
Toon Claes
Sean McGivern
Felipe Artur
Eric Eastwood
Robert Speicher
Valery Sizov
Fatih Acet
mhasbini
Robert Speicher
Luke "Jared" Bennett
Kushal Pandya
Douglas Barbosa Alexandre
James Edwards-Jones
Ruben Davila
Dmitriy Zaporozhets
Jan Christophersen
Dongqing Hu
Oswaldo Ferreira
twonegatives
Jarka Kadlecova
Mike Greiling
Gabriel Gizotti
Rémy Coutable
Bob Van Landuyt
Oswaldo Ferreira
Grzegorz Bizon
Rodolfo Santos
Guilherme Salazar
David Wagner
Nick Thomas
Z.J. van de Weg
Paco Guzman
Thomas Balthazar
Kamil Trzcinski
Timothy Andrew
Lin Jen-Shin
Katarzyna Kobierska
http://jneen.net/
Connor Shea
Scott Le
Mathias Vestergaard
Patricio Cano
Rui Anderson