Fix missing Note access checks in by moving Note#search to updated NoteFinder
Split from !2024 to partially solve https://gitlab.com/gitlab-org/gitlab-ce/issues/23867
## Which fixes are in this MR?
⚠️ - Potentially untested
💣 - No test coverage
🚥 - Test coverage of some sort exists (a test failed when error raised)
🚦 - Test coverage of return value (a test failed when nil used)
✅ - Permissions check tested
### Note lookup without access check
- [x] ✅ app/finders/notes_finder.rb:13 :download_code check
- [x] ✅ app/finders/notes_finder.rb:19 `SnippetsFinder`
- [x] ✅ app/models/note.rb:121 [`Issue#visible_to_user`]
- [x] ✅ lib/gitlab/project_search_results.rb:113
- This is the only use of `app/models/note.rb:121` above, but importantly has no access checks at all. This means it leaks MR comments and snippets when those features are `team-only` in addition to the issue comments which would be fixed by `app/models/note.rb:121`.
- It is only called from SearchController where `can?(current_user, :download_code, @project)` is checked, so commit comments are not leaked.
### Previous discussions
- [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#b915c5267a63628b0bafd23d37792ae73ceae272_13_13 `: download_code` check on commit
- [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2024/diffs#b915c5267a63628b0bafd23d37792ae73ceae272_19_19 `SnippetsFinder` should be used
- `SnippetsFinder` should check if the snippets feature is enabled -> https://gitlab.com/gitlab-org/gitlab-ce/issues/25223
### Acceptance criteria met?
- [x] Tests added for new code
- [x] TODO comments removed
- [x] Squashed and removed skipped tests
- [x] Changelog entry
- [ ] State Gitlab versions affected and issue severity in description
- [ ] Create technical debt issue for NotesFinder.
- Either split into `NotesFinder::ForTarget` and `NotesFinder::Search` or consider object per notable type such as `NotesFinder::OnIssue`. For the first option could create `NotesFinder::Base` which is either inherited from or which can be included in the other two.
- Avoid case statement anti-pattern in this finder with use of `NotesFinder::OnCommit` etc. Consider something on the finder for this? `Model.finder(user, project)`
- Move `inc_author` to the controller, and implement `related_notes` to replace `non_diff_notes`/`mr_and_commit_notes`
See merge request !2035
When a merge request can only be merged when all discussions are
resolved. This feature allows to easily delegate those discussions to a
new issue, while marking them as resolved in the merge request.
The user is presented with a new issue, prepared with mentions of all
unresolved discussions, including the first unresolved note of the
discussion, time and link to the note.
When the issue is created, the discussions in the merge request will get
a system note directing the user to the newly created issue.
Flushing the events cache worked by updating a recent number of rows in
the "events" table. This has the result that on PostgreSQL a lot of dead
tuples are produced on a regular basis. This in turn means that
PostgreSQL will spend considerable amounts of time vacuuming this table.
This in turn can lead to an increase of database load.
For GitLab.com we measured the impact of not using events caching and
found no measurable increase in response timings. Meanwhile not flushing
the events cache lead to the "events" table having no more dead tuples
as now rows are only inserted into this table.
As a result of this we are hereby removing events caching as it does not
appear to help and only increases database load.
For more information see the following comment:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6578#note_18864037
`discussion_id` may not be present when the SELECT call for notes
does not include this attribute. Don't attempt to set the discussion ID
unless the model contains the attribute.
Closes#21119, #21128
This concern provides an optimized/simplified version of the "cache_key"
method. This method is about 9 times faster than the default "cache_key"
method.
The produced cache keys _are_ different from the previous ones but this
is worth the performance improvement. To showcase this I set up a
benchmark (using benchmark-ips) that compares FasterCacheKeys#cache_key
with the regular cache_key. The output of this benchmark was:
Calculating -------------------------------------
cache_key 4.825k i/100ms
cache_key_fast 21.723k i/100ms
-------------------------------------------------
cache_key 59.422k (± 7.2%) i/s - 299.150k
cache_key_fast 543.243k (± 9.2%) i/s - 2.694M
Comparison:
cache_key_fast: 543243.4 i/s
cache_key: 59422.0 i/s - 9.14x slower
To see the impact on real code I applied these changes and benchmarked
Issue#referenced_merge_requests. For an issue referencing 10 merge
requests these changes shaved off between 40 and 60 milliseconds.
* There where path where +1 was stored as +1 not as thumbsup
that was causing problems such as showing thumbsup icon 2 time.
I fixed this to always normalize and store +1 as tumbsup
have been fully redacted and contain cross-project references.
The previous implementation relied on Note#cross_reference_not_visible_for?,
which essentially tries to render all the Markdown references in a system note
and only displays the note if the user can see the referring project. But this
duplicated the work that Banzai::NotesRenderer was doing already. Instead, for
each note we render, we memoize the number of visible user references and
use it later if it is available.
Improves #19273
This commit changes the way certain documents are rendered (currently
only Notes) and how documents are redacted. Previously both rendering
and redacting would run on a per document basis. The result of this was
that for every document we'd have to run countless queries just to
figure out if we could display a set of links or not.
This commit changes things around so that redacting Markdown documents
is no longer tied into the html-pipeline Gem. This in turn allows it to
redact multiple documents in a single pass, thus reducing the number of
queries needed.
In turn rendering issue/merge request notes has been adjusted to take
advantage of this new setup. Instead of rendering Markdown somewhere
deep down in a view the Markdown is rendered and redacted in the
controller (taking the current user and all that into account). This has
been done in such a way that the "markdown()" helper method can still be
used on its own.
This particular commit also paves the way for caching rendered HTML on
object level. Right now there's an accessor method Note#note_html which
is used for setting/getting the rendered HTML. Once we cache HTML on row
level we can simply change this field to be a column and call a "save"
whenever needed and we're pretty much done.
This commit eager loads the award emoji on both the issues and the MRs.
When loading an issue with 108 comments this reduces the query count by
327 queries. On a merge request with the same amount of comments this
saves 148 queries. The large difference is not clear to me at this
point and the total query count is still huge with 387 and 1034
respectively. The biggest problem however, remains the calculation of
participants.
Separate reference gathering from rendering
This is a required step to allow batch processing when gathering references. This in turn would allow grabbing (for example) all mentioned users of an issue/merge request using a single query.
cc @rspeicher @DouweM
See merge request !3969
There are several changes to this module:
1. The use of an explicit stack in Participable#participants
2. Proc behaviour has been changed
3. Batch permissions checking
== Explicit Stack
Participable#participants no longer uses recursion to process "self" and
all child objects, instead it uses an Array and processes objects in
breadth-first order. This allows us to for example create a single
Gitlab::ReferenceExtractor instance and pass this to any Procs. Re-using
a ReferenceExtractor removes the need for running potentially many SQL
queries every time a Proc is called on a new object.
== Proc Behaviour Changed
Previously a Proc in Participable was expected to return an Array of
User instances. This has been changed and instead it's now expected that
a Proc modifies the Gitlab::ReferenceExtractor passed to it. The return
value of the Proc is ignored.
== Permissions Checking
The method Participable#participants uses
Ability.users_that_can_read_project to check if the returned users have
access to the project of "self" _without_ running multiple SQL queries
for every user.
Jarka Kadlecova
Douwe Maan
Bob Van Landuyt
Douwe Maan
Yorick Peterse
Oswaldo Ferreira
Nick Thomas
Z.J. van de Weg
barthc
Stan Hu
dixpac
James Lopez
Paco Guzman
Douglas Barbosa Alexandre
Robert Speicher
Valery Sizov
Grzegorz Bizon