This replaces the repository param.
This allows more flexiblity as sometimes we have highlight content
not related to repository. Sometimes we know ahead of time the language
of the content. Lastly language determination seems better fit as a
logic in the Blob class.
`repository` param is only used to determine the language, which seems
to be the responsiblity of Blob.
This saves about 128 MB of baseline RAM usage per Unicorn and
Sidekiq process (!).
Linguist wasn't detecting languages anymore from CE/EE since
9ae8b57467. However, Linguist::BlobHelper
was still being depended on by BlobLike and others.
This removes the Linguist gem, given it isn't required anymore.
EscapeUtils were pulled in as dependency, but given Banzai depends on
it, it is now added explicitly.
Previously, Linguist was used to detect the best ACE mode. Instead,
we rely on ACE to guess the best mode based on the file extension.
When we added caching, this meant that calling `can_be_resolved_in_ui?` didn't
always call `lines`, which meant that we didn't get the benefit of the
side-effect from that, where it forced the conflict data itself to UTF-8.
To fix that, make this explicit by separating the `raw_content` (any encoding)
from the `content` (which is either UTF-8, or an exception is raised).
This does two things:
- Pass commit oids instead of `Gitlab::Git::Commit`s. We only need the
former.
- Depend on only the target repository for conflict listing. For
conflict resolution, treat one repository as a remote one so that we can
implement it as such in Gitaly.
The Gitaly CommitService is being hammered by n + 1 calls, mostly when
finding commits. This leads to this gRPC being turned of on production:
https://gitlab.com/gitlab-org/gitaly/issues/514#note_48991378
Hunting down where it came from, most of them were due to
MergeRequest#show. To prove this, I set a script to request the
MergeRequest#show page 50 times. The GDK was being scraped by
Prometheus, where we have metrics on controller#action and their Gitaly
calls performed. On both occations I've restarted the full GDK so all
caches had to be rebuild.
Current master, 806a68a81f, needed 435 requests
After this commit, 154 requests
I don't know why this happens exactly, but given an upstream and fork repository
from a customer, both of which required GC, resolving conflicts would corrupt
the fork so badly that it couldn't be cloned.
This isn't a perfect fix for that case, because the MR may still need to be
merged manually, but it does ensure that the repository is at least usable.
My best guess is that when we generate the index for the conflict
resolution (which we previously did in the target project), we obtain a
reference to an OID that doesn't exist in the source, even though we already
fetch the refs from the target into the source.
Explicitly setting the source project as the place to get the merge index from
seems to prevent repository corruption in this way.
We wanted to check that the text could be encoded as JSON, because
conflict resolutions are passed back and forth in that format, so the
file itself must be UTF-8. However, all strings from the repository come
back without an encoding from Rugged, making them ASCII_8BIT.
We force to UTF-8, and reject if it's invalid. This still leaves the
problem of a file that 'looks like' UTF-8 (contains valid UTF-8 byte
sequences), but isn't. However:
1. If the conflicts contain the problem bytes, the user will see that
the file isn't displayed correctly.
2. If the problem bytes are outside of the conflict area, then we will
write back the same bytes when we resolve the conflicts, even though
we though the encoding was UTF-8.
When reading conflicts:
1. Add a `type` field. `text` works as before, and has `sections`;
`text-editor` is a file with ambiguous conflict markers that can only
be resolved in an editor.
2. Add a `content_path` field pointing to a JSON representation of the
file's content for a single file.
3. Hitting `content_path` returns a similar datastructure to the `file`,
but without the `content_path` and `sections` fields, and with a
`content` field containing the full contents of the file (with
conflict markers).
When writing conflicts:
1. Instead of `sections` being at the top level, they are now in a
`files` array. This matches the read format better.
2. The `files` array contains file hashes, each of which must contain:
a. `new_path`
b. `old_path`
c. EITHER `sections` (which works as before) or `content` (with the
full content of the resolved file).
Handle non-UTF-8 conflicts gracefully
## What does this MR do?
If a conflict file isn't in a UTF-8-compatible encoding, we can't resolve it in the UI.
## What are the relevant issue numbers?
Closes#21247.
See merge request !5961
These can't be resolved in the UI because if they aren't in a UTF-8
compatible encoding, they can't be rendered as JSON. Even if they could,
we would be implicitly changing the file encoding anyway, which seems
like a bad idea.
An MR can only be resolved in the UI if:
- It has conflicts.
- It has valid diff_refs (in other words, it supports new diff notes).
- It has no conflicts with one side missing.
- It has no conflicts in binary files.
- It has no conflicts in files too large to display.
- It has no conflicts containing invalid conflict markers.
- Add match line header to expected result for `File#sections`.
- Lowercase CSS colours.
- Remove unused `diff_refs` keyword argument.
- Rename `parent` -> `parent_file`, to be more explicit.
- Skip an iteration when highlighting.
GitLab Bot
gfyoung
Mark Chao
Zeger-Jan van de Weg
Sean McGivern
Alejandro Rodríguez
Maxim Rydkin
Jacob Vosmaer
Douwe Maan
Grzegorz Bizon
Robert Speicher