This adds `rugged_duration_ms` and `rugged_calls` fields to
`api_json.log` and `production_json.log`. This will make it easier to
identify performance issues caused by excessive I/O.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/64676
When passing start_branch on committing from the WebIDE, it's possible
that the branch has changed since editing started, which results in the
change being applied on top of the latest commit in the branch and
overwriting the new changes.
By passing the start_sha instead we can make sure that the change is
applied on top of the commit which the user started editing from.
Currently, MergeToRefService is specifically designed for
createing merge commits from source branch and target branch of
merge reqeusts. We extend this behavior to source branch and any
target ref paths.
It used to be the case that GitLab created symlinks for each repository
to one copy of the Git hooks, so these ran when required. This changed
to set the hooks dynamically on Gitaly when invoking Git.
The side effect is that we didn't need all these symlinks anymore, which
Gitaly doesn't create anymore either. Now that means that the tests in
GitLab-Rails should test for it either.
Related: https://gitlab.com/gitlab-org/gitaly/issues/1392#note_175619926
Attempting to use the API endpoint
/projects/:id/repository/tree?recursive=true would only return a subset
of the results since the full recursive list wasn't actually being
returned.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/61979
We want to optimize the query for the CountDivergingCommits rpc by
removing the --max-count argument now that we have commit graphs
enabled for all repositories during housekeeping. However, we want to
test this first behind a feature flag.
The new two-step Gitaly `Rebase` RPC yields the rebase commit SHA to the
client before proceeding with the rebase.
This avoids an issue where the rebase commit SHA was returned when the
RPC had fully completed, and in some cases this would be after the Rails
`post_receive` worker services had already run. In these situations,
the merge request did not yet have its rebase_commit_sha attribute set
introducing the possibility for bugs (such as previous approvals being
reset).
https://gitlab.com/gitlab-org/gitlab-ee/issues/5966
Gitaly's FetchIntoObjectPool RPC will idempotently fetch objects into an
object pool. If the pool doesn't exist, it will create an empty pool
before attempting the fetch. This change adds client code as well as
specs to cover this behavior.
Inside a wiki, when we show the sidebar or browse to the `pages`,
all page contents are retrieved from Gitaly and that is a waste
of resources, since no content from that pages are going to be
showed.
This MR introduces the method `ProjectWiki#list_pages`,
which uses new wiki_list_pages RPC call to retrieve
pages without content
Also in the `WikisController` we're using the method to show
pages in the sidebar and also on the `pages` page.
A temp reference is only needed to fetch a branch from another project,
as in the case for forked repositories. For branch comparisons within
the same project, we can just use the existing branch names to do the
comparison.
Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/38689#note_126107862
This brings back changes in
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20432.
For users using Gitaly on top of NFS, accessing the Git data directly
via Rugged may be faster than going through than Gitaly. This merge
request introduces the feature flag `rugged_list_commits_by_oid` to
activate the Rugged method.
For one customer, we saw that ListCommitsByOid was the second highest
used endpoint that may be causing increased load.
Add `GetArchiveRequest` to git-archive params.
Modifies `Git::Repository#archive_metadata` to append `path`
to `ArchivePrefix` so it'll not hit the cache of repository archive
when it already exists.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/59777.
In earlier iterations of our implementation of Git object deduplication
we thought we would be making extensive use of Git remotes in pool
repositories in the future, and that we should manage these remotes
carefully from the start. We now expect we only care about one remote,
namely the source project. The other remotes are there only for forensic
purposes.
Before this MR we tried to also remove pool remotes when member projects
got deleted, with the UnlinkRepositoryFromObjectPool RPC. This is
fragile when there are race conditions (see
https://gitlab.com/gitlab-org/gitaly/issues/1568#note_153955926). We
have spent some time making this RPC less fragile in
https://gitlab.com/gitlab-org/gitaly/merge_requests/1151 but looking at
this problem again, I think we should just stop calling it.
The Rugged implementation was recursively scanning the repository to
create `flat_path` because the post-process step was being called from
with a loop. For large repositories, this was significantly slowing
things down. Break the call to `rugged_populate_flat_path` out of this
loop to make this work efficiently.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/59759
Error messages from custom pre-receive hooks now appear in the GitLab
UI.
This is re-enabling a feature that had been disabled in merge request
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18646
The feature had been disabled due to security concerns that information
which was not intended to be public (like stack traces) would leak into
public view.
PreReceiveErrors (from pre-receive, post-receive and update custom
hooks) are now filtered for messages that have been prefixed in a
particular way.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/48132
Allow incomplete commit records to load their full data from gitaly.
Commits can be based on a Hash of data retrieved from PostgreSQL, and
this data can be intentionally incomplete in order to save space.
A new method #gitaly? has been added to Gitlab::Git::Commit, which
returns true if the underlying data source of the Commit is a
Gitaly::GitCommit.
CommitCollection now has a method #enrich which replaces non-gitaly
commits in place with commits from gitaly.
CommitCollection#without_merge_commits has been updated to call this
method, as in order to determine a merge commit we need to have parent
data.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/58805
Allow incomplete commit records to load their full data from gitaly.
Commits can be based on a Hash of data retrieved from PostgreSQL, and
this data can be intentionally incomplete in order to save space.
A new method #gitaly? has been added to Gitlab::Git::Commit, which
returns true if the underlying data source of the Commit is a
Gitaly::GitCommit.
CommitCollection now has a method #enrich which replaces non-gitaly
commits in place with commits from gitaly.
CommitCollection#without_merge_commits has been updated to call this
method, as in order to determine a merge commit we need to have parent
data.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/58805
EE added the ability to set timeouts when handling blobs. Since there's
no particular reason for this to be EE specific we can just backport
this to CE.
This brings back some of the changes in
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20176/diffs.
We discovered another N+1 that hits Gitaly `TreeEntry` via the
`RelativeLinkFilter`:
https://gitlab.com/gitlab-org/gitlab-ce/issues/58657. When a blob is
loaded with many relative links, `TreeEntry` is called for each link to
scan the URI type.
There are multiple paths that hit Gitaly `TreeEntry`, and
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25706 did not
cover all cases. This commit covers another common use case.
For users using Gitaly on top of NFS, accessing the Git data directly
via Rugged may be faster than going through than Gitaly. This merge
request introduces the feature flag `rugged_commit_tree_entry` to
activate the Rugged method.
When `force` is set to `true` and `start_branch` is set, the
branch will be ovewritten with the new commit based on the
`HEAD` of the `start_branch`.
This commit includes changes to update the `gitaly-proto` gem.
This brings back some of the changes in
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20339.
For users using Gitaly on top of NFS, accessing the Git data directly
via Rugged is more performant than Gitaly. This merge request introduces
the feature flag `rugged_find_commit` to activate Rugged paths.
There are also Rake tasks `gitlab:features:enable_rugged` and
`gitlab:features:disable_rugged` to enable/disable these feature
flags altogether.
Part of four Rugged changes identified in
https://gitlab.com/gitlab-org/gitlab-ce/issues/57317.
Adds the ground work for writing into
the merge ref refs/merge-requests/:iid/merge the
merge result between source and target branches of
a MR, without further side-effects such as
mailing, MR updates and target branch changes.
This commit, introduced in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23812,
fixes a problem creating a displaying image diff notes when the image
is stored in LFS. The main problem was that `Gitlab::Diff::File` was
returning an invalid valid in `text?` for this kind of files.
It also fixes a rendering problem with other LFS files, like text
ones. They LFS pointer shouldn't be shown when LFS is enabled
for the project, but they were.
When a project is forked, the new repository used to be a deep copy of everything
stored on disk by leveraging `git clone`. This works well, and makes isolation
between repository easy. However, the clone is at the start 100% the same as the
origin repository. And in the case of the objects in the object directory, this
is almost always going to be a lot of duplication.
Object Pools are a way to create a third repository that essentially only exists
for its 'objects' subdirectory. This third repository's object directory will be
set as alternate location for objects. This means that in the case an object is
missing in the local repository, git will look in another location. This other
location is the object pool repository.
When Git performs garbage collection, it's smart enough to check the
alternate location. When objects are duplicated, it will allow git to
throw one copy away. This copy is on the local repository, where to pool
remains as is.
These pools have an origin location, which for now will always be a
repository that itself is not a fork. When the root of a fork network is
forked by a user, the fork still clones the full repository. Async, the
pool repository will be created.
Either one of these processes can be done earlier than the other. To
handle this race condition, the Join ObjectPool operation is
idempotent. Given its idempotent, we can schedule it twice, with the
same effect.
To accommodate the holding of state two migrations have been added.
1. Added a state column to the pool_repositories column. This column is
managed by the state machine, allowing for hooks on transitions.
2. pool_repositories now has a source_project_id. This column in
convenient to have for multiple reasons: it has a unique index allowing
the database to handle race conditions when creating a new record. Also,
it's nice to know who the host is. As that's a short link to the fork
networks root.
Object pools are only available for public project, which use hashed
storage and when forking from the root of the fork network. (That is,
the project being forked from itself isn't a fork)
In this commit message I use both ObjectPool and Pool repositories,
which are alike, but different from each other. ObjectPool refers to
whatever is on the disk stored and managed by Gitaly. PoolRepository is
the record in the database.
Use shelling out to git to write refs instead of rugged, hoping to
avoid creating invalid refs.
To update HEAD we switched to using `git symbolic-ref`.
By specifying `key`, we get a different lazy batch loader for each
repository, which means that accessing a lazy object from one repository
will only result in that repository's objects being fetched, not those
of other repositories, saving us some unnecessary Gitaly lookups.
This allows users to add patches as attachments to merge request
created via email.
When an email to create a merge request is sent, all the attachments
ending in `.patch` will be applied to the branch specified in the
subject of the email. If the branch did not exist, it will be created
from the HEAD of the repository.
When the patches could not be applied, the error message will be
replied to the user.
The patches can have a maximum combined size of 2MB for now.
Having this in a concern allows us to reuse it for different single
purpose classes that call out to git without going through the
repository every time.
Inlining this code allows us to remove a dependency on gitlab_grit in
gitlab-ce. We can't stop maintaining gitlab_grit yet, since gitaly-ruby
still depends on this gem, but it moves us a step closer.
Removing this dependency also allows us to remove a transitive
dependency on gitlab_grit - which is the whole point of this exercise.
I don't think we can EOL gitlab_grit until it's removed as a dependency
from gitaly-ruby, but this at least gets it out of gitlab-ce.
This saves about 128 MB of baseline RAM usage per Unicorn and
Sidekiq process (!).
Linguist wasn't detecting languages anymore from CE/EE since
9ae8b57467. However, Linguist::BlobHelper
was still being depended on by BlobLike and others.
This removes the Linguist gem, given it isn't required anymore.
EscapeUtils were pulled in as dependency, but given Banzai depends on
it, it is now added explicitly.
Previously, Linguist was used to detect the best ACE mode. Instead,
we rely on ACE to guess the best mode based on the file extension.
Was introduced in the time that GitLab still used NFS, which is not
required anymore in most cases. By removing this, the API it calls will
return empty responses. This interface has to be removed in the next
major release, expected to be 12.0.
Cleanup code, and refactor tests that still use Rugged. After this, there should
be no Rugged code that access the instance's repositories on non-test
environments. There is still some rugged code for other tasks like the
repository import task, but since it doesn't access any repository storage path
it can stay.
- Creates a new column to hold the single patch limit value on
application_settings
- Allows updating this value through the application_settings API
- Calculates single diff patch collapsing limit based on
diff_max_patch_bytes column
- Updates diff limit documentation
- Adds documentation (with warning) as of how one can update this limit
Implements list_last_commits_for_tree to communicate with the
ListLastCommitsForTree Gitaly RPC
Bumps the Gitaly server version
Bumps the Gitaly-Proto gem version
Even if it doesn’t save lines of code, since people will tend to use
code they’ve seen. And `SafeRequestStore` is safer since you
don’t have to remember to check `RequestStore.active?`.
We used to apply this limitation on GitLab when using Rugged. Now that
we've shifted to Gitaly, this parameter needs to be sent via a RPC
request. Currently this value is hardcoded on Gitaly.
Without this parameter, every load of a Wiki page will load all the Wiki pages
in the repository for the sidebar. This is a significant performance penalty
that can significant slow the display of all Wiki pages.
Relates to #40101
This old migration used Rugged to find a commit, while Gitaly is the
prefered way now. By migrating this to Gitaly, Gitaly is now a required
running component for this migration.
Part of https://gitlab.com/gitlab-org/gitaly/issues/1106
After trying to remove the whole method in
8f69014af2902d8d53fe931268bec60f6858f160, this is a more gentle
approach to the method. :)
Prior to this change, new commit detection wasn't implemented in Gitaly,
this was done through: https://gitlab.com/gitlab-org/gitaly/merge_requests/779
As the new implemented got moved around a bit, the whole RevList class
got removed.
Part of https://gitlab.com/gitlab-org/gitaly/issues/1233
Prior to this change, most the commits counted were done through Gitaly.
This removes the last point where this wasn't the case.
This makes the `rugged_count_commits` method obsolete, with its tests.
Closes https://gitlab.com/gitlab-org/gitaly/issues/315
Initially expected this to be a full blown migration, but that wasn't
the case as we could fetch to a reference that would be deleted after
the instanciation of the Diff object.
This is a racy RPC, as #delete_refs is called and garbage collections
runs right than, the commit can't be found later.
Closes https://gitlab.com/gitlab-org/gitaly/issues/771
Gitaly itself hold very little state, other than the data on disk. This
limits the interfaces to set feature flags. Gitaly now has the ability
to interpret the request metadata to check for feature flags.
https://gitlab.com/gitlab-org/gitaly/merge_requests/704
This allows clients control on the Gitaly server, and given that Rails
has an internal chatops interface to set these values this was chosen as
solution.
Known limitation right now, is that this package doesn't support the opt
out that other Gitaly features do.
OPT_OUT status has been removed, and alternative implementation removed.
Also checks if the repository exists before executing the checksum RPC
to guard against NotFound errors.
Closesgitlab-org/gitaly#1105
Prior to this change, this was done through unicorn. In theory this
could time out. Workhorse has been sending these raw patches and diffs
for a long time and is stable in doing so.
Added bonus is the fact that `Commit#to_patch` can be removed.
`Commit#to_diff` too, which closes
https://gitlab.com/gitlab-org/gitaly/issues/324
Closes https://gitlab.com/gitlab-org/gitaly/issues/1196
Fetching commits through #lookup uses rugged, and this is phased out.
Also adds a short circuit in the `feature_enabled?` Gitaly method to
allow Gitaly to be used when objects are set up with `let` before an
example is run.
Gitaly-Ruby is not a Rails application, which made the method not exist
after the vendor. This patch uses direct indexing, which has the same
properties that `#first` has.
Found while working on: https://gitlab.com/gitlab-org/gitaly/merge_requests/699
Storage shards should only be used, the exact paths are the
responsibilty of Gitaly. This will create deprecation warning, and in
this instance the path is initialized, but never used so it could be
removed.
Has been in opt out for 5 months, and within GitLab been in production
for longer than that.
No code needs to be migrated as this is implemented in GoLang over at
Gitaly.
Closes https://gitlab.com/gitlab-org/gitaly/issues/314
This is called repeatedly when viewing a merge request, and this should
improve performance significantly by avoiding shelling out to git every time.
This should help https://gitlab.com/gitlab-com/infrastructure/issues/4027.
Direct disk access is impossible on environments where components run in
their own container. There for constructing the path should not be done
except for exceptional cases.
One of the considerations, instead of logging, was to use Sentry. For
now I've chosen not to use this as the impact of this change is hard to
determine. Getting this in now will allow us to checkout the impact on
dev.gitlab.org and staging.
Clients can now request the attributes from `$GIT_DIR/info/attributes`
through Gitaly. The Gitaly migration is described in gitlab-org/gitaly#1082.
The parser algorithm was implemented in a way it could handle both file
contents or a File handle, and both were already tested.
Other than that, using the boy scout rule, I've removed a class,
InfoAttributes, as it was delegating everything to the parser and
therefor wasn't really needed in my opinion.
Repository archives are always named `<project>-<ref>-<sha>` even if
the ref is a commit. A consequence of always including the sha even
for tags is that packaging a release is more difficult because both
the ref and sha must be known by the packager.
- add `<project>/-/archive/<ref>/<filename>.<format>` route using the
`-` separator to prevent namespace collisions. If the filename is
`<project>-<ref>` or the ref is a sha, the sha will be omitted,
otherwise the default filename will be used.
- deprecate previous archive route `repository/<ref>/archive`
If the ref hasn't been fetched into the target repository yet, this will fail
with a Rugged::ReferencError (assuming we're not using Gitaly). We should handle
this in the same way as a missing ref.
Repository archives are always named `<project>-<ref>-<sha>` even if
the ref is a commit. A consequence of always including the sha even
for tags is that packaging a release is more difficult because both
the ref and sha must be known by the packager.
- add append_sha option (defaults true) to provide a method for
toggling this feature.
Support added to GitLab Workhorse by gitlab-org/gitlab-workhorse!232
When we added caching, this meant that calling `can_be_resolved_in_ui?` didn't
always call `lines`, which meant that we didn't get the benefit of the
side-effect from that, where it forced the conflict data itself to UTF-8.
To fix that, make this explicit by separating the `raw_content` (any encoding)
from the `content` (which is either UTF-8, or an exception is raised).
Fixes gitaly#1057
The old code was calling LastCommitForPath to extract a commit ID _then_
call FindCommit to get a commit it already had in the first place!
By default, --prune is added to the command-line of a `git fetch` operation,
but for repositories with many references this can take a long time to run. We
shouldn't need to run --prune the first time we fetch a new repository.
Adds a test where a branch name is also a valid commit id. Git, the
binary should create an error message which is difficult to parse and
leading to errors later, as seen in: gitlab-org/gitlab-ce#43222
To catch these cases in the future,
gitlab-test@1942eed5cc108b19c7405106e81fa96125d0be22 was created. Which
a branch name matching the commit
When the applied diff contains UTF-8 or some other encoded data, the diff
returned back from the git process may be in ASCII-8BIT format. Writing this
data to stdin may fail if the data because stdin expects this data to be in
UTF-8. By switching the output to binmode, we ensure that the diff will
always be written as-is.
Closesgitlab-org/gitlab-ee#4960
The refs hash is used to determine what branches and tags have a commit
as head in the network graph. The previous implementation depended on
Rugged#references. The problem with this implementation was that it
depended on rugged, but also that it iterated over all references and
thus loading more data than needed if for example the project uses CI/CD
environments, Pipelines, or Merge Requests.
Given only refs are checked the network cares about the GraphHelper#refs
method has no need to reject those, simplifying the method.
Closesgitlab-org/gitaly#880
Douglas Barbosa Alexandre
Stan Hu
Nick Thomas
Markus Koller
Igor
John Cai
Mayra Cabrera
Shinya Maeda
Paul Okstad
GitalyBot
Luke Duncalfe
Patrick Bajao
Zeger-Jan van de Weg
Douwe Maan
Bob Van Landuyt
Francisco Javier López
Oswaldo Ferreira
Rémy Coutable
Jacob Vosmaer
Dmitriy Zaporozhets
Sean McGivern
Luke Duncalfe
Mario de la Ossa
Yorick Peterse
Andrew Newdigate
Oswaldo Ferreira
Kamil Trzciński
Douwe Maan
Takuya Noguchi
gfyoung
Tiago Botelho
Alejandro Rodríguez
Grzegorz Bizon
Michael Kozono
Robert Speicher
Rubén Dávila
Sean McGivern
Lin Jen-Shin
Grzegorz Bizon
Robert Speicher
Mark Chao
Balasankar "Balu" C
Kim Carlbäcker
Sergey Sinev
Ahmad Sherif
🙈 jacopo beschi 🙉
James Ramsay
Eric Eastwood
Jasper Maes
James Edwards-Jones
Micaël Bergeron