6a28967c14
Migrate restoring repo from bundle to Gitaly
...
Closes gitaly#946
2018-01-24 19:02:32 +01:00
24625323a8
Migrate repository bundling to Gitaly
...
Closes gitaly#929
2018-01-24 12:10:31 +01:00
fa52038731
Remove Rugged::Walker from Repository#log
2018-01-18 14:05:14 +00:00
f4de7309ab
Remove unused methods from Gitlab::Git
2018-01-17 09:54:27 +00:00
831019412f
Migrate merged_branch_names to Gitaly
...
Closes gitaly#851
2018-01-11 14:52:15 +01:00
d42c0ecc02
Merge branch 'gitaly-fetch-internal-remote' into 'master'
...
Incorporate RemoteService.FetchInternalRemote Gitaly RPC
Closes gitaly#857
See merge request gitlab-org/gitlab-ce!16140
2018-01-08 10:42:35 +00:00
9ff44c2986
Incorporate RemoteService.FetchInternalRemote Gitaly RPC
2018-01-05 15:45:24 -03:00
33c5630b02
Use --left-right and --max-count for counting diverging commits
2018-01-05 16:52:06 +00:00
58a705aaac
Merge branch 'ce-gitaly-remote-mirror-prep' into 'master'
...
[CE] Move git operations for UpdateRemoteMirrorService into Gitlab::Git
See merge request gitlab-org/gitlab-ce!16023
2018-01-04 16:02:22 +00:00
a0ae886135
Merge branch 'gitaly-merge-nil' into 'master'
...
Handle Gitaly aborted merge due to branch update
Closes gitaly#854
See merge request gitlab-org/gitlab-ce!16116
2018-01-04 11:01:02 +00:00
1c458d17d7
Move delete_remote_branches from Gitlab::Shell to Gitlab::Git::Repository
2018-01-03 23:07:38 -03:00
43308bd826
Move push_remote_branches from Gitlab::Shell to Gitlab::Git::Repository
2018-01-03 23:06:34 -03:00
449b59ec69
Better English
2018-01-03 12:54:47 +01:00
8cf0ea4469
Handle Gitaly aborted merge due to branch update
2018-01-03 12:02:37 +01:00
13932b0b12
Add support for max_count option to Git::Repository#count_commits
2017-12-27 14:16:24 +01:00
87bae6d42f
Merge branch 'gitaly-remotes' into 'master'
...
Incorporate Gitaly's RemoteService RPCs
Closes gitaly#852
See merge request gitlab-org/gitlab-ce!16060
2017-12-25 10:43:04 +00:00
2694355bb6
Incorporate Gitaly's RemoteService RPCs
2017-12-22 20:30:08 -03:00
7354c9f75d
Remove unused method `remote_exists?`
2017-12-22 20:30:07 -03:00
27c95364b5
Replace '.team << [user, role]' with 'add_role(user)' in specs
2017-12-22 19:18:28 +11:00
4b785df27b
Import gitlab_projects.rb from gitlab-shell
...
By importing this Ruby code into gitlab-rails (and gitaly-ruby), we avoid
200ms of startup time for each gitlab_projects subprocess we are eliminating.
By not having a gitlab_projects subprocess between gitlab-rails / sidekiq and
any git subprocesses (e.g. for fork_project, fetch_remote, etc, calls), we can
also manage these git processes more cleanly, and avoid sending SIGKILL to them
2017-12-14 16:00:04 +00:00
6930fa3102
Merge branch 'gitaly-mirroring-prep' into 'master'
...
Gitaly mirroring migration prep
Closes gitaly#816
See merge request gitlab-org/gitlab-ce!15775
2017-12-11 20:50:07 +00:00
03ac8d5d0b
Remove Rugged::Repository#empty?
2017-12-07 15:33:30 +00:00
885a4da208
Add feature flag to use gitaly-ssh mirroring when cloning internal repos
...
This also allows us to simplify the naming since we can make some
fetching methods private.
2017-12-06 18:41:45 -03:00
66127221fe
Gracefully handle case when repository's root ref does not exist
...
This was failing regularly with an Error 500 when the API branches endpoint
was used.
Closes #40615
2017-11-30 23:21:44 -08:00
dd11f0e053
Merge branch 'dm-cleanup-fetch-and-mirror-methods' into 'master'
...
Clean up repository fetch and mirror methods
See merge request gitlab-org/gitlab-ce!15424
2017-11-23 17:01:47 +00:00
0e6beaf50c
Clean up repository fetch and mirror methods
2017-11-23 16:59:50 +01:00
5951320075
Make sure repository is restored
2017-11-23 18:26:41 +08:00
7df1cb528e
Move identical merged branch check to merged_branch_names
2017-11-23 18:26:41 +08:00
4b78f70fc4
Merge branch 'gitaly-delete-refs' into 'master'
...
Incorporate Gitaly's RefService.DeleteRefs RPC
Closes gitaly#740
See merge request gitlab-org/gitlab-ce!15460
2017-11-20 15:20:40 +00:00
3f0c9e9708
Fix Gitlab::Git::Repository#remote_tags using unexisting variable
2017-11-20 09:08:09 +00:00
38730a2d07
Incorporate Gitaly's RefService.DeleteRefs RPC
2017-11-17 18:08:50 -03:00
de301d13bb
Prepare Repository#fetch_source_branch for migration
2017-11-10 14:45:23 +00:00
4dfe26cd8b
Rewrite the GitHub importer from scratch
...
Prior to this MR there were two GitHub related importers:
* Github::Import: the main importer used for GitHub projects
* Gitlab::GithubImport: importer that's somewhat confusingly used for
importing Gitea projects (apparently they have a compatible API)
This MR renames the Gitea importer to Gitlab::LegacyGithubImport and
introduces a new GitHub importer in the Gitlab::GithubImport namespace.
This new GitHub importer uses Sidekiq for importing multiple resources
in parallel, though it also has the ability to import data sequentially
should this be necessary.
The new code is spread across the following directories:
* lib/gitlab/github_import: this directory contains most of the importer
code such as the classes used for importing resources.
* app/workers/gitlab/github_import: this directory contains the Sidekiq
workers, most of which simply use the code from the directory above.
* app/workers/concerns/gitlab/github_import: this directory provides a
few modules that are included in every GitHub importer worker.
== Stages
The import work is divided into separate stages, with each stage
importing a specific set of data. Stages will schedule the work that
needs to be performed, followed by scheduling a job for the
"AdvanceStageWorker" worker. This worker will periodically check if all
work is completed and schedule the next stage if this is the case. If
work is not yet completed this worker will reschedule itself.
Using this approach we don't have to block threads by calling `sleep()`,
as doing so for large projects could block the thread from doing any
work for many hours.
== Retrying Work
Workers will reschedule themselves whenever necessary. For example,
hitting the GitHub API's rate limit will result in jobs rescheduling
themselves. These jobs are not processed until the rate limit has been
reset.
== User Lookups
Part of the importing process involves looking up user details in the
GitHub API so we can map them to GitLab users. The old importer used
an in-memory cache, but this obviously doesn't work when the work is
spread across different threads.
The new importer uses a Redis cache and makes sure we only perform
API/database calls if absolutely necessary. Frequently used keys are
refreshed, and lookup misses are also cached; removing the need for
performing API/database calls if we know we don't have the data we're
looking for.
== Performance & Models
The new importer in various places uses raw INSERT statements (as
generated by `Gitlab::Database.bulk_insert`) instead of using Rails
models. This allows us to bypass any validations and callbacks,
drastically reducing the number of SQL queries and Gitaly RPC calls
necessary to import projects.
To ensure the code produces valid data the corresponding tests check if
the produced rows are valid according to the model validation rules.
2017-11-07 23:24:59 +01:00
045795d0d9
Merge branch 'remove-ensure-ref-fetched-from-controllers' into 'master'
...
removed the #ensure_ref_fetched from all controllers
Closes #36061
See merge request gitlab-org/gitlab-ce!15129
2017-11-06 17:10:18 +00:00
dea6d054cd
Encapsulate git operations for mirroring in Gitlab::Git
2017-11-03 14:33:24 -03:00
88d2517e36
Add `Gitlab::Git::Repository#fetch` command
2017-11-03 14:33:24 -03:00
cd88fa8f80
removed the #ensure_ref_fetched from all controllers
...
also, I refactored the MergeRequest#fetch_ref method to express
the side-effect that this method has.
MergeRequest#fetch_ref -> MergeRequest#fetch_ref!
Repository#fetch_source_branch -> Repository#fetch_source_branch!
2017-11-03 08:13:11 -04:00
6a5e8f1046
Merge branch 'gitaly-ff-merge' into 'master'
...
Incorporate Gitaly's OperationService.UserFFBranch RPC
Closes gitaly#693
See merge request gitlab-org/gitlab-ce!15040
2017-11-02 16:11:53 +00:00
d6066870cc
Fix encoding issue with Repository.ls_files
2017-11-02 08:29:03 +00:00
e65bbc0ca4
Merge branch 'jej/lfs-change-detection' into 'master'
...
Detect changes to LFS pointers for pruning and integrity check
See merge request gitlab-org/gitlab-ce!14785
2017-11-01 17:33:41 +00:00
fb3f9c6e50
Detect changes to LFS pointers for pruning and integrity check
...
Gitlab::Git::Blob.batch_lfs_metadata can be used to check for LFS pointers. It uses a lazy enumorator and filters by blob size
2017-11-01 16:41:05 +00:00
37cc50f843
Incorporate Gitaly's OperationService.UserFFBranch RPC
2017-10-31 16:28:09 -03:00
57d7ed05d9
Fetch the merged branches at once
2017-10-27 15:55:08 +00:00
a64601b929
Move all rugged operation for ff_merge inside Gitlab::Git
...
We also delete some unused code related to the aforementioned feature.
2017-10-24 14:31:52 -03:00
0aff29f96b
Merge Merge Requests via Gitaly
2017-10-13 14:07:31 +02:00
05f46b820e
Merge branch 'fix/rugged-alternate-paths' into 'master'
...
Fix the format of rugged alternate directory list
Closes #39046
See merge request gitlab-org/gitlab-ce!14840
2017-10-12 15:37:14 +00:00
400557f0ba
Merge branch 'gitaly-fetch-remote' into 'master'
...
Add `Gitlab::Git::Repository#fetch` command
Closes gitaly#586
See merge request gitlab-org/gitlab-ce!14772
2017-10-12 14:53:32 +00:00
a24abf39bf
Fix the format of rugged alternate directory list
...
Fixes #39046
2017-10-12 14:20:04 +02:00
4378f56c7b
Pass git object dir attributes as relative paths to Gitaly
...
Fixes gitaly#629
2017-10-11 15:46:55 +02:00
1731934398
Add `Gitlab::Git::Repository#fetch` command
2017-10-09 18:25:24 -03:00
0ce6785851
Replaces `tag: true` into `:tag` in the specs
...
Replaces all the explicit include metadata syntax in the specs (tag:
true) into the implicit one (:tag).
Added a cop to prevent future errors and handle autocorrection.
2017-10-07 13:57:54 +02:00
147e2b21be
Let fetch_ref pull from Gitaly instead of from disk
2017-10-04 12:57:41 +02:00
79719cf003
Add OperationService.UserDeleteBranch Gitaly RPC
2017-10-03 21:06:34 -03:00
403712f06e
Make Repository#has_visible_content more efficient
2017-09-29 13:08:44 +00:00
b63e8d64c1
Handle error when fetching ref for MR with deleted source branch
...
If the ref doesn't exist, and the source branch is deleted, we can't get it back
easily. Previously, we ignored this error by shelling out, so replicate that
behaviour.
2017-09-28 11:11:10 +01:00
34eeac6108
Use Gitlab::Git's Popen on that module's code
...
This allows the current Gitaly migration to depend on less code outside
of the Gitlab::Git module
2017-09-20 13:21:54 +02:00
41ef94e777
Migrate creating/deleting a branch to Gitaly
2017-09-06 17:25:17 +02:00
e10437de40
Migrate Gitlab::Git::Repository#find_branch to Gitaly
2017-09-05 17:29:45 +00:00
c5553ce772
Use `git update-ref --stdin -z` to delete refs
2017-08-30 22:01:23 +08:00
51ceacb1fc
Add missing third argument to `Git::Repository#initialize` in spec
2017-08-24 17:34:05 -04:00
e46a3d2fda
Merge branch 'git-operation-user' into 'master'
...
Move GitHooksService into Gitlab::Git
See merge request !13739
2017-08-24 14:32:17 +00:00
fb49c94e49
Delegate Repository::branch_exists? and ref_exists? to Gitlab::Git
2017-08-24 09:20:04 +00:00
37904108b9
Fix inconsistent number of branches when remote branches are present
...
Users of project mirrors would see that the number of branches did not
match the number in the branch dropdown because remote branches were
counted when Rugged was in use. With Gitaly, only local branches
are counted.
Closes #36934
2017-08-23 22:07:32 -07:00
65f83941c3
Make gl_repository a G::G::Repository attribute
2017-08-23 10:45:20 +02:00
faaed0aa1f
Delete unused Gitlab::Git methods
2017-08-18 14:12:17 +02:00
aaa887feba
Client Implementation: RefService::RefExists
2017-08-18 09:43:58 +01:00
0da644072f
Merge branch 'malformed-submodule' into 'master'
...
Handle missing .gitmodules when getting submodule urls
See merge request !13509
2017-08-15 09:00:44 +00:00
a64760d6d0
Merge branch 'gitaly-404-commit-list-files' into 'master'
...
Migrate `Git::Repository.ls_files` to Gitaly
Closes gitaly#404
See merge request !13302
2017-08-14 15:44:36 +00:00
d9b6fd4fba
Handle missing .gitmodules when getting submodule urls
2017-08-14 10:53:02 +02:00
c8b802471b
Enable the RSpec/HookArgument cop and auto-correct offenses
2017-08-10 19:29:42 -04:00
7b10885046
Migrate Git::Repository.ls_files to Gitaly
2017-08-10 16:10:47 +02:00
c946ee1282
Enable the Layout/SpaceBeforeBlockBraces cop
...
Signed-off-by: Rémy Coutable <remy@rymai.me>
2017-08-09 11:52:22 +02:00
e363fbf71a
Move `deltas` and `diff_from_parents` logic to Gitlab::Git::Commit
...
This helps keep the abstraction layers simpler, and also keep the
interface of those methods consistent, in case of implementation
changes.
2017-08-07 23:33:40 -04:00
c21ae07e33
Refactor Gitlab::Git::Commit to include a repository
2017-08-07 22:34:34 -04:00
727b6d16e5
Merge branch 'master' into 'bvl-nfs-circuitbreaker'
...
# Conflicts:
# app/models/repository.rb
# spec/models/repository_spec.rb
2017-08-07 09:13:10 +00:00
9b21668663
Merge branch 'gitaly-renames-annotations' into 'master'
...
Gitaly deletions and annotations
See merge request !13280
2017-08-07 08:40:53 +00:00
3598e60bf2
Add a Circuitbreaker for storage paths
2017-08-04 15:38:48 +02:00
3a8f4a3460
Remove unnecessary 'raw' alias
2017-08-03 15:52:14 +02:00
29a07fd3a1
Rename 'commits_between' to 'rugged_commits_between'
2017-08-03 15:48:10 +02:00
215e0911e5
Migrate Gitlab::Git::Repository#count_commits to Gitaly
...
Closes gitaly#415
2017-08-02 00:33:04 +02:00
67de82cf5f
Add option to use CommitLanguages RPC
2017-08-01 10:48:46 +02:00
8e3f2ecfa9
Incorporate RefsService.FindAllBranches Gitaly RPC
2017-07-27 15:40:54 -04:00
8065adcc1e
Improve Gitlab::Git::Repository specs for #branches
2017-07-27 15:39:31 -04:00
432bb22308
Remove unused Gitlab::Git operations
2017-07-27 15:39:31 -04:00
a89f18bf2c
Renamed Gitaly services
2017-07-18 07:59:36 +00:00
8b31ecd3d9
Merge branch 'remove-git-search-files' into 'master'
...
Remove Repository#search_files
See merge request !12879
2017-07-17 11:13:49 +00:00
682b328787
Remove Repository#search_files
2017-07-14 18:17:31 +02:00
08b462b9b0
Remove deprecated #heads method
2017-07-14 15:11:42 +02:00
b2ecf0aa35
Change Git::Repository#log to return Commits
2017-07-13 14:54:21 +02:00
1dab640357
Merge branch 'feature/migrate-commit-count-to-gitaly' into 'master'
...
Migrate Gitlab::Git::Repository#commit_count to Gitaly
Closes gitaly#355
See merge request !12688
2017-07-07 15:21:05 +00:00
c393d88df3
Migrate Gitlab::Git::Repository#commit_count to Gitaly
...
Closes gitaly#355
2017-07-07 13:47:34 +02:00
9e4aef263a
Merge branch 'gitaly-submodule-url-for' into 'master'
...
Migrate #submodule_url_for to Gitaly
See merge request !12629
2017-07-05 11:25:39 +00:00
b67d1d64cc
Migrate #submodule_url_for to Gitaly
2017-07-05 11:09:03 +02:00
ec35a9e860
Remove unnecessary clear_stubs calls
2017-07-04 13:02:11 +02:00
8a62f304ef
Add a UTF-8 encoding matcher
2017-06-30 14:32:42 +02:00
7648f11381
Remove unnecessary contexts
2017-06-30 14:32:42 +02:00
d3bcf8ac2a
Fix gitaly ref encoding bugs
2017-06-28 16:15:59 +02:00
5dd6d9a9ec
Merge branch 'submodules-private' into 'master'
...
Make Gitlab::Git::Repository#submodules private
See merge request !12479
2017-06-28 09:25:45 +00:00
cf131bf713
Make Gitlab::Ggit::Repository#submodules private
2017-06-27 16:03:35 +02:00
144e37c667
Remove Gitlab::Git::Repository#find_all
2017-06-27 12:30:51 +02:00
Ahmad Sherif
Jacob Vosmaer (GitLab)
Sean McGivern
Alejandro Rodríguez
Lin Jen-Shin (godfat)
Dmitriy Zaporozhets
blackst0ne
Nick Thomas
Robert Speicher
Zeger-Jan van de Weg
Stan Hu
Douwe Maan
Douwe Maan
Yorick Peterse
micael.bergeron
Rémy Coutable
Kim Carlbäcker
James Edwards-Jones
Jacopo
Sean McGivern
Ahmad Sherif
Robert Speicher
Andrew Newdigate
Bob Van Landuyt