e407c2d18f
Don't allow Repository#log with limit zero
2018-01-29 15:27:46 +00:00
3f79376c76
Set alternate object directories in run_git
2018-01-29 08:51:25 +00:00
af95ba2fb4
Merge branch 'feature/migrate-repo-restorer-to-gitaly' into 'master'
...
Migrate restoring repo from bundle to Gitaly
Closes gitaly#946
See merge request gitlab-org/gitlab-ce!16623
2018-01-25 19:03:51 +00:00
26c4dea7ad
Merge branch 'lint-rugged' into 'master'
...
Add a lint check to restrict use of Rugged
See merge request gitlab-org/gitlab-ce!16656
2018-01-25 17:52:01 +00:00
5c68c839c0
Merge branch 'contrain-run-git' into 'master'
...
Make Gitlab::Git::Repository#run_git private
See merge request gitlab-org/gitlab-ce!16622
2018-01-25 16:17:02 +00:00
9e6ce87727
Make Gitaly RepositoryExists opt-out
2018-01-25 13:59:35 +00:00
6d6f7536bd
Look for rugged with static analysis
2018-01-25 14:05:11 +01:00
eab9ffeff6
Merge branch 'revert-430b3f0e' into 'master'
...
Revert "Merge branch 'remove-rugged-walk' into 'master'"
See merge request gitlab-org/gitlab-ce!16676
2018-01-25 11:15:07 +00:00
d2bf53942f
Merge branch 'jej/gitattributes-check-at-ref' into 'master'
...
Can parse root .gitattributes file for a ref
See merge request gitlab-org/gitlab-ce!16418
2018-01-25 11:01:36 +00:00
7363aa0b9d
Revert "Merge branch 'remove-rugged-walk' into 'master'"
...
This reverts merge request !16525
2018-01-25 11:03:29 +01:00
49e9347e73
Make Gitlab::Git::Repository#run_git private
2018-01-25 11:00:15 +01:00
29fa930baf
Migrate .batch_lfs_pointers to Gitaly
...
Closes gitaly#921
2018-01-24 19:02:51 +01:00
6a28967c14
Migrate restoring repo from bundle to Gitaly
...
Closes gitaly#946
2018-01-24 19:02:32 +01:00
24625323a8
Migrate repository bundling to Gitaly
...
Closes gitaly#929
2018-01-24 12:10:31 +01:00
0683d31ab1
Can parse root .gitattributes file for a ref
2018-01-24 05:15:37 +00:00
09fc32173f
Merge branch 'gitaly-user-commit-files' into 'master'
...
Incorporate Gitaly's OperationService.UserCommitFiles RPC
Closes gitaly#890
See merge request gitlab-org/gitlab-ce!16307
2018-01-23 09:34:59 +00:00
2e0951e93f
Incorporate Gitaly's OperationService.UserCommitFiles RPC
2018-01-22 13:34:51 -03:00
f344f7cecc
Merge branch 'gitaly-write-ref-error-handling' into 'master'
...
Wrap Rugged-exceptions in Gitlab::Git::Repository#write_ref
See merge request gitlab-org/gitlab-ce!16570
2018-01-22 16:09:45 +00:00
728d7e0cf7
Move error-handling to lib/gitlab/git
2018-01-22 15:40:38 +01:00
31268b1b1a
_never_ unset gitlab.full_path
2018-01-19 18:08:07 +01:00
e1925f6d68
Move rugged-call from Project#write_repository_config to Git::Repository#write_config
2018-01-19 11:45:06 +01:00
58aa32bcea
Wrap Rugged-exceptions in Gitlab::Git::Repository#write_ref
2018-01-18 21:31:40 +01:00
fa52038731
Remove Rugged::Walker from Repository#log
2018-01-18 14:05:14 +00:00
f4de7309ab
Remove unused methods from Gitlab::Git
2018-01-17 09:54:27 +00:00
16e895801e
Merge branch 'feature/migrate-is-rebase-in-progress-to-gitaly' into 'master'
...
Migrate rebase_in_progress? to Gitaly
Closes gitaly#866
See merge request gitlab-org/gitlab-ce!16286
2018-01-11 22:27:16 +00:00
587242656f
Merge branch 'feature/migrate-rebase-to-gitaly' into 'master'
...
Migrate Gitlab::Git::Repository#rebase to Gitaly
Closes gitaly#863
See merge request gitlab-org/gitlab-ce!16259
2018-01-11 22:24:50 +00:00
729f05f0e3
Adds Rubocop rule for line break around conditionals
2018-01-11 16:34:01 +00:00
ccb3fe1f42
Merge branch 'feature/migrate-merged-branch-names-to-gitaly' into 'master'
...
Migrate merged_branch_names to Gitaly
Closes gitaly#851
See merge request gitlab-org/gitlab-ce!16157
2018-01-11 15:22:49 +00:00
831019412f
Migrate merged_branch_names to Gitaly
...
Closes gitaly#851
2018-01-11 14:52:15 +01:00
42265d445f
Migrate Gitlab::Git::Repository#rebase to Gitaly
...
Closes gitaly#863
2018-01-10 13:28:10 +01:00
92d62ff69d
Migrate rebase_in_progress? to Gitaly
...
Closes gitaly#866
2018-01-10 13:24:29 +01:00
657065b73e
Client-prep Gitlab::Git::Repository#write_ref
2018-01-09 15:20:44 +01:00
d42c0ecc02
Merge branch 'gitaly-fetch-internal-remote' into 'master'
...
Incorporate RemoteService.FetchInternalRemote Gitaly RPC
Closes gitaly#857
See merge request gitlab-org/gitlab-ce!16140
2018-01-08 10:42:35 +00:00
c2b7147c2f
Fix error 500 when viewing commit and merge request diffs
...
Due to the refactoring in !16082 , `Blob#batch` no longer falls back
to a default `blob_size_limit`. Since `Repository#batch_blobs` was using
a default `nil` value, this would cause issues in the `Blob#find_by_rugged`
method.
This fix here is to be consistent and use a non-nil default value in
`Repository#batch_blobs`.
The problem was masked in development and tests because Gitaly is always
enabled by default for all features.
Closes #41735
2018-01-07 04:49:07 -08:00
9ff44c2986
Incorporate RemoteService.FetchInternalRemote Gitaly RPC
2018-01-05 15:45:24 -03:00
0b07be594e
Move git operations for multi_action into Gitlab::Git
2018-01-05 15:17:53 -03:00
33c5630b02
Use --left-right and --max-count for counting diverging commits
2018-01-05 16:52:06 +00:00
4dd0c55f05
Merge branch 'fix/prepare-rebase-to-be-migrated-to-gitaly' into 'master'
...
Prepare Gitlab::Git::Repository#rebase for Gitaly migration
See merge request gitlab-org/gitlab-ce!16192
2018-01-05 10:44:24 +00:00
58a705aaac
Merge branch 'ce-gitaly-remote-mirror-prep' into 'master'
...
[CE] Move git operations for UpdateRemoteMirrorService into Gitlab::Git
See merge request gitlab-org/gitlab-ce!16023
2018-01-04 16:02:22 +00:00
0ba0f9de08
Prepare Gitlab::Git::Repository#rebase for Gitaly migration
2018-01-04 12:39:45 +01:00
062ea80e64
Merge branch 'feature/add-max-count-to-count-commits-rpc' into 'master'
...
Add support for max_count option to Git::Repository#count_commits
See merge request gitlab-org/gitlab-ce!16145
2018-01-04 10:58:47 +00:00
1c458d17d7
Move delete_remote_branches from Gitlab::Shell to Gitlab::Git::Repository
2018-01-03 23:07:38 -03:00
43308bd826
Move push_remote_branches from Gitlab::Shell to Gitlab::Git::Repository
2018-01-03 23:06:34 -03:00
57906a5322
Merge branch 'gitaly-conflict-resolver' into 'master'
...
Gitaly conflict resolver
Closes gitaly#813
See merge request gitlab-org/gitlab-ce!15755
2018-01-03 08:39:16 +00:00
0ea75205df
Merge branch 'gitaly-opt-out' into 'master'
...
Unmark Gitaly features as OPT_OUT until the gRPC proxy issue is fixed
Closes #38333
See merge request gitlab-org/gitlab-ce!16155
2017-12-28 16:37:55 +00:00
7ed28b9069
Unmark Gitaly features as OPT_OUT until the gRPC proxy issue is fixed
2017-12-28 12:09:34 -03:00
04b987563a
Fix feature flags for Gitaly's RemoteService RPCs
2017-12-27 20:14:36 +00:00
0aa87bbe13
Incorporate ConflictsService.ResolveConflicts Gitaly RPC
2017-12-27 15:14:25 -03:00
351f205c05
Incorporate ConflictsService.ListConflictFiles Gitaly RPC
2017-12-27 15:12:30 -03:00
13932b0b12
Add support for max_count option to Git::Repository#count_commits
2017-12-27 14:16:24 +01:00
87bae6d42f
Merge branch 'gitaly-remotes' into 'master'
...
Incorporate Gitaly's RemoteService RPCs
Closes gitaly#852
See merge request gitlab-org/gitlab-ce!16060
2017-12-25 10:43:04 +00:00
2694355bb6
Incorporate Gitaly's RemoteService RPCs
2017-12-22 20:30:08 -03:00
7354c9f75d
Remove unused method `remote_exists?`
2017-12-22 20:30:07 -03:00
a2d39b8010
Use gitlab-styles
...
Signed-off-by: Rémy Coutable <remy@rymai.me>
2017-12-22 08:34:44 -06:00
28fba5ed99
Revert "Merge branch 'repo-write-ref-client-prep' into 'master'"
...
This reverts merge request !15712
2017-12-20 18:29:52 +00:00
44f0b2c4be
Merge branch '41016-import-gitlab-shell-projects' into 'master'
...
Import gitlab-shell gitlab_projects.rb as at v5.10.2
Closes #41016
See merge request gitlab-org/gitlab-ce!15853
2017-12-15 09:54:42 +00:00
ee9ad1b1f6
Merge branch 'feature/migrate-merge-base-to-gitaly' into 'master'
...
Migrate Gitlab::Git::Repository#merge_base_commit to Gitaly
Closes gitaly#808
See merge request gitlab-org/gitlab-ce!15770
2017-12-14 18:58:38 +00:00
4b785df27b
Import gitlab_projects.rb from gitlab-shell
...
By importing this Ruby code into gitlab-rails (and gitaly-ruby), we avoid
200ms of startup time for each gitlab_projects subprocess we are eliminating.
By not having a gitlab_projects subprocess between gitlab-rails / sidekiq and
any git subprocesses (e.g. for fork_project, fetch_remote, etc, calls), we can
also manage these git processes more cleanly, and avoid sending SIGKILL to them
2017-12-14 16:00:04 +00:00
835a5db376
Migrate Gitlab::Git::Repository#merge_base_commit to Gitaly
...
Closes gitaly#808
2017-12-13 17:40:19 +01:00
b655a4a790
Merge branch 'repo-write-ref-client-prep' into 'master'
...
Move Repository#write_ref to Git::Repository#write_ref
Closes gitaly#793
See merge request gitlab-org/gitlab-ce!15712
2017-12-12 16:36:58 +00:00
dad4c0b645
Move Repository#write_ref to Git::Repository#write_ref
2017-12-12 16:05:38 +01:00
6930fa3102
Merge branch 'gitaly-mirroring-prep' into 'master'
...
Gitaly mirroring migration prep
Closes gitaly#816
See merge request gitlab-org/gitlab-ce!15775
2017-12-11 20:50:07 +00:00
41b0c0e9be
Migrate Git::Repository#fsck to Gitaly
2017-12-08 00:27:11 +00:00
03ac8d5d0b
Remove Rugged::Repository#empty?
2017-12-07 15:33:30 +00:00
95009cefc2
Unify mirror addition operations to prepare for Gitaly migration
2017-12-06 19:08:29 -03:00
885a4da208
Add feature flag to use gitaly-ssh mirroring when cloning internal repos
...
This also allows us to simplify the naming since we can make some
fetching methods private.
2017-12-06 18:41:45 -03:00
887a373916
Migrate Gitlab::Git::Repository#revert to Gitaly
...
Closes gitaly#780
2017-12-05 14:09:15 +01:00
515c619c18
Merge branch 'ce-gitaly-worktree-refactor' into 'master'
...
[CE] Move git operations for squash and rebase into Gitlab::Git
See merge request gitlab-org/gitlab-ce!15673
2017-12-05 11:46:17 +00:00
f3a3bd50ea
Merge branch 'feature/migrate-cherry-pick-to-gitaly' into 'master'
...
Migrate Gitlab::Git::Repository#cherry_pick to Gitaly
Closes gitaly#737
See merge request gitlab-org/gitlab-ce!15517
2017-12-04 19:20:01 +00:00
f91c5c5bbe
Backport Squash/Rebase refactor from EE
2017-12-04 16:04:53 -03:00
8be9567c64
Migrate Gitlab::Git::Repository#cherry_pick to Gitaly
...
Closes gitaly#737
2017-12-04 19:21:07 +01:00
12f61e0d2c
Move SingleRepositoryWorker#fsck into Gitlab::Git::Repository
2017-12-04 15:57:20 +01:00
3e78b5ae8f
Merge branch 'sh-fix-root-ref-repository' into 'master'
...
Gracefully handle case when repository's root ref does not exist
Closes #40615
See merge request gitlab-org/gitlab-ce!15678
2017-12-01 11:02:32 +00:00
66127221fe
Gracefully handle case when repository's root ref does not exist
...
This was failing regularly with an Error 500 when the API branches endpoint
was used.
Closes #40615
2017-11-30 23:21:44 -08:00
4925ec5087
We could simply count the commits
2017-11-30 16:03:28 +08:00
3c6a4d6363
Ensure MRs always use branch refs for comparison
...
If a merge request was created with a branch name that also matched a tag name,
we'd generate a comparison to or from the tag respectively, rather than the
branch. Merging would still use the branch, of course.
To avoid this, ensure that when we get the branch heads, we prepend the
reference prefix for branches, which will ensure that we generate the correct
comparison.
2017-11-28 17:01:38 +00:00
dd11f0e053
Merge branch 'dm-cleanup-fetch-and-mirror-methods' into 'master'
...
Clean up repository fetch and mirror methods
See merge request gitlab-org/gitlab-ce!15424
2017-11-23 17:01:47 +00:00
0e6beaf50c
Clean up repository fetch and mirror methods
2017-11-23 16:59:50 +01:00
7df1cb528e
Move identical merged branch check to merged_branch_names
2017-11-23 18:26:41 +08:00
571f1dda17
Add FetchSourceBranch Gitaly call
2017-11-22 10:19:42 +00:00
f9565e3039
Batchload blobs for diff generation
...
After installing a new gem, batch-loader, a construct can be used to
queue data to be fetched in bulk. The gem was also introduced in both
gitlab-org/gitlab-ce!14680 and gitlab-org/gitlab-ce!14846 , but those mrs
are not merged yet.
For the generation of diffs, both the old blob and the new blob need to
be loaded. This for every file in the diff, too. Now we collect all
these so we do 1 fetch. Three `.allow_n_plus_1_calls` have been removed,
which I expect to be valid, but this needs to be confirmed by a full CI
run.
Possibly closes:
- https://gitlab.com/gitlab-org/gitlab-ce/issues/37445
- https://gitlab.com/gitlab-org/gitlab-ce/issues/37599
- https://gitlab.com/gitlab-org/gitlab-ce/issues/37431
2017-11-21 13:53:26 +01:00
4b78f70fc4
Merge branch 'gitaly-delete-refs' into 'master'
...
Incorporate Gitaly's RefService.DeleteRefs RPC
Closes gitaly#740
See merge request gitlab-org/gitlab-ce!15460
2017-11-20 15:20:40 +00:00
202ab62874
Merge branch 'fix-conflict-highlighting' into 'master'
...
Fix conflict highlighting
See merge request gitlab-org/gitlab-ce!15463
2017-11-20 10:17:43 +00:00
38730a2d07
Incorporate Gitaly's RefService.DeleteRefs RPC
2017-11-17 18:08:50 -03:00
64a9e53bd1
Fix conflict highlighting
...
Conflicts used to take a `Repository` and pass that to
`Gitlab::Highlight.highlight`, which would call `#gitattribute` on the
repository. Now they use a `Gitlab::Git::Repository`, which didn't have that
method defined - but defining it on `Gitlab::Git::Repository` does make it
available on `Repository` through `method_missing`, so we can do that and both
cases will work.
2017-11-17 17:57:48 +00:00
181cd299f9
Adds Rubocop rule for line break after guard clause
...
Adds a rubocop rule (with autocorrect) to ensure line break after guard clauses.
2017-11-16 17:58:29 +01:00
3eff7b6773
Merge branch 'zj-move-repo-exists-opt-out' into 'master'
...
Repository Exists check is OPT_OUT for Gitaly
See merge request gitlab-org/gitlab-ce!15287
2017-11-13 10:28:12 +00:00
de301d13bb
Prepare Repository#fetch_source_branch for migration
2017-11-10 14:45:23 +00:00
90d74ce2df
Repository Exists check is OPT_OUT for Gitaly
...
Moving more git operations to be executed by Gitaly, now the check if a
repository exists is an opt out endpoint.
Can be disabled, for the time being, by performing in the rails console:
> Feature.get('gitaly_repository_exists').disable
=> true
Part of gitlab-org/gitaly#314
2017-11-09 09:32:21 +01:00
4dfe26cd8b
Rewrite the GitHub importer from scratch
...
Prior to this MR there were two GitHub related importers:
* Github::Import: the main importer used for GitHub projects
* Gitlab::GithubImport: importer that's somewhat confusingly used for
importing Gitea projects (apparently they have a compatible API)
This MR renames the Gitea importer to Gitlab::LegacyGithubImport and
introduces a new GitHub importer in the Gitlab::GithubImport namespace.
This new GitHub importer uses Sidekiq for importing multiple resources
in parallel, though it also has the ability to import data sequentially
should this be necessary.
The new code is spread across the following directories:
* lib/gitlab/github_import: this directory contains most of the importer
code such as the classes used for importing resources.
* app/workers/gitlab/github_import: this directory contains the Sidekiq
workers, most of which simply use the code from the directory above.
* app/workers/concerns/gitlab/github_import: this directory provides a
few modules that are included in every GitHub importer worker.
== Stages
The import work is divided into separate stages, with each stage
importing a specific set of data. Stages will schedule the work that
needs to be performed, followed by scheduling a job for the
"AdvanceStageWorker" worker. This worker will periodically check if all
work is completed and schedule the next stage if this is the case. If
work is not yet completed this worker will reschedule itself.
Using this approach we don't have to block threads by calling `sleep()`,
as doing so for large projects could block the thread from doing any
work for many hours.
== Retrying Work
Workers will reschedule themselves whenever necessary. For example,
hitting the GitHub API's rate limit will result in jobs rescheduling
themselves. These jobs are not processed until the rate limit has been
reset.
== User Lookups
Part of the importing process involves looking up user details in the
GitHub API so we can map them to GitLab users. The old importer used
an in-memory cache, but this obviously doesn't work when the work is
spread across different threads.
The new importer uses a Redis cache and makes sure we only perform
API/database calls if absolutely necessary. Frequently used keys are
refreshed, and lookup misses are also cached; removing the need for
performing API/database calls if we know we don't have the data we're
looking for.
== Performance & Models
The new importer in various places uses raw INSERT statements (as
generated by `Gitlab::Database.bulk_insert`) instead of using Rails
models. This allows us to bypass any validations and callbacks,
drastically reducing the number of SQL queries and Gitaly RPC calls
necessary to import projects.
To ensure the code produces valid data the corresponding tests check if
the produced rows are valid according to the model validation rules.
2017-11-07 23:24:59 +01:00
045795d0d9
Merge branch 'remove-ensure-ref-fetched-from-controllers' into 'master'
...
removed the #ensure_ref_fetched from all controllers
Closes #36061
See merge request gitlab-org/gitlab-ce!15129
2017-11-06 17:10:18 +00:00
dea6d054cd
Encapsulate git operations for mirroring in Gitlab::Git
2017-11-03 14:33:24 -03:00
cd88fa8f80
removed the #ensure_ref_fetched from all controllers
...
also, I refactored the MergeRequest#fetch_ref method to express
the side-effect that this method has.
MergeRequest#fetch_ref -> MergeRequest#fetch_ref!
Repository#fetch_source_branch -> Repository#fetch_source_branch!
2017-11-03 08:13:11 -04:00
6a5e8f1046
Merge branch 'gitaly-ff-merge' into 'master'
...
Incorporate Gitaly's OperationService.UserFFBranch RPC
Closes gitaly#693
See merge request gitlab-org/gitlab-ce!15040
2017-11-02 16:11:53 +00:00
e65bbc0ca4
Merge branch 'jej/lfs-change-detection' into 'master'
...
Detect changes to LFS pointers for pruning and integrity check
See merge request gitlab-org/gitlab-ce!14785
2017-11-01 17:33:41 +00:00
fb3f9c6e50
Detect changes to LFS pointers for pruning and integrity check
...
Gitlab::Git::Blob.batch_lfs_metadata can be used to check for LFS pointers. It uses a lazy enumorator and filters by blob size
2017-11-01 16:41:05 +00:00
37cc50f843
Incorporate Gitaly's OperationService.UserFFBranch RPC
2017-10-31 16:28:09 -03:00
57d7ed05d9
Fetch the merged branches at once
2017-10-27 15:55:08 +00:00
efdf51d838
Make local_branches OPT_OUT
2017-10-26 09:52:52 +00:00
a64601b929
Move all rugged operation for ff_merge inside Gitlab::Git
...
We also delete some unused code related to the aforementioned feature.
2017-10-24 14:31:52 -03:00
Jacob Vosmaer (GitLab)
Robert Speicher
Rémy Coutable
Douwe Maan
Ahmad Sherif
James Edwards-Jones
Alejandro Rodríguez
Kim "BKC" Carlbäcker
🙈 jacopo beschi 🙉
Sean McGivern
Stan Hu
Lin Jen-Shin (godfat)
Dmitriy Zaporozhets
Grzegorz Bizon
Nick Thomas
Zeger-Jan van de Weg
Sean McGivern
Douwe Maan
Zeger-Jan van de Weg
Jacopo
Yorick Peterse
micael.bergeron
Andrew Newdigate