Commit Graph

195 Commits

Author SHA1 Message Date
Nick Thomas 4aa76dddec
Remove dead MySQL code
None of this code can be reached any more, so it can all be removed
2019-07-23 16:53:03 +01:00
Dominic Renner ad29681cab Replaced config access
Replaced Settings with Gitlab.config for uniformity
2019-03-18 17:26:18 +00:00
Frederic Van Espen 094f219506 trim empty line 2019-03-07 13:34:17 +01:00
Frederic Van Espen 31dfc31aaa Merge branch 'master' into incremental-backups 2019-03-07 11:14:27 +01:00
Frederic Van Espen 562a1fc87d fix option test 2019-03-07 09:51:53 +01:00
Frederic Van Espen 935b698384 introduce optional rsyncable flag 2019-03-07 09:38:14 +01:00
Stan Hu b103f61450 Add convenience methods for creating project and Wiki repositories
This makes it easier to access other project arguments in the future.
2019-02-05 21:33:10 -08:00
Stan Hu 82b6e5378a Send project name with Gitaly repository requests
When hashed storage is in use, it's helpful to have the project
name associated with the request.

Closes https://gitlab.com/gitlab-org/gitaly/issues/1394
2019-02-05 21:33:09 -08:00
Jason Colyer c74820db07 Modify file restore to rectify tar issue
Fixed typo in run_pipeline! function
  -> Open3.pieline to Open3.pipeline
2019-01-17 10:07:29 -06:00
Jason Colyer f88b464bf3 Modify file restore to rectify tar issue
Adjusted run_pipeline! function
  to match rubocop syntax requirements
Adjusted run_pipeline! function
  to use variable instead of string interpolation
2019-01-17 10:07:29 -06:00
Jason Colyer 99b6f37fa5 Modify file restore to rectify tar issue
Changed run_pipeline! function to have use $ in regex
Changed run_pipeline! function to use Open3.pipeline
instead of Open3.pipeline_start
2019-01-17 10:07:06 -06:00
Jason Colyer 6de60ef295 Modified files.rb to handle tar error
Changed the run_pipeline! function to instead use Open3.pipeline_start
In doing so, the code is able to grab the last STDERR. In the case
where the error is the one from older tar versions (relating to ./),
it will not raise a Backup::Error. All other instances of command
failure will result in Backup::Error being raised.
2019-01-16 08:29:40 -06:00
Pepijn Van Eeckhoudt 1b3ffdf250 Add SSE-C key configuration option for Amazon S3 remote backups 2019-01-09 16:39:20 +00:00
Frederic Van Espen 3bb3f7ddd0 fix code style 2018-12-27 13:43:45 +01:00
Frederic Van Espen e8419fb5bb set the --rsyncable option for gzip 2018-12-26 15:35:20 +01:00
Frederic Van Espen 7a58eb2e16 Allow to override part of the backup filename 2018-12-26 15:33:11 +01:00
Michael Kozono d2d85c31fe Avoid extra storage bucket perm and query
Specifically, the `ListAllMyBuckets` permission.

This works if you know the directory exists.

See more:

* https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23981
* https://stackoverflow.com/a/12288581/1992201
2018-12-21 10:34:05 -08:00
Zeger-Jan van de Weg 89a407dc3b
Restore Object Pools when restoring an object pool
Pool repositories are persisted in the database, and when the DB is
restored, the data need to be restored on disk. This is done by
resetting the state machine and rescheduling the object pool creation.

This is not an exact replica of the state like at the time of the
creation of the backup. However, the data is consistent again.

Dumping isn't required as internally GitLab uses git bundles which
bundle all refs and include all objects in the bundle that they require,
reduplicating as more repositories get backed up. This does require more
data to be stored.

Fixes https://gitlab.com/gitlab-org/gitaly/issues/1355
2018-12-13 09:00:03 +01:00
gfyoung c858f70d07 Enable frozen string for lib/gitlab/*.rb 2018-10-22 07:00:50 +00:00
Balasankar "Balu" C 91ef6989d1
Add installation type to backup information file 2018-10-05 19:34:22 +05:30
gfyoung 3836d69119 Enable frozen string in lib/api and lib/backup
Partially addresses #47424.

Had to make changes to spec files because
stubbing methods on frozen objects is a mess
in RSpec and leads to failures:

https://github.com/rspec/rspec-mocks/issues/1190
2018-09-29 21:04:50 -07:00
Zeger-Jan van de Weg d7afed34c4
Remove feature flags from lib/backup
Moved to OPT_OUT in 7d14b725a0, Now, by
removing the feature gates, this is an mandatory feature.

Related issues:
- https://gitlab.com/gitlab-org/gitaly/issues/526
- https://gitlab.com/gitlab-org/gitaly/issues/1194

Closes https://gitlab.com/gitlab-org/gitaly/issues/749
Closes https://gitlab.com/gitlab-org/gitaly/issues/1212
Closes https://gitlab.com/gitlab-org/gitaly/issues/1195
2018-07-31 15:24:19 +02:00
Jacob Vosmaer 7d14b725a0 Move repo backup RPC's to opt-out 2018-06-25 16:21:54 +02:00
Ahmad Hassan 099eb8ae40 Restore custom hooks for wiki 2018-06-21 13:56:00 +02:00
Ahmad Hassan 9d7e73277e Move backup custom hooks up a level 2018-06-20 11:43:59 +02:00
Ahmad Hassan 5308b51ef4 Output done for wiki and repo 2018-06-19 16:24:16 +02:00
Ahmad Hassan 69de7b42cf Migrate repo backup to gitaly 2018-06-15 18:27:25 +02:00
Douwe Maan f068479e63 Merge branch 'use-restore-custom-hooks-gitaly' into 'master'
Use RestoreCustomHooks RPC in restore rake task

See merge request gitlab-org/gitlab-ce!19370
2018-06-08 11:49:13 +00:00
Ahmad Hassan 59a8c79f05 Use RestoreCustomHooks RPC in restore rake task 2018-06-07 14:26:20 +02:00
Jacob Vosmaer 6dfb0b4b31 Consistently use tar lookup function 2018-06-07 13:39:34 +02:00
Jacob Vosmaer (GitLab) a0808df0b6 Find and mark more Git disk access locations 2018-06-05 15:51:14 +00:00
Jacob Vosmaer (GitLab) b529d04b69 Use Gitaly's DeleteAllRepositories RPC during backup restore 2018-05-28 09:58:14 +00:00
Douwe Maan ba58a66a55 Merge branch 'migrate-restore-repo-to-gitaly' into 'master'
Use gitaly repository_service.CreateRepositoryFromBundle to restore (rake task)

Closes charts/gitlab#390

See merge request gitlab-org/gitlab-ce!18644
2018-05-24 18:28:23 +00:00
Stan Hu e6fca03bbd Fix backup creation and restore for specific Rake tasks 2018-05-24 14:58:25 +00:00
Ahmad Hassan 84e6ddb41f Remove unneeded gitaly_migrate function 2018-05-24 14:16:31 +02:00
Ahmad Hassan 9c4f967785 Take restoring custom hooks and the skip flag out of scope 2018-05-23 13:47:56 +02:00
Ahmad Hassan 80197bdc62 Remove gitaly_migrate blocks and replace by early return 2018-05-21 11:45:41 +02:00
Ahmad Hassan 125be1865b Move restore_custom_hooks to its seperate function 2018-05-18 11:55:54 +03:00
Ahmad Hassan 62f2988713 Return from prepare directory if gitaly skip is enabled 2018-05-17 15:36:58 +03:00
Ahmad Hassan 42cabcb1c4 Call path_to_repo only when using gitaly 2018-05-17 15:34:31 +03:00
DJ Mountney 889c62c1e3 Link to upstream gitaly issues for the missing restore paths 2018-05-16 13:28:15 +03:00
DJ Mountney b8b82aa50e Fix the paths for wiki restore 2018-05-15 16:43:04 +03:00
DJ Mountney d5bd61e82f Output project name before restoring each project repo 2018-05-15 16:43:04 +03:00
DJ Mountney 4ea9458bbe Remove hooks restore that is no longer necessary 2018-05-15 16:43:04 +03:00
DJ Mountney 2bcc324f26 Add back some of the non-gitaly restore functionality behind gates 2018-05-15 16:43:04 +03:00
Ahmad Hassan 0734da16f6 Wrap create_from_bundle in a begin rescue block 2018-05-15 16:43:04 +03:00
Ahmad Hassan 8457262125 Better repo restore progress logging 2018-05-15 16:43:04 +03:00
Ahmad Hassan fb54eb76ee Remove useless print line 2018-05-15 16:43:04 +03:00
Ahmad Hassan 3b5e9f0608 Use gitaly repository_service.CreateFromBundle to restore 2018-05-15 16:43:04 +03:00
Bob Van Landuyt e3ff928c75 Describe workaround when restore fails because of `Errno::EBUSY`
When `Errno::EBUSY` is raised during restore, this could indicate that
the directory being restored into is a mountpoint. In this case we
explain the user how to retry the restore.
2018-04-23 17:24:29 +02:00