Commit Graph

114 Commits

Author SHA1 Message Date
Rémy Coutable b5526a2d13 Merge branch 'gl-version-backup-file' into 'master'
Add GitLab version to backup file name

See merge request !10901
2017-04-26 18:22:33 +00:00
Maxim Rydkin 6a541c7905 Decrease ABC threshold to 57.08 2017-04-26 10:54:11 +00:00
Achilleas Pipinellis 43f73107ca Expose GitLab version in backup file
Instead of doing hacks like http://stackoverflow.com/a/26082612/974710
2017-04-25 11:36:52 +02:00
Achilleas Pipinellis 4740c99451 Merge branch 'db-backup-ssl-support' into 'master'
Database SSL support for backup script.

Closes #1440

See merge request !9715
2017-04-24 13:06:00 +00:00
Guillaume c67271cd78 Database SSL support for backup script. 2017-04-05 18:10:17 +02:00
Jacob Vosmaer 65b68338e7 Fix custom hooks restore 2017-03-29 14:48:06 +02:00
Sean McGivern a5db7f5425 Merge branch '28447-hybrid-repository-storages' into 'master'
Update storage settings to allow extra values per shard

See merge request !9597
2017-03-07 16:55:03 +00:00
Rémy Coutable 8526c9f3c4 Merge branch 'backup_storage_class' into 'master'
Add storage class configuration option for Amazon S3 remote backups

See merge request !9599
2017-03-06 17:42:30 +00:00
Pawel Chojnacki d13669c98b Remove remnants of git annex 2017-03-03 18:33:47 +01:00
Alejandro Rodríguez 0b9d56f960 Update storage settings to allow extra values per shard
This will be necessary when adding gitaly settings. This version
doesn't make any functional changes, but allows us to include this
breaking change in 9.0 and add the needed extra settings in the future
with backwards compatibility
2017-03-03 12:13:30 -03:00
Jon Keys 7a7ec1ac0f Add storage class configuration option for Amazon S3 remote backups 2017-02-28 16:30:58 -05:00
Douwe Maan 87c9df29ce Don’t exclude some file in lib from rubocop 2017-02-23 09:32:42 -06:00
Drew Blessing d6584c94a4 # This is a combination of 2 commits.
# This is the 1st commit message:

Add `copy` backup strategy to combat file changed errors

The backup Rake task used to stream data directly from the live
data directory into the backup. Under many circumstances this worked
OK. However, really active instances would experience a 'file changed
as we read it' error - especially with data like the registry. This
now copies the data first, then compresses it. It will take a bit
more disk space while the backup is in progress, but it's a
necessary thing.

# The commit message #2 will be skipped:

#	Add env var
2017-02-21 16:03:54 -06:00
Dmitriy Zaporozhets 6676b4f0dd
Use Namespace#full_path instead of Namespace#path
Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>
2017-02-14 18:38:20 +02:00
Kamil Trzcinski 732a821d4f Fix specs 2017-01-31 22:50:40 +00:00
Kamil Trzcinski 120f9abaa1 Add GitLab Pages
- The pages are created when build artifacts for `pages` job are uploaded
- Pages serve the content under: http://group.pages.domain.com/project
- Pages can be used to serve the group page, special project named as host: group.pages.domain.com
- User can provide own 403 and 404 error pages by creating 403.html and 404.html in group page project
- Pages can be explicitly removed from the project by clicking Remove Pages in Project Settings
- The size of pages is limited by Application Setting: max pages size, which limits the maximum size of unpacked archive (default: 100MB)
- The public/ is extracted from artifacts and content is served as static pages
- Pages asynchronous worker use `dd` to limit the unpacked tar size
- Pages needs to be explicitly enabled and domain needs to be specified in gitlab.yml
- Pages are part of backups
- Pages notify the deployment status using Commit Status API
- Pages use a new sidekiq queue: pages
- Pages use a separate nginx config which needs to be explicitly added
2017-01-31 22:50:39 +00:00
Adam Niedzielski 82692ea214 Restore backup correctly when "BACKUP" environment variable is passed 2017-01-09 18:38:53 +01:00
Sean McGivern 2fc1e643a7 Fix Backup::Manager#remove_old 2016-12-07 13:27:44 +00:00
David Gerő 0fbb5a86db Add Human Readable Timestamp to backup tar file 2016-11-30 11:41:53 +01:00
tiagonbotelho 96f050fa34 Backups do not fail anymore when using tar on annex and custom_hooks
only.
2016-10-27 11:24:35 +01:00
Dirk Hörner 2e44284054 lib/backup: fix broken permissions when creating repo dir
This commit fixes a typo where the mode argument to FileUtils.mkdir()
would be passed in decimal rather than octal format, yielding bad
permissions.
2016-08-30 13:35:50 +02:00
bogdanvlviv 96ebc8c4f7 Use `File::exist?` instead of `File::exists?`
Since version ruby-2.2.0, method `File::exists?` is deprecated.
2016-08-11 13:54:45 +03:00
Stan Hu c6ff77d4b8 Fix backup restore
Set permissions of backup dir to g+s

Closes #20188
2016-07-25 06:23:07 -07:00
Alejandro Rodríguez 86359ec854 Refactor repository paths handling to allow multiple git mount points 2016-06-29 22:30:31 -04:00
Stan Hu f03df22815 Only create the backup directory if it is local
Closes #12710
2016-06-10 10:17:54 -07:00
Connor Shea 903946c78a
Replace colorize gem with rainbow.
Colorize is a gem licensed under the GPLv2, so we can’t use it in GitLab without relicensing GitLab under the terms of the GPL. Rainbow is licensed under the MIT license and does the exact same thing as Colorize, so Rainbow was added in place of Colorize.

The syntax is slightly different for Rainbow vs. Colorize, and was updated in accordance.

The gem is still a dependency of Spinach, so it’s included in the development/test environments, but won’t be packaged with the actual product, and therefore doesn’t require we relicense the product.

An attempt at relicensing Colorize was made, but didn’t succeed as the library owner never responded.

Rainbow library: https://github.com/sickill/rainbow
Relevant issue regarding licensing in GitLab's gems: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3775
2016-06-03 10:37:09 -06:00
Robert Speicher b826c9ff17 Create the specified remote directory during backup
This is idempotent, so there's no harm calling it if the directory
already exists.

Closes #12710
2016-05-31 13:29:37 -04:00
Kamil Trzcinski 718fe268c4 Improve after review 2016-05-24 22:52:31 -05:00
Kamil Trzcinski 12bd781031 Fix backups if registry is disabled 2016-05-23 14:18:39 -05:00
Kamil Trzcinski 143cd58c39 Added backup of container registry 2016-05-16 17:25:38 -05:00
Jeroen Nijhof 897f18032a Make sure there is a connection before using ActiveRecord 2016-02-08 17:31:24 +01:00
Marin Jankovski 3cc2b48a82 Backup LFS objects same as any upload. 2015-11-19 10:43:45 +01:00
Marin Jankovski bf3c0aafaa USe reject. 2015-11-18 18:14:13 +01:00
Marin Jankovski f5e3d93c28 Check which folders and archives should be packed before passing to tar command. 2015-11-18 13:01:35 +01:00
Kamil Trzcinski d0e3e823a2 Implement Build Artifacts
- Offloads uploading to GitLab Workhorse
- Use /authorize request for fast uploading
- Added backup recipes for artifacts
- Support download acceleration using X-Sendfile
2015-11-10 12:51:50 +01:00
Robert Speicher d09d62b6b8 Replace all usages of `git` command with configurable binary path
Closes #3311
2015-11-03 17:11:09 -05:00
Robert Speicher 60c5b52ee9 Explicitly require backup/files 2015-11-02 15:04:43 -05:00
Jacob Vosmaer 58260a0327 Do no rely on basename of builds, uploads 2015-10-16 17:16:17 +02:00
Jacob Vosmaer 901f544578 Remove superfluous mkdir -p 2015-10-06 15:38:21 +02:00
Jacob Vosmaer 7d58489fd9 Remove unused variable 2015-10-06 15:22:08 +02:00
Jacob Vosmaer e789644783 Keep old path: db/database.sql.gz
Documentation elsewhere refers to this internal path, let's keep
it.
2015-10-06 15:21:15 +02:00
Jacob Vosmaer 7b71727c56 Remove old "files" tarball explicitly 2015-10-06 15:10:13 +02:00
Jacob Vosmaer 90ddf140b9 Reduce disk IO during SQL backup
By using light gzip compression we can save a lot of disk IO during
the backup.
2015-10-06 15:06:15 +02:00
Jacob Vosmaer d3734fbd89 Use tar for intermediate backup storage
During the backup we create an intermediate copy of two directories:
builds and uploads. Instead of creating many small files with 'cp
-r', we now use tar (and fast gzip) to create single intermediate
files. This saves on disk IO and disk space while creating a backup.
2015-10-06 14:43:19 +02:00
Paul Beattie 253d2320ab Add support for AWS S3 Server-Side Encryption support
This adds support for AWS S3 SSE with S3 managed keys, this means the
data is encrypted at rest and the encryption is handled transparently to
the end user as well as in the AWS Console. This is optional and not
required to make S3 uploads work.
2015-09-25 17:06:31 +01:00
Valery Sizov d7242054ba add pg_schema to backup config 2015-09-18 16:29:13 +03:00
Kamil Trzcinski 2f2b9f67c2 Fix backup tests 2015-09-16 00:06:41 +02:00
Kamil Trzcinski c9d914a392 Fix builds directory store 2015-09-15 22:53:38 +02:00
Kamil Trzcinski ed18e04bb3 Cleanup CI backup => migrate with GitLab 2015-09-15 20:14:21 +02:00
Jacob Vosmaer 4cd06867c7 Do not delete the SQL dump too early
The change in baa157926d broke backup
restore fucnctionality. This would not lead to data loss, but it
prevented the restore script from working. This bug exists only in
7.14.0 release candidate versions, not in 7.13.

Reported in https://github.com/gitlabhq/gitlabhq/issues/9571 .
2015-08-21 16:12:46 +02:00