Commit Graph

385 Commits

Author SHA1 Message Date
James Lopez 07009a1f48
Add Object Storage to GitLab project import
- Refactor uploads manager
- Refactor importer, update import spec
- Add more object storage specs
2018-08-02 11:50:19 +02:00
Zeger-Jan van de Weg 79a5d76801
Add repository languages for projects
Our friends at GitHub show the programming languages for a long time,
and inspired by that this commit means to create about the same
functionality.

Language detection is done through Linguist, as before, where the
difference is that we cache the result in the database. Also, Gitaly can
incrementaly scan a repository. This is done through a shell out, which
creates overhead of about 3s each run. For now this won't be improved.

Scans are triggered by pushed to the default branch, usually `master`.
However, one exception to this rule the charts page. If we're requesting
this expensive data anyway, we just cache it in the database.

Edge cases where there is no repository, or its empty are caught in the
Repository model. This makes use of Redis caching, which is probably
already loaded.

The added model is called RepositoryLanguage, which will make it harder
if/when GitLab supports multiple repositories per project. However, for
now I think this shouldn't be a concern. Also, Language could be
confused with the i18n languages and felt like the current name was
suiteable too.

Design of the Project#Show page is done with help from @dimitrieh. This
change is not visible to the end user unless detections are done.
2018-08-01 18:58:29 +02:00
Jacob Vosmaer (GitLab) 3f0e6d9205 More Gitaly cleanup: fetch_ref, allow disk access blocks 2018-07-27 08:43:19 +00:00
Jacob Vosmaer (GitLab) 4147a2dbbd Annotate Gitlab::Git::Repository#fetch_ref 2018-07-24 06:33:55 +00:00
Stan Hu 1025059d1f Rename WHITELISTED_FILENAMES => IGNORED_FILENAMES 2018-07-17 10:30:53 +02:00
Stan Hu eda8156e5c Fix symlink vulnerability in project import
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/49133
2018-07-17 10:30:52 +02:00
Sean McGivern 01ad732f8d Merge branch '48745-project-exports-fail-when-uploads-have-been-migrated-to-object-storage' into 'master'
Resolve "Project exports fail when uploads have been migrated to object storage"

Closes #48745

See merge request gitlab-org/gitlab-ce!20484
2018-07-16 10:28:21 +00:00
James Lopez 97ce0607d5
refactor code based on feedback 2018-07-16 11:45:11 +02:00
James Lopez 07ec2c7bd0
use fileuploader dynamic path method in uploads manager and add spec 2018-07-13 16:13:13 +02:00
James Lopez 1263367c3f
fix typo in uploads manager 2018-07-13 15:49:40 +02:00
James Lopez 16756c20de
add small comment to download method in uploads manager 2018-07-13 15:44:02 +02:00
James Lopez dae0d809ea
refactor uploads manager 2018-07-13 15:41:24 +02:00
James Lopez 4223a6fe45
refactor uploads manager to grab uploads in batches 2018-07-12 11:08:19 +02:00
James Lopez 8ea2027f43
fix uploads restorer missing path 2018-07-12 08:46:24 +02:00
Mark Chao a63bce1a4b Resolve "Rename the `Master` role to `Maintainer`" Backend 2018-07-11 14:36:08 +00:00
James Lopez e2b7880b9d
fix spec 2018-07-11 15:58:42 +02:00
James Lopez fa2e3162f8
added changelog, small refactor 2018-07-11 15:02:59 +02:00
James Lopez 3c31de7520
refactor uploads manager 2018-07-11 14:52:48 +02:00
James Lopez 414939c97c
fix uploads restorer 2018-07-11 12:02:01 +02:00
James Lopez 3a114c2d11
fix specs 2018-07-11 11:36:59 +02:00
James Lopez 7c9f21683c
add restore method 2018-07-11 10:24:59 +02:00
James Lopez a27d4d9e52
update uploads saver 2018-07-10 16:33:40 +02:00
James Lopez 2c1e66d4d2
Fix avatar saver and spec 2018-07-10 16:23:45 +02:00
James Lopez 874a4ff16a
add more object storage specs 2018-07-10 15:29:31 +02:00
James Lopez 4c7b120acf
add upload manager and spec 2018-07-10 11:40:39 +02:00
James Lopez 44be867d21
fix permissions 2018-07-06 18:07:53 +02:00
James Lopez a2bf164154
Update Import/Export to use object storage (based on aa feature flag) 2018-07-06 15:46:18 +02:00
James Lopez 4eebd231b8
update code based on feedback 2018-06-27 11:53:51 +02:00
James Lopez 792082948c
fix space 2018-06-25 12:04:23 +02:00
James Lopez 3a14ae3ae3
refactor code based on feedback 2018-06-25 09:42:07 +02:00
James Lopez ec394d4908
add some more comments 2018-06-22 11:27:52 +02:00
James Lopez 35d69ccf95
add more specs and refactor more relation factory code 2018-06-22 09:48:44 +02:00
James Lopez 3d3e441c91
refactor code based on feedback 2018-06-22 09:00:10 +02:00
James Lopez b0ddc55dc7
refactor code once again to fix IID issues 2018-06-20 16:18:40 +02:00
James Lopez 8522173e0a
refactor code once again to fix IID issues 2018-06-20 15:45:15 +02:00
James Lopez cc061cd1c6
fix some more edge cases 2018-06-20 11:57:22 +02:00
James Lopez 4e9b094b29
refactor finder 2018-06-20 10:27:40 +02:00
James Lopez 825c68e2ed
fix some edge cases 2018-06-19 17:08:02 +02:00
James Lopez a19e08feca
refactor relation factory 2018-06-19 15:27:30 +02:00
James Lopez b99bc6d380
add group finder spec and logic 2018-06-19 09:54:47 +02:00
Jacob Vosmaer (GitLab) a0808df0b6 Find and mark more Git disk access locations 2018-06-05 15:51:14 +00:00
Oswaldo Ferreira 54ad5fb8a2 Take two for MR metrics population background migration 2018-05-30 11:51:29 -03:00
Mayra Cabrera 0033e57269 Merge branch 'security-fj-import-export-assignment' into 'master'
[master] Arbitrary assignment of Project fields using "Import project" from a Gitlab tar.gz

Closes #2678

See merge request gitlab/gitlabhq!2378
2018-05-29 18:43:04 +00:00
Stan Hu 3126e89eb8
Add a unique and not null constraint on the project_features.project_id column
This commit has two migrations:

1. The first prunes duplicate rows in the project_features table and leaves
   the row with the highest ID.  Since the behavior was indeterministic before
   and depended on which row the database decided to use, this change at least
   makes the permissions consistent. For example, in some cases, the Wiki may
   have been disabled but enabled in another entry.

2. The second adds a non-null constraint on the project_features.project_id
   column.

Closes #37882

Fixes a significant part of gitlab-com/migration#408.

We found that we were overcounting Wikis because of these duplicates.
On GitLab.com, there are 56 rows with duplicate entries by project_id, and 16,661 rows with NULL project_id values.
2018-05-15 14:20:14 +02:00
Francisco Javier López 761d890a68
Blacklisting attributes in the project import process 2018-05-14 09:28:33 +02:00
Tiago Botelho 9a13059332 Backports every CE related change from ee-5484 to CE 2018-05-07 11:59:51 +02:00
Dylan Griffith 1b9c1ac3ad Merge branch 'master' into 10244-add-project-ci-cd-settings 2018-04-24 15:07:34 +10:00
Rubén Dávila 2d7efd9637 Don't include lfs_file_locks data in export bundle 2018-04-19 23:30:32 -05:00
Yorick Peterse 392c411bdc
Introduce new ProjectCiCdSetting
This model and the corresponding table will be used for storing settings
specific to CI/CD, starting with the "group_runners_enabled" boolean.

The model is called ProjectCiCdSetting and not ProjectCiCdSettings. The
project exporter doesn't like plural model names as it uses "classify"
which turns those into singular (so "ProjectCiCdSettings" becomes
"ProjectCiCdSetting", producing an error if said class is undefined).

The initial work in this commit was done by Dylan Griffith, with most of
the migration work being done by Yorick Peterse.
2018-04-16 14:05:35 +02:00
Francisco Javier López f20912df03 Extend API for importing a project export with overwrite support 2018-04-06 15:23:49 +00:00