Commit Graph

80 Commits

Author SHA1 Message Date
Micaël Bergeron e3d32e8a26 fix specs 2018-02-01 12:14:46 -05:00
Micaël Bergeron 2057a6acde port of 594e6a0a625^..f74c90f68c6 2018-02-01 12:14:46 -05:00
Nick Thomas 93ea3234df
Use the DatabaseCleaner 'deletion' strategy instead of 'truncation' 2018-01-24 17:25:55 +00:00
Rémy Coutable d8eb961d0b
Fix a migration spec messing up the MergeRequestDiff DB schema
Signed-off-by: Rémy Coutable <remy@rymai.me>
2018-01-23 17:42:37 +01:00
Sean McGivern 9fd6440d4a Prevent background migration spec from polluting later models
If the schema changes after 20171114162227 for any of these models, and specs
after this one use factories, then those factories will use the models with
outdated column information cached.

We shouldn't really use factories in migration specs, but this is a special case
because there is a lot of git-related setup code in the model that would be
painful to copy to the migration. Instead, we just manually reset the column
information for the models we could pollute.
2018-01-16 10:58:07 +00:00
Jan Provaznik e6a1db6d9e Denormalize commits count for merge request diffs
For each MR diff an extra 'SELECT COUNT()' is executed
to get number of commits for the diff. Overall time to get counts for
all MR diffs may be quite expensive. To speed up loading of MR info,
information about number of commits is stored in a MR diff's extra column.

Closes #38068
2018-01-10 20:40:02 +01:00
Jarka Kadlecová 153ea18301 Refactor matchers for background migrations 2018-01-05 13:40:31 +01:00
Douwe Maan d2891822e7 Merge branch 'mk-no-op-delete-conflicting-redirects' into 'master'
Prevent excessive DB load due to faulty DeleteConflictingRedirectRoutes background migration

See merge request gitlab-org/gitlab-ce!16205
2018-01-04 15:57:45 +00:00
Yorick Peterse 51562aafc7 Merge branch 'sh-validate-path-project-import' into 'master'
Avoid leaving a push event empty if payload cannot be created

See merge request gitlab-org/gitlab-ce!16214
2018-01-04 12:17:20 +00:00
Stan Hu 57dc5a521b Avoid leaving a push event empty if payload cannot be created
If the payload cannot be created for some reason, we could be left with a nil
push event payload, which causes Error 500s when viewing the dashboard. Guard
against this error and log when it happens.

Avoids problems seen in #38823
2018-01-03 22:49:02 -08:00
Michael Kozono f635277228 Make DeleteConflictingRedirectRoutes no-op
Both the post-deploy and background migration.
2018-01-03 12:23:20 -08:00
Oswaldo Ferreira bf8c20729b Cache merged and closed events data in merge_request_metrics table 2018-01-02 17:45:25 -02:00
Rémy Coutable 4af9d592c5 Replace factory_girl_rails with factory_bot_rails
I've followed the [upgrade guide](https://github.com/thoughtbot/factory_bot/blob/4-9-0-stable/UPGRADE_FROM_FACTORY_GIRL.md) and ran these two commands:

```
grep -e FactoryGirl **/*.rake **/*.rb -s -l | xargs sed -i "" "s|FactoryGirl|FactoryBot|"
grep -e factory_girl **/*.rake **/*.rb -s -l | xargs sed -i "" "s|factory_girl|factory_bot|"
```

Signed-off-by: Rémy Coutable <remy@rymai.me>
2017-12-14 15:01:55 +01:00
Douwe Maan 58017e17cc Merge branch 'bvl-backport-migration-spec-changes-to-ce' into 'master'
Create models directly in migration specs

See merge request gitlab-org/gitlab-ce!15874
2017-12-13 14:16:22 +00:00
Michael Kozono acde0e3f5c Fix spec by avoiding monkeypatching 2017-12-12 14:25:33 -08:00
Bob Van Landuyt 5ac98f9315 Create models directly in migration specs
Instead of using the factories. Since the factories might be using
columns that aren't available in the schema at version the particular
spec is running in.
2017-12-12 09:44:21 +01:00
Michael Kozono 24c348f0d1 Fix specs for MySQL 2017-12-06 12:20:28 -08:00
Michael Kozono 03cba8c0f1 Fix specs after rebase
Later migrations added fields to the EE DB which were used by factories which were used in these specs.

And in CE on MySQL, a single appearance row is enforced.

The migration and migration specs should not depend on the codebase staying the same.
2017-12-06 09:34:25 -08:00
Michael Kozono 869d08b581 Process normal paths in batch containing bad paths 2017-12-05 14:57:58 -08:00
Michael Kozono 77be7efcf5 Guarantee all IDs are included 2017-12-05 11:19:23 -08:00
Michael Kozono dd4b35f864 Fix test for MySQL 2017-12-01 15:26:42 -08:00
Michael Kozono 473ddfb453 Don’t recreate deleted uploads 2017-12-01 15:26:42 -08:00
Michael Kozono 908aacddda Filter existing uploads with one query 2017-12-01 15:26:42 -08:00
Michael Kozono 7549d17f72 Refactor 2017-12-01 15:26:41 -08:00
Michael Kozono 67b58ffdc3 Get rid of tracked field
It makes a debugging slightly easier, but is not necessary, and is a waste of resources.
2017-12-01 15:26:41 -08:00
Michael Kozono edb5cac46c Use bulk inserts 2017-12-01 15:26:41 -08:00
Michael Kozono f5fa398724 Attempt to fix spec in CI 2017-12-01 15:26:41 -08:00
Michael Kozono 87529ce582 Move temp table creation into the prepare job
* Hopefully fixes spec failures in which the table doesn’t exist
* Decouples the background migration from the post-deploy migration, e.g. we could easily run it again even though the table is dropped when finished.
2017-12-01 15:26:41 -08:00
Michael Kozono 7fd2643419 Fix Rubocop offenses 2017-12-01 15:26:41 -08:00
Michael Kozono dd8680a7ae Drop temporary tracking table when finished 2017-12-01 15:26:41 -08:00
Michael Kozono d530085685 Refactor specs 2017-12-01 15:26:41 -08:00
Michael Kozono 3dc0b118ec Store paths relative to CarrierWave.root
So the path on source installs cannot be too long for our column.

And fix the column length test since Route.path is limited to 255 chars, it doesn’t matter how many nested groups there are.
2017-12-01 15:26:41 -08:00
Michael Kozono 1ba4d417fe Clean up after test 2017-12-01 15:26:41 -08:00
Michael Kozono c77a353dca Remove unnecessary clearing
Since duplicate inserts are now ignored.
2017-12-01 15:26:41 -08:00
Michael Kozono 1807bc1646 Reword test 2017-12-01 15:26:41 -08:00
Michael Kozono a210cb6b82 Rename table to untracked_files_for_uploads 2017-12-01 15:26:41 -08:00
Michael Kozono 2ab3031bd3 Refactor, no change in behavior 2017-12-01 15:26:41 -08:00
Michael Kozono ffbaf19fe8 Fix Rubocop offenses 2017-12-01 15:26:41 -08:00
Michael Kozono 13e0ee3735 Test batch processing 2017-12-01 15:26:40 -08:00
Michael Kozono 1bae010b63 Calculate checksums
by copy-pasting in the whole `Upload` class.

Also, fix `Namespace` `model_type` (it should not be `Group`).
2017-12-01 15:26:40 -08:00
Michael Kozono 3a0ad99d59 Add untracked files to uploads 2017-12-01 15:26:40 -08:00
Michael Kozono 8315c66a56 Kick off follow up background migration jobs
To process the unhashed_upload_files table.
2017-12-01 15:26:40 -08:00
Michael Kozono b6ea41d130 Find and store unhashed upload file paths 2017-12-01 15:26:40 -08:00
Yorick Peterse 124fa93a8a Merge branch 'remove-mr-diff-serialised-columns' into 'master'
Remove serialised diff and commit columns

Closes #39533

See merge request gitlab-org/gitlab-ce!15582
2017-11-29 13:33:06 +00:00
Bob Van Landuyt 4c4109e1b9 Create fork networks for forks for which the source was deleted.
That way we can join forks-of-forks into the same network even if
their original source was deleted.

Consider the following:

  `awesome-oss/badger` is forked to `coolstuff/badger`, which is
  forked to `user-a/badger` which in turn is forked to
  `user-b/badger`.

  When `awesome-oss/badger` is deleted, we will now create a fork
  network with `coolstuff/badger` as the root. The `user-a/badger`
  and `user-b/badger` projects will be added to the network.
2017-11-29 12:11:48 +01:00
Sean McGivern 4ebbfe5d3e Remove serialised diff and commit columns
The st_commits and st_diffs columns on merge_request_diffs historically held the
YAML-serialised data for a merge request diff, in a variety of formats.

Since 9.5, these have been migrated in the background to two new tables:
merge_request_diff_commits and merge_request_diff_files. That has the advantage
that we can actually query the data (for instance, to find out how many commits
we've stored), and that it can't be in a variety of formats, but must match the
new schema.

This is the final step of that journey, where we drop those columns and remove
all references to them. This is a breaking change to the importer, because we
can no longer import diffs created in the old format, and we cannot guarantee
the export will be in the new format unless it was generated after this commit.
2017-11-28 16:13:40 +00:00
Stan Hu 5cecff893d Convert migration to populate latest merge request ID into a background migration
This is to smear updates over a few hours to avoid causing excessive
replication lag as seen in https://gitlab.com/gitlab-com/infrastructure/issues/3235.
2017-11-17 10:23:48 +00:00
Bob Van Landuyt f961744bde Isolate the fork network background migrations
Before the `PopulateForkNetworksRange` spec would also call the
`CreateForkNetworkMemberships` which we would count on in the spec.

With this, I'm isolating that, and counting only records created in
this particular migration instead.
2017-11-15 10:13:57 +01:00
Bob Van Landuyt aaf18bb8c8 Don't try to create fork network memberships for forks of forks
In case the root project of a Fork-of-fork is deleted, the ForkNetwork
and the membership for that fork network is never created. In this
case we shouldn't try to create the membership, since the parent
membership will never be created.

This means that these fork networks will be lost.
2017-11-14 11:19:19 +01:00
Sean McGivern 9245bfc293 Handle null serialised commits in background migration
This is already handled for diffs, but not commits.
2017-10-17 12:15:40 +01:00