Commit Graph

89 Commits

Author SHA1 Message Date
Douwe Maan 36ad91d75a Merge branch 'mk-delete-conflicting-redirects-mysql' into 'master'
Clean up redirect routes that conflict with regular routes

Closes #36229

See merge request gitlab-org/gitlab-ce!13783
2017-09-18 08:24:52 +00:00
Yorick Peterse ac702af822
Fix setting share_with_group_lock
Prior to this commit running
Namespace#force_share_with_group_lock_on_descendants would result in
updating _all_ namespaces in the namespaces table, not just the
descendants. This is the result of ActiveRecord::Relation#update_all not
taking into account the CTE. To work around this we use the CTE query as
a sub-query instead of directly calling #update_all.

To prevent this from happening the relations returned by
Gitlab::GroupHierarchy are now marked as read-only, resulting in an
error being raised when methods such as #update_all are used.

Fortunately on GitLab.com our statement timeouts appear to have
prevented this query from actually doing any damage other than causing
a very large amount of dead tuples.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/37916
2017-09-15 17:38:46 +02:00
Michael Kozono c923208721 Spread out the work a little 2017-09-14 14:17:23 -07:00
Michael Kozono ee4f73916f Extract helper for queuing background jobs 2017-09-14 14:17:23 -07:00
Yorick Peterse 5eab624d3c
Improve migrations using triggers
This adds a bunch of checks to migrations that may create or drop
triggers. Dropping triggers/functions is done using "IF EXISTS" so we
don't throw an error if the object in question has already been dropped.
We now also raise a custom error (message) when the user does not have
TRIGGER privileges. This should prevent the schema from entering an
inconsistent state while also providing the user with enough information
on how to solve the problem.

The recommendation of using SUPERUSER permissions is a bit extreme but
we require this anyway (Omnibus also configures users with this
permission).

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/36633
2017-08-29 13:02:44 +02:00
Grzegorz Bizon faf92651aa Fix invalid default argument in migration helpers 2017-08-23 08:58:55 +02:00
Grzegorz Bizon 193b199672 Add Sidekiq migration helpers for migrating queues 2017-08-22 12:47:20 +02:00
Stan Hu 718ecd4eb5 Fix Error 500s when attempting to destroy a protected tag
Due to a missing `on_delete: :cascade`, users would hit the error that
looked like:

```
PG::ForeignKeyViolation: ERROR: update or delete on table "protected_tags"
violates foreign key constraint "fk_rails_f7dfda8c51" on table
"protected_tag_create_access_levels" DETAIL: Key (id)=(1385) is still
referenced from table "protected_tag_create_access_levels". : DELETE FROM
"protected_tags" WHERE "protected_tags"."id" = 1385
```

Closes #36013
2017-08-20 03:32:31 -07:00
Grzegorz Bizon 5cd6f8c76e Add a foreign key to `merge_requests.head_pipeline_id` 2017-07-18 10:51:19 +02:00
Paul Charlton cb3b4a15e6 Support multiple Redis instances based on queue type 2017-07-11 03:35:47 +00:00
Yorick Peterse af1f6844c9
Added code for defining SHA attributes
These attributes are stored in binary in the database, but exposed as
strings. This allows one to query/create data using plain SHA1 hashes as
Strings, while storing them more efficiently as binary.
2017-06-29 15:37:37 +02:00
Bob Van Landuyt 7c53fcf11f Adjust for new static-analysis failures 2017-06-27 13:05:02 +02:00
Bob Van Landuyt 397d3fd7d0 Only do one query for updating routes 2017-06-27 12:26:52 +02:00
Bob Van Landuyt 60561aca22 Update routes directly by ID instead of filtering by path 2017-06-26 17:49:30 +02:00
Bob Van Landuyt 66ba0b0a34 Clear the cache for projects one-by-one 2017-06-26 17:49:30 +02:00
Bob Van Landuyt 8c850380a9 Add punctuation to log messages 2017-06-26 17:49:30 +02:00
Bob Van Landuyt 171f2d97dc Keep failed renames in redis 2017-06-26 17:49:30 +02:00
Bob Van Landuyt d6a0c288c8 Use the migration name as a key in redis 2017-06-26 17:49:30 +02:00
Bob Van Landuyt 229ac39a4c Don't break rolling back when a namespace or project was renamed 2017-06-26 17:49:30 +02:00
Bob Van Landuyt 1ebb225536 More logging so we know we have the rename in redis 2017-06-26 17:49:30 +02:00
Bob Van Landuyt c98ed42d01 Revert renames from a migration 2017-06-26 17:49:30 +02:00
Bob Van Landuyt 152cba56e4 Revert namespace renames 2017-06-26 17:49:30 +02:00
Bob Van Landuyt 0faff42d7c Add methods to revert project renames 2017-06-26 17:49:30 +02:00
Bob Van Landuyt 3e84b6336f Track all renames in redis 2017-06-26 17:48:41 +02:00
Grzegorz Bizon 5331a49d0b Merge branch 'master' into fix/gb/improve-updating-column-in-batches-helper
* master: (56 commits)
  File view buttons
  Don't reset the session when the example failed, because we need capybara-screenshot to have access to it
  Resolve "MR comment + system note highlight don't have the same width"
  Add feature spec for dashboard state filter tabs
  Wording of Mysql support.
  a new feature checklist and more elaborate documentation requirements
  Filter archived project in API v3 only if param present
  Revert to using links instead of buttons in Issuable Index tabs.
  Do not run the codeclimate job on docs-only changes
  Only show gray footer space if environment actions exist
  Migrate Gitlab::Git::Blob.find to Gitaly
  Backport filtered search lazy token consistent state fix
  Add a comment explaining how the branch clean up happens
  Fix Github::Representation::PullRequest#source_branch_exists?
  Add CHANGELOG
  Fix GitHub importer performance on branch existence check
  Rebuild the dynamic path before validating it
  Rename stage ref migration specs to match a class name
  Enable Style/DotPosition Rubocop 👮
  Revert "Merge branch 'winh-merge-request-related-issues' into 'master'"
  ...

Conflicts:
	db/post_migrate/20170526185921_migrate_build_stage_reference.rb
2017-06-22 12:37:38 +02:00
Grzegorz Bizon 0430b76441 Enable Style/DotPosition Rubocop 👮 2017-06-21 13:48:12 +00:00
Grzegorz Bizon e4d42a62d9 Raise if updating columns in batches within a transaction 2017-06-21 14:31:49 +02:00
Yorick Peterse 9a3bea6707
Put an upper limit on update batches
When using update_column_in_batches the upper limit on the batch size is
now 1000. This ensures that for very large tables we don't lock tens of
thousands of rows during the update. This in turn should reduce the
likelyhood of running into deadlocks.
2017-06-19 14:18:41 +02:00
blackst0ne bc00806a4e Add database helpers 'add_timestamps_with_timezone' and 'timestamps_with_timezone' 2017-06-13 22:44:13 +11:00
Nick Thomas eb21f93267 Only use DROP INDEX CONCURRENTLY on postgreql 9.2+ 2017-05-25 18:12:42 +01:00
Bob Van Landuyt f62f0c258d Fixes for the rename reserved paths helpers 2017-05-18 20:08:32 +00:00
Douwe Maan beec97f6cf Disallow NULL on renamed column after default has been set 2017-05-15 16:01:09 -05:00
Rémy Coutable ae8511df13 Merge branch 'rename-column-concurrently-defaults' into 'master'
Fix adding defaults for concurrent column renames

See merge request !11335
2017-05-15 07:50:44 +00:00
James Lopez 48ffec9e74 Update rename_base.rb 2017-05-12 16:00:18 +00:00
Bob Van Landuyt 3760a7c230 Only rename namespaces and projects with an invalid path
For exact matches, not namespaces that end with an invalid path
2017-05-12 17:46:16 +02:00
Yorick Peterse 3fc20c0117
Fix adding defaults for concurrent column renames
By adding the default value _after_ adding the column we avoid updating
all rows in a table, saving a lot of time and unnecessary work in the
process.
2017-05-12 16:27:30 +02:00
Bob Van Landuyt 49fb31db41 Add a new column before creating rename triggers
MySQL doesn't allow us to create a trigger for a column that doesn't
exist yet. Failing with this error:

```
Mysql2::Error: Unknown column 'build_events' in 'NEW': CREATE TRIGGER trigger_6a80c097c862_insert
BEFORE INSERT
ON `services`
FOR EACH ROW
SET NEW.`build_events` = NEW.`job_events`
```

Creating the new column before creating the trigger avoids this.
2017-05-11 16:29:42 +02:00
Bob Van Landuyt 29f2001102 Update comments 2017-05-02 13:49:09 +02:00
Bob Van Landuyt 2e2a63c866 Rename child namespaces in migrationhelpers 2017-05-01 11:14:24 +02:00
Bob Van Landuyt b85f2fa4c7 Clear html cache for a projects milestones 2017-05-01 11:14:24 +02:00
Bob Van Landuyt ce0102d8b7 Remove dependecy on `User` 2017-05-01 11:14:24 +02:00
Bob Van Landuyt 99a03fd6e9 Move ReservedPathsMigration into V1 namespace 2017-05-01 11:14:24 +02:00
Bob Van Landuyt 12735eefcd Minor style adjustments 2017-05-01 11:14:24 +02:00
Bob Van Landuyt 39efd0c030 Clear cached markdown after renaming projects 2017-05-01 11:14:24 +02:00
Bob Van Landuyt 389057f001 Rename Projects & Namespaces based on entire paths 2017-05-01 11:14:24 +02:00
Bob Van Landuyt 27f54bebb2 Use objects for renaming namespaces and projects 2017-05-01 11:14:24 +02:00
Bob Van Landuyt 579d8891d5 Rename projects in a migrationhelper 2017-05-01 11:14:24 +02:00
Bob Van Landuyt 7508ee5667 Make renaming records in the database reusable
So we can use it for projects
2017-05-01 11:14:24 +02:00
Bob Van Landuyt 58bc628d30 Rename namespace-paths in a migration helper 2017-05-01 11:14:24 +02:00
Bob Van Landuyt 2f95e6a0c4 Move `replace_sql` into `Database::MigrationHelpers` 2017-05-01 11:14:24 +02:00