Also, fixing some calls to the :project factory with the :test_repo
trait since this trait is already included in the :project factory.
Signed-off-by: Rémy Coutable <remy@rymai.me>
Previously, we used Psych, which would:
1. Check if a string was encoded as binary, and not ASCII-compatible.
2. Add the !binary tag in that case.
3. Convert to base64.
We need to do the same thing, using a new column in place of the tag.
1. Rescue all errors that `Projects::DestroyService` might throw, to prevent the
worker from leaving things in an inconsistent state
2. Unmark the project as `pending_delete`
3. Add a `delete_error` text column to `projects`, and save the error message in
there, to be shown to the project masters/owners.
This is an ID-less table with just three columns: an association to the merge
request diff the commit belongs to, the relative order of the commit within the
merge request diff, and the commit SHA itself.
Previously we stored much more information about the commits, so that we could
display them even when they were deleted from the repo. Since 8.0, we ensure
that those commits are kept around for as long as the target repo itself is, so
we don't need to duplicate that data in the database.
* upstream/master: (39 commits)
Fix rubocop offenses
Make entrypoint and command keys to be array of strings
Add issuable-list class to shared mr/issue lists to fix new responsive layout
New navigation breadcrumbs
Restore timeago translations in renderTimeago.
Automatically hide sidebar on smaller screens
Fix typo in IssuesFinder comment
Remove placeholder note when award emoji slash command is applied
Make setSidebarHeight more efficient with SidebarHeightManager.
Update CHANGELOG.md for 9.3.3
Resolve "More actions dropdown hidden by end of diff"
Use Gitaly 0.14.0
Improve support for external issue references
Make issuables_count_for_state public
Only verifies top position after the request has finished to account for errors
Clarify counter caching for users without project access
Make finders responsible for counter cache keys
Add changelog entry for issue / MR tab counting optimisations
Don't count any confidential issues for non-project-members
Cache total issue / MR counts for project by user type
...
This adds an ID-less table containing one row per file, per merge request
diff. It has a column for each attribute on Gitlab::Git::Diff that is serialised
currently, with the advantage that we can easily query the attributes of this
new table.
It does not migrate existing data, so we have fallback code when the legacy
st_diffs column is present instead. For a merge request diff to be valid, it
should have at most one of:
* Rows in this new table, with the correct merge_request_diff_id.
* A non-NULL st_diffs column.
It may have neither, if the diff is empty.
* feature/gb/migrate-pipeline-stages: (76 commits)
redesign caching of application settings
Fix binary encoding error on MR diffs
Fix missing tooltip and ARIA labels for accessibility
Add info on using self-signed certs with Registry
Actually clean gitlab-test path when TestEnv.set_repo_refs fails
Introduce optimistic locking support via optional parameter last_commit_id on File Update API
Move issuable bulk edit form into a new sidebar.
Add PowerShell to CI variable docs
Responsive environment tables
Accept a username for User-level Events API
Introduce an Events API
Update GitLab Pages to v0.4.3
Allow numeric pages domain
Remove references to old settings location
Resolve "API: Environment info missed"
Fix Projects API spec
Resolve "When changing project visibility setting, change other dropdowns automatically"
Update explanation of job-level variable override to fit example
change headings to improve SEO
backports changed import logic from pull mirroring feature into CE
...
* commit '6ac1caa01a4c059f5bcb7c9da2e83001e5469f73': (76 commits)
redesign caching of application settings
Fix binary encoding error on MR diffs
Fix missing tooltip and ARIA labels for accessibility
Add info on using self-signed certs with Registry
Actually clean gitlab-test path when TestEnv.set_repo_refs fails
Introduce optimistic locking support via optional parameter last_commit_id on File Update API
Move issuable bulk edit form into a new sidebar.
Add PowerShell to CI variable docs
Responsive environment tables
Accept a username for User-level Events API
Introduce an Events API
Update GitLab Pages to v0.4.3
Allow numeric pages domain
Remove references to old settings location
Resolve "API: Environment info missed"
Fix Projects API spec
Resolve "When changing project visibility setting, change other dropdowns automatically"
Update explanation of job-level variable override to fit example
change headings to improve SEO
backports changed import logic from pull mirroring feature into CE
...
* master: (524 commits)
Improve user experience around slash commands in instant comments
Fix LFS timeouts when trying to save large files
retryable? is now available for CommitStatus
Resolve "Documentation of `.gitlab-ci.yml` states incorrect example for variables"
Fix test failures
Add slugify project path to CI enviroment variables
Fixed typo: craeted -> created
32118 Make New environment empty state btn lowercase
Expose import_status in Projects API
32832 Remove overflow from comment form for confidential issues and vertically aligns confidential issue icon
Fix test failures
Allow manual bypass of auto_sign_in_with_provider
Fix keys seed
Allow users to be hard-deleted from the API
fixup some classnames and media queries
Enable the Style/PreferredHashMethods cop
Lint our factories creation in addition to their build
Don’t schedule workers from inside transactions
Allow scheduling from after_commit hooks
Forbid Sidekiq scheduling in transactions
...
Conflicts:
db/schema.rb
spec/spec_helper.rb
* master: (524 commits)
Improve user experience around slash commands in instant comments
Fix LFS timeouts when trying to save large files
retryable? is now available for CommitStatus
Resolve "Documentation of `.gitlab-ci.yml` states incorrect example for variables"
Fix test failures
Add slugify project path to CI enviroment variables
Fixed typo: craeted -> created
32118 Make New environment empty state btn lowercase
Expose import_status in Projects API
32832 Remove overflow from comment form for confidential issues and vertically aligns confidential issue icon
Fix test failures
Allow manual bypass of auto_sign_in_with_provider
Fix keys seed
Allow users to be hard-deleted from the API
fixup some classnames and media queries
Enable the Style/PreferredHashMethods cop
Lint our factories creation in addition to their build
Don’t schedule workers from inside transactions
Allow scheduling from after_commit hooks
Forbid Sidekiq scheduling in transactions
...
Conflicts:
app/serializers/pipeline_entity.rb
db/schema.rb
spec/factories/ci/stages.rb
spec/lib/gitlab/import_export/safe_model_attributes.yml
spec/services/ci/create_pipeline_service_spec.rb
spec/spec_helper.rb
Improvements and refactorings were made while adding role based permissions for protected tags to EE. This doesn’t backport the feature, but should improve code quality and minimize divergence.
* implemented logging of project and system web hooks
* implemented UI for user area (project hooks)
* implemented UI for admin area (system hooks)
* implemented retry of logged webhook
* NOT imeplemented log remover
This commit introduces the usage of Common Table Expressions (CTEs) to
efficiently retrieve nested group hierarchies, without having to rely on
the "routes" table (which is an _incredibly_ inefficient way of getting
the data). This requires a patch to ActiveRecord (found in the added
initializer) to work properly as ActiveRecord doesn't support WITH
statements properly out of the box.
Unfortunately MySQL provides no efficient way of getting nested groups.
For example, the old routes setup could easily take 5-10 seconds
depending on the amount of "routes" in a database. Providing vastly
different logic for both MySQL and PostgreSQL will negatively impact the
development process. Because of this the various nested groups related
methods return empty relations when used in combination with MySQL.
For project authorizations the logic is split up into two classes:
* Gitlab::ProjectAuthorizations::WithNestedGroups
* Gitlab::ProjectAuthorizations::WithoutNestedGroups
Both classes get the fresh project authorizations (= as they should be
in the "project_authorizations" table), including nested groups if
PostgreSQL is used. The logic of these two classes is quite different
apart from their public interface. This complicates development a bit,
but unfortunately there is no way around this.
This commit also introduces Gitlab::GroupHierarchy. This class can be
used to get the ancestors and descendants of a base relation, or both by
using a UNION. This in turn is used by methods such as:
* Namespace#ancestors
* Namespace#descendants
* User#all_expanded_groups
Again this class relies on CTEs and thus only works on PostgreSQL. The
Namespace methods will return an empty relation when MySQL is used,
while User#all_expanded_groups will return only the groups a user is a
direct member of.
Performance wise the impact is quite large. For example, on GitLab.com
Namespace#descendants used to take around 580 ms to retrieve data for a
particular user. Using CTEs we are able to reduce this down to roughly 1
millisecond, returning the exact same data.
== On The Fly Refreshing
Refreshing of authorizations on the fly (= when
users.authorized_projects_populated was not set) is removed with this
commit. This simplifies the code, and ensures any queries used for
authorizations are not mutated because they are executed in a Rails
scope (e.g. Project.visible_to_user).
This commit includes a migration to schedule refreshing authorizations
for all users, ensuring all of them have their authorizations in place.
Said migration schedules users in batches of 5000, with 5 minutes
between every batch to smear the load around a bit.
== Spec Changes
This commit also introduces some changes to various specs. For example,
some specs for ProjectTeam assumed that creating a personal project
would _not_ lead to the owner having access, which is incorrect. Because
we also no longer refresh authorizations on the fly for new users some
code had to be added to the "empty_project" factory. This chunk of code
ensures that the owner's permissions are refreshed after creating the
project, something that is normally done in Projects::CreateService.
Shinya Maeda
Rémy Coutable
Sean McGivern
Sean McGivern
Timothy Andrew
Kamil Trzciński
Felipe Artur
Lin Jen-Shin
Alejandro Rodríguez
Adam Niedzielski
Phil Hughes
Grzegorz Bizon
Tim Zallmann
James Edwards-Jones
Jarka Kadlecova
Douwe Maan
Dmitriy Zaporozhets
Alexander Randa
Douwe Maan
Yorick Peterse
Bob Van Landuyt
Gabriel Mazetto
Regis Boudinot
Felipe Artur