This finishes the procedure for migrating events from the old format
into the new format. Code no longer uses the old setup and the database
tables used during the migration process are swapped, with the old table
being dropped.
While the database migration can be reversed this will 1) take a lot of
time as data has to be coped around 2) won't restore data in the
"events.data" column as we have no way of restoring this.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/37241
this will remove the need make N queries (per-note) at the
cost of having to mark notes with an attribute
this opens up the possibility for other special roles for notes
The following optimizations were performed:
- Add new association to GroupMember and ProjectMember
This new association will allow us to check if a user is a member of a
Project or Group through a single query instead of two.
- Optimize retrieving of Members when adding multiple Users
Given the user can soon have multiple config sources for CI, we now store
what type at the time of the pipeline run we chose. This will give us
insight into what triggered the new pipeline so we can display it to the
enduser.
Behind an application setting, which defaults to false, this commit
implements the implied CI/CD config. Which means that in the case we
can't find the `.gitlab-ci.yml` on the commit we want to start a
pipeline for, we fall back to an implied configuration.
For now the Bash template has been copied to
`Auto-Devops.gitlab-ci.yml` so the tests actually work.
Fixes#34777
* master: (275 commits)
Decrease Metrics/PerceivedComplexity threshold to 17
Upgrade mail and nokogiri gems due to security issues
Link out to stackoverflow answer on setting swappiness
Document swappiness recomendations in the requirements doc
Fix invalid attribute used for time-ago-tooltip component
Update latest artifacts doc
Add changelog entry for flipping verify_certificates
Default LDAP config verify_certificates to true
Update share project with groups docs
remove accidental console.log from karma tests
update specs to match reorganized monitoring components
Remove tooltips from new sidebar
Use `git update-ref --stdin -z` to delete refs
Don't use public_send in destroy_conditionally! helper
Remove unused expressions policy from ci/cd config
Simplify code for appending strategies in CI/CD config
Raise exception when simplifiable ci entry incomplete
Add changelog entry
Fix MySQL failure for emoji autocomplete
max-width for lazy-loaded images (this was removed in the original MR through merge resolution most probably)
...
Conflicts:
lib/gitlab/ci/config/entry/policy.rb
`allowed_key_types` is removed and the `minimum_<type>_bits` fields are
renamed to `<tech>_key_restriction`. A special sentinel value (`-1`) signifies
that the key type is disabled.
This also feeds through to the UI - checkboxes per key type are out, inline
selection of "forbidden" and "allowed" (i.e., no restrictions) are in.
As with the previous model, unknown key types are disallowed, even if the
underlying ssh daemon happens to support them. The defaults have also been
changed from the lowest known bit size to "no restriction". So if someone
does happen to have a 768-bit RSA key, it will continue to work on upgrade, at
least until the administrator restricts them.
This is an amalgamation of:
* Cory Hinshaw: Initial implementation !5552
* Rémy Coutable: Updates !9350
* Nick Thomas: Resolve conflicts and add ED25519 support !13712
This ensures the issues/MR cache of the sidebar is only updated when the
state or confidential flags changes, instead of changing this for every
update.
* upstream/master: (225 commits)
Add changelog entry
Backports EE 2756 logic to CE.
Make rubocop happy
Make profile settings dropdown consistent
Add filter by my reaction
Update spec initialization with it being a shared component
Update identicon path and selector
Renamed to `identicon` and make shared component
Merge branch 'master-i18n' into 'master'
Fix broken Frontend JS guide
Replace 'project/star.feature' spinach test with an rspec analog
Adds position fixed to right sidebar
Fixes the margin of the top buttons of the pipeline page
Remove commented out code
Better align fallback image emojis
Decrease Metrics/CyclomaticComplexity threshold to 15
Add changelog
Respect the default visibility level when creating a group
Further break with_repo_branch_commit into parts
Make sure inspect doesn't generate crazy string
...
There are some redundancies in the validation steps, and that is to
preserve current error messages behavior
Also few specs have to be changed in order to fix madness in validation
logic.
Every project page displays a navigation menu that in turn displays the
number of open issues and merge requests. This means that for every
project page we run two COUNT(*) queries, each taking up roughly 30
milliseconds on GitLab.com. By caching these numbers and refreshing them
whenever necessary we can reduce loading times of all these pages by up
to roughly 60 milliseconds.
The number of open issues does not include confidential issues. This is
a trade-off to keep the code simple and to ensure refreshing the data
only needs 2 COUNT(*) queries instead of 3. A downside is that if a
project only has 5 confidential issues the counter will be set to 0.
Because we now have 3 similar counting service classes the code
previously used in Projects::ForksCountService has mostly been moved to
Projects::CountService, which in turn is reused by the various service
classes.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/36622
* master: (115 commits)
Use event-based waiting in Gitlab::JobWaiter
Make sure repository's removal work for legacy and hashed storages
Use `@hashed` prefix for hashed paths on disk, to avoid collision with existing ones
Refactor project and storage types
Prevent using gitlab import task when hashed storage is enabled
Some codestyle changes and fixes for GitLab pages
Removed some useless code, codestyle changes and removed an index
Fix repository reloading in some specs
Changelog
Moving away from the "extend" based factory to a more traditional one.
Enable automatic hashed storage for new projects by application settings
New storage is now "Hashed" instead of "UUID"
Add UUID Storage to Project
Move create_repository back to project model as we can use disk_path and share it
Codestyle: move hooks to the same place and move dependent methods to private
Use non-i18n values for setting new group-level issue/MR button text
indexes external issue tracker
copyedit
indexes user/search/ from /user/index
Correctly encode string params for Gitaly's TreeEntries RPC
...
This changes the style of push events that remove tags or branches so
they don't display the commit details. This prevents displaying commit
details such as:
000000 . --broken encoding
Instead we now simply display the header such as:
Administrator deleted branch example-branch
This is displayed in the same style as events for newly created
branches/tags.
This commit also ensures that if no commit message is present we simply
don't display anything, instead of "--broken encoding".
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/36685
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/36722
This changes the caching mechanism so we cache both current _and_ future
broadcast messages, then manually filter out those we don't want to
display. This ensures we don't need any additional queries while still
being able to display the right messages at the right time.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/36661
* master: (1000 commits)
Fix username autocomplete group name with no avatar alignment
Fix 'Projected tags' typo in protected_tags_spec.rb
Many Repo Fixes
Repo Editor Fixes
Docs: New index for permissions
link article from CI index
link tech articles from the landing page
new articles come first
fix relative link
fix date format
Fixed changed files dropdown not being shown
Update publication date
Remove deprecated field from workhorse API responses
Fix API responses when dealing with txt files
Make sure MySQL would not use CURRENT_TIMESTAMP
Add two more project templates
Allow usage of any_projects? with an Array
Copyedit Artifactory and GitLab article
Rename Artifactory and GitLab article file
Display GPG status loading spinner only when Ajax request is made
...
The number of forks of a project doesn't change very frequently and
running a COUNT(*) every time this information is requested can be quite
expensive. We also end up running such a COUNT(*) query at least twice
on the homepage of a project.
By caching this data and refreshing it when necessary we can reduce
project homepage loading times by around 60 milliseconds (based on the
timings of https://gitlab.com/gitlab-org/gitlab-ce).
* upstream/master: (47 commits)
Update CHANGELOG.md for 9.4.5
Update charlock_holmes
add a changelog entry
switch to multi-line before block
restructure the #new_key notification spec
don't send devise notifications to the ghost user
reset_delivered_emails before testing #new_key
skip the :read_project check for new_project_member
move the member spec to be with the other ones
add a spec for new_group_member
add a spec for never emailing the ghost user
rubocop fix
a membership with no user is always notifiable
check notifiability for more emails
add Member#notifiable?(type, opts)
make NotificationRecipient a little more customizable
Add notes about database performance for MySQL
fix confidential border issue as well as confidential styles leaking on new MR
Migrate force push check to Gitaly
Add option to disable project export on instance
...
* master: (66 commits)
fix confidential border issue as well as confidential styles leaking on new MR
Migrate force push check to Gitaly
Add option to disable project export on instance
Better categorize test coverage results
Add option to disable project export on instance - db changes
Better caching and indexing of broadcast messages
Include the `is_admin` field in the `GET /users/:id` API when current user is an admin
Document rspec-retry and rspec-flaky
Fix cop description
Retrieve and sync flaky specs report from and to S3
Use a new RspecFlakyListener to detect flaky specs
Fix formatting of patch_versions.md [skip ci]
Enable Timecop safe mode
Show error message for API 500 error in tests, and
Fix merge request diff deserialisation when too_large was absent
Delete correct key from `session` after authenticating using U2F
Bumps omniauth-ldap gem version to 2.0.4
Pending delete projects no longer return 500 error in Admins projects view
Do not run the `ee_compat_check` job for stableish branches
Update gitlab.po: Missing 'r' in "Fouché" that comes from "Fourcher" verb.
...
Caching of BroadcastMessage instances has been changed so a cache stays
valid as long as the default cache expiration time permits, instead of
the cache being expired after 1 minute. When modifying broadcast
messages the cache is flushed automatically.
To remove the need for performing sequence scans on the
"broadcast_messages" table we also add an index on (starts_at, ends_at,
id), permitting PostgreSQL to use an index scan to get all necessary
data.
Finally this commit adds a few NOT NULL constraints to the table to
match the Rails validations.
Fixesgitlab-org/gitlab-ce#31706
* upstream/master:
Fix merge request diff deserialisation when too_large was absent
Bumps omniauth-ldap gem version to 2.0.4
Remove an unnecessary `let` in spec/features/projects/user_edits_files_spec.rb
Remove documentation about rules that are now enforced by RuboCop
Enable the RSpec/HookArgument cop and auto-correct offenses
* upstream/master: (62 commits)
Update gitlab.po: Missing 'r' in "Fouché" that comes from "Fourcher" verb.
Docs: update user docs index
Fix minor typos in views
Fix Layout/SpaceBeforeBlockBraces violation in bin/changelog_spec
Merge branch 'rs-alphanumeric-ssh-params' into 'security-9-4'
Merge branch 'import-symlinks-9-3' into 'security-9-3'
Fix wrong method call on prometheus histogram
Document new all-in-one Helm chart - docs
Fix 404 on link path
Fix line numbers not matching up to code in code viewer.
Hide overflow-x on collapsed sidebar
removed global use of breakpoint checker
Increase performance of the breakpoint size checker
Filter sensitive query string parameters from NGINX access logs
Added a template for database changes
Render new issue link in failed job as a regular link instead of a UJS one
Include RE2 in the upgrade docs
Remove affix plugin from issuable sidebar with new navigation
Fix linter error
alternative route for download archive
...
This changes various controllers to use the new EventCollection class
for retrieving events. This class uses a JOIN LATERAL query on
PostgreSQL to retrieve queries in a more efficient way, while falling
back to a simpler / less efficient query for MySQL.
The EventCollection class also includes a limit on the number of events
to display to prevent malicious users from cycling through all events,
as doing so could put a lot of pressure on the database.
JOIN LATERAL is only supported on PostgreSQL starting with version 9.3.0
and as such this optimisation is only used when using PostgreSQL 9.3 or
newer.
This commit migrates events data in such a way that push events are
stored much more efficiently. This is done by creating a shadow table
called "events_for_migration", and a table called "push_event_payloads"
which is used for storing push data of push events. The background
migration in this commit will copy events from the "events" table into
the "events_for_migration" table, push events in will also have a row
created in "push_event_payloads".
This approach allows us to reclaim space in the next release by simply
swapping the "events" and "events_for_migration" tables, then dropping
the old events (now "events_for_migration") table.
The new table structure is also optimised for storage space, and does
not include the unused "title" column nor the "data" column (since this
data is moved to "push_event_payloads").
== Newly Created Events
Newly created events are inserted into both "events" and
"events_for_migration", both using the exact same primary key value. The
table "push_event_payloads" in turn has a foreign key to the _shadow_
table. This removes the need for recreating and validating the foreign
key after swapping the tables. Since the shadow table also has a foreign
key to "projects.id" we also don't have to worry about orphaned rows.
This approach however does require some additional storage as we're
duplicating a portion of the events data for at least 1 release. The
exact amount is hard to estimate, but for GitLab.com this is expected to
be between 10 and 20 GB at most. The background migration in this commit
deliberately does _not_ update the "events" table as doing so would put
a lot of pressure on PostgreSQL's auto vacuuming system.
== Supporting Both Old And New Events
Application code has also been adjusted to support push events using
both the old and new data formats. This is done by creating a PushEvent
class which extends the regular Event class. Using Rails' Single Table
Inheritance system we can ensure the right class is used for the right
data, which in this case is based on the value of `events.action`. To
support displaying old and new data at the same time the PushEvent class
re-defines a few methods of the Event class, falling back to their
original implementations for push events in the old format.
Once all existing events have been migrated the various push event
related methods can be removed from the Event model, and the calls to
`super` can be removed from the methods in the PushEvent model.
The UI and event atom feed have also been slightly changed to better
handle this new setup, fortunately only a few changes were necessary to
make this work.
== API Changes
The API only displays push data of events in the new format. Supporting
both formats in the API is a bit more difficult compared to the UI.
Since the old push data was not really well documented (apart from one
example that used an incorrect "action" nmae) I decided that supporting
both was not worth the effort, especially since events will be migrated
in a few days _and_ new events are created in the correct format.
This caches the result of Appearance.first in a similar fashion to how
ApplicationSetting instances are cached. We also add some NOT NULL
constraints to the table and correct the timestamp types.
Fixesgitlab-org/gitlab-ce#36066, fixesgitlab-org/gitlab-ce#31698
* master: (42 commits)
alternative route for download archive
Add missing command to compile GetText files
Prevent user from changing username with container registry tags
Rename RPC 'Exists' to 'RepositoryExists'
Fix the sticky changes bar on commits page
Fix ee_compat_check when EE branch uses a prefix
Removed display styles when hiding the fly out navigation
Explain why we use select all for project_url_constrainer.rb
Fix pikaday being undefined
Add a helper to stub storage settings with defaults
Enable the Layout/SpaceBeforeBlockBraces cop
update Install from Source instructions
Translations can be picked without asking for exceptions
Ask for exceptions in advance
Don't require stackprof in Gemfile
Synchronous Korean translation in zanata
Use full path of user's avatar in webhooks
Update icon color on hover
Align all nav items in sidebar
Fix height of collapsed sidebar items
...
* master: (481 commits)
Make sure that we have author and committer
disable file upload button while uploading
Fix bar chart does not display label at hour 0
Fixed activity not loading on project homepage
Expose noteable_iid in Note
Fix fly-out width when it has long items
Add a test to show that threshold 40 would corrupt
Add changelog entry
Raise encoding confidence threshold to 50
Fix the /projects/:id/repository/commits endpoint to handle dots in the ref name when the project full path contains a `/`
Fix the /projects/:id/repository/tags endpoint to handle dots in the tag name when the project full path contains a `/`
Add Italian translations of Pipeline Schedules
Restrict InlineJavaScript for haml_lint to dev and test environment
Incorporate Gitaly's CommitService.FindCommit RPC
Move `deltas` and `diff_from_parents` logic to Gitlab::Git::Commit
fix repo_edit_button_spec.js
fix test failures in repo_preview_spec.js
fix repo_loading_file_spec tests
Refactor Gitlab::Git::Commit to include a repository
use 100vh instead of flip flopping between the two - works on all suported browsers
...
The method User#projects_limit_left would run "personal_projects.count"
but such a query is not memoized. As a result multiple calls to
User#projects_limit_left would result in multiple COUNT(*) queries being
executed.
To work around this this commit adds User#personal_projects_count which
simply memoizes the result of the COUNT(*) in an instance variable.
* master: (162 commits)
Since mysql is not a priority anymore, test it less
Add container registry and spam logs icons
Fix different Markdown styles
Backport to CE for:
Make new dropdown dividers full width
Bump GITLAB_SHELL_VERSION and GITALY_VERSION to support unhiding refs
Install yarn via apt in update guides
Use long curl options
Remove monkey-patched Array.prototype.first() and last() methods
Openshift Getting Started
35659 Rename Pipelines tab to CI / CD in new navigation
Don't bother going through an entire Banzai pipeline for empty text
Add active state for pipelines settings on old nav
Bump rspec to 3.6.0
Resolve "Specific Async Script Loading by using a Page Variable"
Revert "Merge branch 'rs-warm-capybara-only-in-ci' into 'master'"
another rubocop style fix
Use mixin for new dropdown style
Migrate Repository#last_commit_for_path to Gitaly
Migrate blame loading to Gitaly
...
`:mailer` is needed to pick it easily, while
`type: :mailer` is needed for picking it automatically for
tests located in spec/mailers/*_spec.rb
It's a bit complicated in spec/services/notification_service_spec.rb
but we'll leave it alone for now.
* master: (76 commits)
Add code review guidelines related to Build [CI skip].
Make time span dropdown style on cycle analytics page consistent
Add 204. Remove duplicated method.
Make sure we didn't commit conflicts
Fix bug in blob test
Always fetch branches before finding the merge base, otherwise we could find an outdated merge base
Fixes dropdown margin in sidebar
Docs add blog articles
Inline script cleanup globals and easy
Add option to use CommitLanguages RPC
CI fixes for gitaly-ruby
fix
Allow logged in users to read user list under public restriction
Small refactor in LegacyNamespace and moved back send_update_instructions
Rename ensure_dir_exist -> ensure_storage_path_exist
Added some extra TODOs for the Legacy Storage refactor
Make disk_path keyword argument and optional
Rename more path_with_namespace -> full_path or disk_path
Rename path_with_namespace -> disk_path when dealing with the filesystem
Rename many path_with_namespace -> full_path
...
* commit '9a3b283402b8cc1c86802c526f19a459ce09c2e3': (270 commits)
Add a note about EFS and GitLab log files
Projects logo are not centered vertically on projects page
Fix spec/features/projects/branches_spec
Fixup POST /v3/:id/hooks and PUT /v3/:id/hooks/:hook_id
Fix a spec that was assuming to be on the wrong page
Add copy about search terms to ux guide
Update documentation of user creation by replacing the 'confirm' param with 'skip_confirmation'
Fix replying to commit comments on MRs from forks
Fix 500 error when rendering avatar for deleted project creator
Load and process at most 100 commits when pushing into default branch
Ensure Gitlab::Application.routes.default_url_options are set correctly in Capybara + :js specs
Add log messages to clarify log messages about API CSRF token verification failure
Update gitlab_flow.md, Teatro seems to be completely dead, see also https://forum.gitlab.com/t/gitlab-flow-documentation-teatro/7774
Fix diff commenting results just after changing view
Update CHANGELOG.md for 9.4.2
none is not a CSS Value for sizes ;-)
Merge issuable "reopened" state into "opened"
Make access level more compatible with EE
Add link to JIRA article in docs
Expand pipeline_trigger_service_spec by godfat request
...
when omitting the end part of the key ('-----END PGP PUBLIC KEY
BLOCK-----') the error message was not about the key anymore, but about
the missing fingerprint and primary_keyid, which was confusing for the
user.
the new validation checks that the end also matches the expected format.
Sean McGivern
Annabel Dunstone Gray
Kamil Trzcinski
Tiago Botelho
Michael Kozono
Rubén Dávila
Yorick Peterse
Alexander Keramidas
Zeger-Jan van de Weg
micael.bergeron
Sean McGivern
Alexis Reigel
Shinya Maeda
Grzegorz Bizon
Jose Ivan Vargas
Jacob Schatz
Robert Speicher
Jacob Vosmaer
Marin Jankovski
Felipe Artur
Tim Zallmann
Douwe Maan
Grzegorz Bizon
Douwe Maan
Nick Thomas
Jacopo
Hiroyuki Sato
Lin Jen-Shin
Mark Fletcher
Maxim Rydkin
Oswaldo Ferreira
Mike Greiling
Gabriel Mazetto
Andrew Newdigate
Saverio Miroddi
Robin Bobbitt
Rémy Coutable
vanadium23
Robert Speicher
haseeb
Filipa Lacerda
Dmitriy Zaporozhets
Alejandro Rodríguez
Stan Hu
blackst0ne
Jarka Kadlecova
Ahmad Sherif
Bob Van Landuyt
Bob Van Landuyt
haseeb
Mehdi Lahmam
wendy0402