Resolve "Gitaly::CommitService: Encoding::UndefinedConversionError: U+C124 from UTF-8 to ASCII-8BIT"
Closes#42161
See merge request gitlab-org/gitlab-ce!16637
Search query is especially slow if a user searches a generic string
which matches many records, in such case search can take tens of
seconds or time out. To speed up the search query, we search only for
first 1000 records, if there is >1000 matching records we just display
"1000+" instead of precise total count supposing that with such amount
the exact count is not so important for the user.
Because for issues even limited search was not fast enough, 2-phase
approach is used for issues: first we use simpler/faster query to get
all public issues, if this exceeds the limit, we just return the limit.
If the amount of matching results is lower than limit, we re-run more
complex search query (which includes also confidential issues).
Re-running the complex query should be fast enough in such case because the
amount of matching issues is lower than limit.
Because exact total_count is now limited, this patch also switches to
to "prev/next" pagination.
Related #40540
Gitlab::Profiler.profile takes a URL to profile, and returns the ruby-prof
profile result.
This is currently only for interactive use (in a console) or from the script
runner. Note that when using this interactively, changes to the application
within that console session will be used, so this can be used to test proposed
performance changes.
Prior to this commit we would essentially update all rows in a table,
even those where the source column (e.g. `issues.closed_at`) was NULL.
This in turn could lead to statement timeouts when using the default
batch size of 10 000 rows per job.
To work around this we don't schedule jobs for rows where the source
value is NULL. We also don't update rows where the source column is NULL
(as an extra precaution) or the target column already has a non-NULL
value. Using this approach it should be possible to update 10 000 rows
in the "issues" table in about 7.5 - 8 seconds.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/42158
Don't allow line breaks on HTTP headers
See merge request gitlab/gitlabhq!2277
(cherry picked from commit 7fc0a6fc096768a5604d6dd24d7d952e53300c82)
073b8f9c Don't allow line breaks on HTTP headers
[10.3] Migrate `can_push` column from `keys` to `deploy_keys_project`
See merge request gitlab/gitlabhq!2276
(cherry picked from commit f6ca52d31bac350a23938e0aebf717c767b4710c)
1f2bd3c0 Backport to 10.3
A file containing /:\d+:/ in its contents would break the search results if
those contents were part of the results, because we were splitting on colons,
which can't work with untrusted input.
Changing to use the null byte as a separator is much safer.
If the schema changes after 20171114162227 for any of these models, and specs
after this one use factories, then those factories will use the models with
outdated column information cached.
We shouldn't really use factories in migration specs, but this is a special case
because there is a lot of git-related setup code in the model that would be
painful to copy to the migration. Instead, we just manually reset the column
information for the models we could pollute.
* upstream/master: (621 commits)
Add a note about GitLab QA page objects validator to docs
Refactor dispatcher projects blame and blob path
Update export message to mention we can download the file from the UI
Fix Ctrl+Enter keyboard shortcut saving comment/note edit
fix case where tooltip messes up :last-child selector
Add reason to keep postgresql 9.2 for CI
Remove warning noise in ProjectImportOptions
Add changelog entry
Add RedirectRoute factory
Update Ingress extra cost note to be more generic
Fix Rubocop offense
Refactor dispatcher project branches path
Revert "Revert "Fix Route validation for unchanged path""
Document that we need rsync for backing up
Docs: move article "Laravel and Envoy w/ CI/CD"
Recommend against the use of EFS
Adds Rubocop rule for line break around conditionals
Update CHANGELOG.md for 10.1.6
Filter out build traces from logged parameters
Refactored project:n* imports in dispatcher.js
...
For each MR diff an extra 'SELECT COUNT()' is executed
to get number of commits for the diff. Overall time to get counts for
all MR diffs may be quite expensive. To speed up loading of MR info,
information about number of commits is stored in a MR diff's extra column.
Closes#38068
Migration is done through a small refactoring, which makes us call
endpoins which are performing the same actions for namespaces.
Tests are added to ensure only the project is removed that should be
removed.
Closesgitlab-org/gitaly#873
Originally from branch 'fix-authorized-keys-enabled-default-2738' via merge request https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2240
Removed background migrations which were intended to fix state after using Gitlab
without a default having been set
Squashed commits:
Locally, if Spring was not restarted, `current_application_settings` was still cached, which prevented the migration from editing the file. This will also ensure that any app server somehow hitting old cache data will properly default this setting regardless.
Retroactively fix migration
This allows us to identify customers who ran the broken migration. Their `authorized_keys_enabled` column does not have a default at this point.
We will fix the column after we fix the `authorized_keys` file.
Fix authorized_keys file if needed
Add default to authorized_keys_enabled setting
Reminder: The original migration was fixed retroactively a few commits ago, so people who did not ever run GitLab 9.3.0 already have a column that defaults to true and disallows nulls. I have tested on PostgreSQL and MySQL that it is safe to run this migration regardless.
Affected customers who did run 9.3.0 are the ones who need this migration to fix the authorized_keys_enabled column.
The reason for the retroactive fix plus this migration is that it allows us to run a migration in between to fix the authorized_keys file only for those who ran 9.3.0.
Tweaks to address feedback
Extract work into background migration
Move batch-add-logic to background migration
Do the work synchronously to avoid multiple workers attempting to add batches of keys at the same time.
Also, make the delete portion wait until after adding is done.
Do read and delete work in background migration
Fix Rubocop offenses
Add changelog entry
Inform the user of actions taken or not taken
Prevent unnecessary `select`s and `remove_key`s
Add logs for action taken
Fix optimization
Reuse `Gitlab::ShellAdapter`
Guarantee the earliest key
Fix migration spec for MySQL
Originally branch 'mk-toggle-writing-to-auth-keys-1631'
See merge request !2004
Squashed commits:
Add authorized_keys_enabled to Application Settings
Ensure default settings are exposed in UI
Without this change, `authorized_keys_enabled` is unchecked when it is nil, even if it should be checked by default.
Add “Speed up SSH operations” documentation
Clarify the reasons for disabling writes
Add "How to go back" section
Tweak copy
Update Application Setting screenshot
Add find key by base64 key or fingerprint to the internal API
See merge request !250
Squashed changes:
Add unique index to fingerprint
Add new index to schema
Add internal api to get ssh key by fingerprint
Change API endpoint to authorized_keys
Add InsecureKeyFingerprint that calculates the fingerprint without shelling out
Add require for gitlab key fingerprint
Remove uniqueness of fingerprint index
Remove unique option from migration
Fix spec style in fingerprint test
Fix rubocop complain
Extract insecure key fingerprint to separate file
Change migration to support building index concurrently
Remove those hideous tabs
This removes all usage of soft removals except for the "pending delete"
system implemented for projects. This in turn simplifies all the query
plans of the models that used soft removals. Since we don't really use
soft removals for anything useful there's no point in keeping it around.
This _does_ mean that hard removals of issues (which only admins can do
if I'm not mistaken) can influence the "iid" values, but that code is
broken to begin with. More on this (and how to fix it) can be found in
https://gitlab.com/gitlab-org/gitlab-ce/issues/31114.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/37447
Due to the refactoring in !16082, `Blob#batch` no longer falls back
to a default `blob_size_limit`. Since `Repository#batch_blobs` was using
a default `nil` value, this would cause issues in the `Blob#find_by_rugged`
method.
This fix here is to be consistent and use a non-nil default value in
`Repository#batch_blobs`.
The problem was masked in development and tests because Gitaly is always
enabled by default for all features.
Closes#41735
This adds a minimum interval to BackgroundMigrationWorker, ensuring
background migrations of the same class only run once every 5 minutes.
This prevents a thundering herd problem where scheduled migrations all
run at once due to their delays having been expired (e.g. as the result
of a queue being paused for a long time).
If a job was recently executed it's rescheduled with a delay that equals
the remaining time of the job's lease. This means that if the lease
expires in two minutes we only need to wait two minutes, instead of
five.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/41624
* master: (24 commits)
Allow local tests to use a modified Gitaly
Update check.md
add deprecation and removal issue to docs
Add status attribute to runner api entity
Fix typos in a code comment
Just try to detect and assign once
Modify `LDAP::Person` to return username value based on attributes
Create Kubernetes based on Application Templates
Fix a bug where charlock_holmes was used needlessly to encode strings
Ignore the Migration/Datetime cop in a migration that fix a column type to datetime_with_timezone
Use `__` instead of `s__` when context is not required
Enclose props in quotes
Update some Gitaly annotations in Gitlab::Shell
Prepare Gitlab::Git::Repository#rebase for Gitaly migration
add missing changelog
refactor spec, add docs
deprecate check integrity task
add lock specs
add locks chek
remove max-depth flag so it works with subgroups
...
`Gitlab::LDAP::Person` did not respect the LDAP attributes username
configuration and would simply return the uid value. There are
cases where users would like to specify a different username field
to allow more friendly GitLab usernames. For example, it's common
in AD to have sAMAccountName be an employee ID like `A12345` while
the local part of the email address is more human-friendly.
Ahmad Sherif
Robert Speicher
Rémy Coutable
Grzegorz Bizon
Jacob Vosmaer
Douwe Maan
Stan Hu
Yorick Peterse
Nick Thomas
James Edwards-Jones
Jan Provaznik
Lin Jen-Shin
Alejandro Rodríguez
Sean McGivern
Sean McGivern
Oswaldo Ferreira
James Lopez
James Edwards-Jones
Grzegorz Bizon
Mayra Cabrera
Francisco Javier López
Zeger-Jan van de Weg
Valery Sizov
Michael Kozono
Pablo Carranza
Shinya Maeda
Kamil Trzciński
Matija Čupić
Jarka Kadlecová
Filipa Lacerda
Drew Blessing