The benefit of running the danger_local Rake task
is that it displays some informational messages
for clarity and also exits with 1 if there's any
output.
As per https://gitlab.com/gitlab-org/gitlab-ce/issues/46043, project
templates should be squashed before updating, so that repositories
created from these templates don't include the full history of the
backing repository.
This rake task had been broken for a while. This fixes the breakages,
adds a test to help avoid future breakages, and adds a few ergonomic
improvements to the task itself.
This backport changes made by EE to the files in the scripts/ directory.
This comes with a few changes to some scripts to make them work in the
single codebase setup.
In some cases ActiveSession.cleanup was not called after authentication,
so for some user ActiveSession lookup keys grew without ever cleaning
up. This Rake task manually iterates over the lookup keys and removes
ones without existing ActiveSession.
Adds an unset_rugged rake task that unsets all rugged feature flags.
Also fixes the existing disable_rugged task to have it explicitly
disable feature flags instead of just unsetting them.
This adds the rake task rake
gitlab:cleanup:orphan_job_artifact_files. This rake task cleans all
orphan job artifact files it can find on disk.
It performs a search on the complete folder of all artifacts on
disk. Then it filters out all the job artifact ID for which it could
not find a record with matching ID in the database. For these, the
file is deleted from disk.
If you chose to use the rollback migration feature on your current
version for example, you should still have a way to migrate, being
that you are still on a supported migration path.
This backports all EE schema changes to CE, including EE migrations,
ensuring both use the same schema.
== Updated tests
A spec related to ghost and support bot users had to be modified to make
it pass. The spec in question assumes that the "support_bot" column
exists when defining the spec. In the single codebase setup this is not
the case, as the column is backported in a later migration. Any attempt
to use a different schema version or use of "around" blocks to
conditionally disable specs won't help, as reverting the backport
migration would also drop the "support_bot" column. Removing the
"support_bot" tests entirely appears to be the only solution.
We also need to update some foreign key tests now that we have
backported the EE columns. Fortunately, these changes are very minor.
== Backporting migrations
This commit moves EE specific migrations (except those for the Geo
tracking database) and related files to CE, and also removes any traces
of the ee/db directory.
Some migrations had to be modified or removed, as they no longer work
with the schema being backported. These migrations were all quite old,
so we opted for removing them where modifying them would take too much
time and effort.
Some old migrations were modified in EE, while also existing in CE. In
these cases we took the EE code, and in one case removed them entirely.
It's not worth spending time trying to merge these changes somehow as we
plan to remove old migrations around the release of 12.0, see
https://gitlab.com/gitlab-org/gitlab-ce/issues/59177 for more details.
It used to be the case that GitLab created symlinks for each repository
to one copy of the Git hooks, so these ran when required. This changed
to set the hooks dynamically on Gitaly when invoking Git.
The side effect is that we didn't need all these symlinks anymore, which
Gitaly doesn't create anymore either. Now that means that the tests in
GitLab-Rails should test for it either.
Related: https://gitlab.com/gitlab-org/gitaly/issues/1392#note_175619926
Files in lib will be eager loaded and hence will require haml_lint to be
loaded. Since this is only a development dependency, we can't assume
this gem will be available in production, so it should never be loaded
in production.
Many customers forget to include the gitlab-secrets.json file. This adds
a warning that both gitlab-secrets.json and gitlab.rb are not included
in the backup.
The `#reload` makes to load all objects into memory,
and the main purpose of `#reload` is to drop the association cache.
The `#reset` seems to solve exactly that case.
This backports EE specific changes for the Rake task `gitlab:env:info`,
wrapping them in a conditional. There is no way to inject code in the
middle of a Rake task in EE, so unfortunately this is the best we can
do.
In this commit, some methods that aren't being used
are removed from `Gitlab::Shell`. They are the ff:
- `#remove_keys_not_found_in_db`
- `#batch_read_key_ids`
- `#list_key_ids`
The corresponding methods in `Gitlab::Keys` have been
removed as well.
- Only storing fixtures in one place
- This place changes whether we are in CE or EE
We discovered with @winh that only fixtures located under
spec/javascripts/fixtures are used, even in EE so there's no need to
clean/create fixtures in ee/spec/javascripts/fixtures.
Signed-off-by: Rémy Coutable <remy@rymai.me>
This is a small polishing on the storage migration and storage rollback
rake tasks. By aborting a migration while a rollback is already
scheduled we want to prevent unexpected consequences.
Created new emojis map in public folder
Renamed folder to emojis
Loading now the emojis from Localstorage or from Server
Moved all emojis to the public folder
Loading the emojis.json file now through AJAX
Loads now the map in the web element instead when building the emoji tag
Updated the custom Element Setup to modern method
Fixed Emoji Karma Specs for async loading
Loading now the emojis from Localstorage or from Server
Loads now the map in the web element instead when building the emoji tag
Fixed problem with FIXTURE_PATH for emojis fixtures
Fixes Linting Error in gemojione.rake
Fixed Emoji Karma Specs
Fix static type check in gemojione and check if already registered
Testing if the Emoji Support Check is failing Rspec
Change of CLass Name, returning true on check to test
Fixes failing Emoji RSpec Tests
Moved Emojis into public/-/emojis/1/
Fixed Linting Errors in gl_emoji
Fix to fixtures creation for emojis
Fixed path spec for new subdirectory -/emojis
Optimized emojis.json output
Fix for Emoji Spec failure due to unicode dataset
Better catch handling for emojis
This brings back some of the changes in
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20339.
For users using Gitaly on top of NFS, accessing the Git data directly
via Rugged is more performant than Gitaly. This merge request introduces
the feature flag `rugged_find_commit` to activate Rugged paths.
There are also Rake tasks `gitlab:features:enable_rugged` and
`gitlab:features:disable_rugged` to enable/disable these feature
flags altogether.
Part of four Rugged changes identified in
https://gitlab.com/gitlab-org/gitlab-ce/issues/57317.
Rollback is done similar to Migration for the Hashed Storage.
It also shares the same ExclusiveLease key to prevent both happening
at the same time.
All Hashed Storage related workers now share the same queue namespace
which allows for assigning dedicated workers easily.
If there are any clients connected to the DB, PostgreSQL won't let you
drop the database. It's possible that Sidekiq, Unicorn, or some other
client will be hanging onto a connection, preventing the DROP DATABASE
from working. To workaround this problem, this method cancels all the
connections so that the db:reset command will work.
Note that there's still a slight possibility a client connects after its
connection is terminated. If this is an issue, we could solve it by
revoking CONNECT access, but for now it seems this works.
Closes https://gitlab.com/gitlab-org/gitlab-development-kit/issues/450
Specs were reviewed and improved to better cover the current behavior.
There was some standardization done as well to facilitate the
implementation of the rollback functionality.
StorageMigratorWorker was extracted to HashedStorage namespace were
RollbackerWorker will live one as well.
To separate the different kinds of repositories we have at GitLab this
table will be renamed to pool_repositories. A project can, for now at
least, be member of none, or one of these. The table will get additional
columns in a later merge request where more logic is implemented for the
model.
Further included is a small refactor of logic around hashing ids for the
disk_path, mainly to ensure a previous implementation is reusable.
The disk_path for the pool_repositories table no longer has a NOT NULL
constraint, but given the hashing of the ID requires the DB to assign
the record an ID, an after_create hook is used to update the value.
A related MR is:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23143, adding
tables for 'normal' repositories and wiki_repositories.
Also, since Gitaly now takes care of checking for storage paths
existence/accessibility, we can remove those check from the
gitlab:gitlab_shell_check task and advance further into 0 direct disk
approach on gitlab-rails
We started syncing all the wiki regardless of the fact it's disabled or
not. We couldn't do that in one stage because of needing of smoth update
and deprecating things. This is the second stage that finally removes
unused columns in the geo_node_status table.
Fast lookup of authorized SSH keys in the database was ported to CE in
v10.4. This change adds the option to enable the setting via the restore
rake task and assumes yes if the force env variable is set.
If doing a schema load, the post_migrations should also be marked as up,
even if SKIP_POST_DEPLOYMENT_MIGRATIONS was set, otherwise future
migration runs will be broken.
Rake tasks cleaning up the Git storage were still using direct disk
access, which won't work if these aren't attached. To mitigate a
migration issue was created.
To port gitlab:cleanup:dirs, and gitlab:cleanup:repos, a new RPC was
required, ListDirectories. This was implemented in Gitaly, through
https://gitlab.com/gitlab-org/gitaly/merge_requests/868.
To be able to use the new RPC the Gitaly server was bumped to v0.120.
This is an RPC that will not use feature gates, as this doesn't scale on
.com so there is no way to test it at scale. Futhermore, we _know_ it
doesn't scale, but this might be a useful task for smaller instances.
Lastly, the tests are slightly updated to also work when the disk isn't
attached. Eventhough this is not planned, it was very little effort and
thus I applied the boy scout rule.
Closes https://gitlab.com/gitlab-org/gitaly/issues/954
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/40529
While reviewing this branch it was noted that this task was broken.
Given the limited value this task brings, and given the fact that much
of the old behaviour has been moved to be automatically fixed, removing
was a good option.
Suggestion in:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20806#note_91261001
These tasks are happening through housekeeping right now, by default
ever 10th push. This removes the need for these tasks.
Side note, this removes one of my first contributions to GitLab, as back
than I introduced these tasks through: 54e6c0045b
Closes https://gitlab.com/gitlab-org/gitaly/issues/768
Making the `gettext:updated_check` by completely regenerating the gitlab.pot.
This avoids an issue where `gettext:find` would not pick up on changes
if the file isn't removed first.
Since we are only using `.po` files for translation we can shave off
some setup time.
Generating the `mo` files also regenerates the PO-files, including
fuzzy translations. Those cause a lot of bugs, and we don't need them
anymore since the `po` files are generated by an external translation service.
Adds a new method 'puts_time' that prepends the time of a
message when printing it. All instances of 'progress.puts'
in the gitlab:backup:restore tasks are replaced with puts_time.
Example output:
2018-06-03 16:33:25 -0400 -- Restoring uploads ..
Closes#46448
GitLab Bot
Peter Leitzen
Dinesh Panda
Ash McKenzie
GitLab Release Tools Bot
shampton
Jan Provaznik
Michael Kozono
Jarka Košanová
Hordur Freyr Yngvason
Lin Jen-Shin
Yorick Peterse
Andreas Brandl
Heinrich Lee Yu
Douwe Maan
Jeremy Jackson
Nick Thomas
Imre Farkas
Mike Greiling
Stan Hu
Brett Walker
Douglas Barbosa Alexandre
John Cai
Felipe Artur
Gabriel Mazetto
Mayra Cabrera
Lukas Eipert
Dmitriy Zaporozhets
Toon Claes
DJ Mountney
Bob Van Landuyt
Zeger-Jan van de Weg
Sarah Groff Hennigh-Palermo
Rémy Coutable
Kamil Trzciński
Ben Bodenmiller
Cindy Pallares
Tim Zallmann
Robert Speicher
John Jarvis
Wei-Meng Lee
Patrick Bajao
Ray Paik
Sean McGivern
Jacob Vosmaer
Jose Torres
Tiago Botelho
Alejandro Rodríguez
William George
Michael Bisbjerg
Dylan Griffith
Valery Sizov
Balasankar "Balu" C
Marcel Amirault
James Lopez
Jasper Maes
Shinya Maeda
Mark Chao
Bob Van Landuyt
Grzegorz Bizon
Rubén Dávila
Will Chandler
James Lopez
Gabriel Mazetto
Harish Ved