ActiveModel::Serialization is simple in that it recursively calls
`as_json` on each object to serialize everything. However, for a model
like a Project, this can generate a query for every single association,
which can add up to tens of thousands of queries and lead to memory
bloat.
To improve this, we can do several things:
1. We use `tree:` and `preload:` to automatically generate
a list of all preloads that could be used to serialize
objects in bulk.
2. We observe that a single project has many issues, merge requests,
etc. Instead of serializing everything at once, which could lead to
database timeouts and high memory usage, we take each top-level
association and serialize the data in batches.
For example, we serialize the first 100 issues and preload all of
their associated events, notes, etc. before moving onto the next
batch. When we're done, we serialize merge requests in the same way.
We repeat this pattern for the remaining associations specified in
import_export.yml.
This brings a significant refactor to how
we handle `import_export.yml`, merge it with EE
and how we handle that for reader and saver.
This is meant to simplify the code, and remove
a ton of conditions to handle different models
of the structure.
This is also meant to prepare the structure
to extend it much easier, like adding `preload:`
or additional object types when needed.
This does not change the behavior of import/export,
rather unifies and simplifies the current implementation.
Most of the project templates bundled with GitLab have an empty string
set for `external_authorization_classification_label` in their
`project.json`. When imported this overrides the default label set on
the instance, causing the label to appear empty.
This change sets empty labels to nil during import, allowing the
default label to override it and be applied to the project.
Detect if pipeline runs for a GitHub pull request
When using a mirror for CI/CD only we register a pull_request
webhook. When a pull_request webhook is received, if the
source branch SHA matches the actual head of the branch in the
repository we create immediately a new pipeline for the
external pull request. Otherwise we store the
pull request info for when the push webhook is received.
When using "only/except: external_pull_requests" we can detect
if the pipeline has a open pull request on GitHub and create or
not the job based on that.
Previously imports would fail if a merge request note included a
suggestion with an
`ActiveRecord::HasManyThroughCantAssociateThroughHasOneOrManyReflection`
exception.
This was happening because suggestions were listed as a descendant of
merge requests, but this doesn't work because suggestions are directly
associated with notes, not merge requests, and that association is lost.
Rails also disallows creating intializing a has_many association through
a different object.
We fix this by making `suggestions` a child of `notes` within a merge
request. This doesn't fix previously broken exported project exports,
but new exports will work.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65880
- Added Importable to models/list.rb
- Did unless: :importable? on board validation
- Created changelog
- Modified haml to show issue boards are importable
- Added needed spec tests
- Modified project.json to include board information
- Added relevant models to all_models
- Added relevant models to import_export
- Added relevant models to safe_model_attributes
- Fix `O(n)` complexity of `append_or_update_attribute`,
we append objects to an array and re-save project
- Remove the usage of `keys.include?` as it performs `O(n)`
search, instead use `.has_key?`
- Remove the usage of `.keys.first` as it performs a copy
of all keys, instead use `.first.first`
A project can have the same `LfsObject` linked with up to three
`LfsObjectsProject` records. Each of these records would be for a
different repository, recorded in the `repository_type` property. The
different repositories at time of writing are "project", "wiki", and
"design". See https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13894
This change exports the list of `repository_type`s as a JSON mapping of
oid => repository_types, which are imported to recreate the correct
`LfsObjectsProject` records.
https://gitlab.com/gitlab-org/gitlab-ee/issues/11090
This MR introduces tracking of the `rebase_jid` for merge requests. As
with `merge_ongoing?`, `rebase_in_progress?` will now return true if a
rebase is proceeding in sidekiq.
After one release, we should remove the Gitaly-based lookup of rebases.
It is much better to track this kind of thing via the database.
This backports all EE schema changes to CE, including EE migrations,
ensuring both use the same schema.
== Updated tests
A spec related to ghost and support bot users had to be modified to make
it pass. The spec in question assumes that the "support_bot" column
exists when defining the spec. In the single codebase setup this is not
the case, as the column is backported in a later migration. Any attempt
to use a different schema version or use of "around" blocks to
conditionally disable specs won't help, as reverting the backport
migration would also drop the "support_bot" column. Removing the
"support_bot" tests entirely appears to be the only solution.
We also need to update some foreign key tests now that we have
backported the EE columns. Fortunately, these changes are very minor.
== Backporting migrations
This commit moves EE specific migrations (except those for the Geo
tracking database) and related files to CE, and also removes any traces
of the ee/db directory.
Some migrations had to be modified or removed, as they no longer work
with the schema being backported. These migrations were all quite old,
so we opted for removing them where modifying them would take too much
time and effort.
Some old migrations were modified in EE, while also existing in CE. In
these cases we took the EE code, and in one case removed them entirely.
It's not worth spending time trying to merge these changes somehow as we
plan to remove old migrations around the release of 12.0, see
https://gitlab.com/gitlab-org/gitlab-ce/issues/59177 for more details.
This resolves an "ArgumentError: comparison of String with 0 failed"
issue where the visibility_level is stored as a string in the project
import data because the value comes directly from the Web form. This
problem happened upon creating a project from a template or restoring a
project.
We now cast the value to an integer to guard against these kinds of
failures.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/61692
This new table will be used to store the external_dashboard_url which
allows users to add a link to their external dashboards (ex Grafana)
to the Metrics dashboard.
Renamed UrlValidator to AddressableUrlValidator to avoid 'url:' naming collision with ActiveModel::Validations::UrlValidator in 'validates' statement.
Make use of the options attribute of the parent class ActiveModel::EachValidator.
Add more options: allow_nil, allow_blank, message.
Renamed 'protocols' option to 'schemes' to match the option naming from UrlValidator.
Adds the groundwork needed in order to persist multi-line suggestions,
while providing the parsing strategy which will be reused for the
**Preview** as well.
These are data columns that store runtime configuration
of build needed to execute it on runner and within pipeline.
The definition of this data is that once used, and when no longer
needed (due to retry capability) they can be freely removed.
They use `jsonb` on PostgreSQL, and `text` on MySQL (due to lacking
support for json datatype on old enough version).
Webhook URLs were recently encrypted in the database via
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21645, and as a
result they are no longer exported. When an exported project with
Webhooks is imported, the project import service will fail with
`URI::InvalidURIError: bad URI(is not URI?)` due to a blank URL.
We avoid this by disabling the export/import of Webhooks in
the first place.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/53513
This commit adds a name to each release, defaulting it to tag name,
keeps track of the SHA when a new release is created and tracks the
current user as release author.
Determined by running the script:
```
included = `git grep --name-only ShellAdapter`.chomp.split("\n")
used = `git grep --name-only gitlab_shell`.chomp.split("\n")
included - used
```
GitLab Bot
DJ Mountney
Kamil Trzciński
Will Chandler
Fabio Pitino
dineshpanda
Brett Walker
Stan Hu
Jason Colyer
Luke Duncalfe
Ash McKenzie
George Koltsov
George Koltsov
Mayra Cabrera
Nick Thomas
Yorick Peterse
Shinya Maeda
Robert Speicher
Liam McAndrew
James Lopez
Heinrich Lee Yu
charlieablett
Reuben Pereira
Francisco Javier López
Thong Kuah
John Jarvis
Oswaldo Ferreira
Igor Drozdov
Pirate Praveen
Andrew Newdigate
James Lopez
Lin Jen-Shin
Douwe Maan
Heinrich Lee Yu
Alessio Caiazza
Rémy Coutable
Grzegorz Bizon
Tiago Botelho
Steve Azzopardi
gfyoung
George Tsiolis