Move global ci entries handling from legacy to new config
## What does this MR do?
This MR moves responsibility of handling global CI config entries (like `image`, `services`), from legacy `GitlabCiYamlProcessor` to new CI Config
## Why was this MR needed?
This is the next iteration of CI configuration refactoring
## What are the relevant issue numbers?
#15060
## Does this MR meet the acceptance criteria?
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
See merge request !4820
Note: This feature was developed independently on master while this was
in review. I've removed the conflicting bits and left the relevant
additions, mainly a test for `Gitlab::Git::Hook`. The original commit
message follows:
1. `gitlab-shell` outputs errors to `stderr`, but we weren't using this
information, prior to this commit. Now we capture the `stderr`, and
display it in the flash message when branch creation fails.
2. This can be used to display better errors for other git operation
failures with small tweaks.
3. The return value of `Gitlab::Git::Hook#trigger` is changed from a
simple `true`/`false` to a tuple of `[status, errors]`. All usages
and tests have been updated to reflect this change.
4. This is only relevant to branch creation _from the Web UI_, since SSH
and HTTP pushes access `gitlab-shell` either directly or through
`gitlab-workhorse`.
5. A few minor changes need to be made on the `gitlab-shell` end. Right
now, the `stderr` message it outputs is prefixed by "GitLab: ", which
shows up in our flash message. This is better removed.
* master: (98 commits)
Enable Style/EmptyLines cop, remove redundant ones
Update CHANGELOG
Cache results from jQuery selectors to retrieve namespace name
Fix import button when import fail due the namespace already been taken
Fix snippets comments not displayed
Fix emoji paths in relative root configurations
Exclude requesters from Project#members, Group#members and User#members
Upgrade Thin from 1.6.1 to 1.7.0.
Many squashed commits
Cache autocomplete results
Upgrade Sidekiq from 4.1.2 to 4.1.4.
Upgrade seed-fu from 2.3.5 to 2.3.6
use has_many relationship with events
Support creating a todo on issuables via API
Expose target, filter by state as string
Add todos API documentation and changelog
Improve the request / withdraw access button
Metrics for Rouge::Plugins::Redcarpet and Rouge::Formatters::HTMLGitlab
Groundwork for Kerberos SPNEGO (EE feature)
Update CHANGELOG 8.9.5 for runners related fixes
...
Fix emoji paths in relative root configurations
## What does this MR do?
If a site specifies a relative URL root, emoji files would omit the path from the URL, leading to lots of 404s.
## Are there points in the code the reviewer needs to double check?
At first, I tried to use `ActionView::Helpers::AssetUrlHelper.asset_url` since this is what it's intended to do. But this helper function is extremely slow, and it took minutes to generate the URLs for the hundreds of links needed for each emoji.
## Why was this MR needed?
Because emojis were broken in relative URL installations
## What are the relevant issue numbers?
#15642
## Does this MR meet the acceptance criteria?
- [X] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
- [X] Added for this feature/bug
- [x] All builds are passing
- [X] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [X] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [X] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !5027
This stands as an alternative to using OAuth to access a user's Github
repositories. This is setup in such a way that it can be used without OAuth
configuration.
From a UI perspective, the how to import modal has been replaced by a full
page, which includes a form for posting a personal access token back to the
Import::GithubController.
If the user has logged in via GitHub, skip the Personal Access Token and go
directly to Github for an access token via OAuth.
Fix privilege escalation issue with OAuth external users
Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/19312
This MR fixes a privilege escalation issue, where manually set external users would be reverted back to internal users if they logged in via OAuth and that provider was not in the `external_providers` list.
/cc @douwe
See merge request !1975
Enable Style/UnneededCapitalW Rubocop cop
## What does this MR do?
This MR enables Rubocop cop that checks if `%W[]`, which supports interpolation, is needed.
## What are the relevant issue numbers?
#17478
See merge request !5010
If Redis were not running or USE_DB were set to false, the
application settings retrieval would fail completely. This
change only attempts to use the cache if the system actually
wants to connect to the DB and rescues any failures in talking to
Redis.
Closes#17557
No reason to split it into a separate gem when the gem barely did anything. We can use gemojione directly, making updating gemojione that much easier.
Also fix the Rake task and update gemojione to 2.6.1. This adds the EmojiOne Spring update.
Changelog: https://github.com/jonathanwiesel/gemojione/blob/master/CHANGELOG.md
Process.clock_gettime allows getting the real time in nanoseconds as
well as allowing one to get a monotonic timestamp. This offers greater
accuracy without the overhead of having to allocate a Time instance. In
general using Time.now/Time.new is about 2x slower than using
Process.clock_gettime(). For example:
require 'benchmark/ips'
Benchmark.ips do |bench|
bench.report 'Time.now' do
Time.now.to_f
end
bench.report 'clock_gettime' do
Process.clock_gettime(Process::CLOCK_MONOTONIC, :millisecond)
end
bench.compare!
end
Running this benchmark gives:
Calculating -------------------------------------
Time.now 108.052k i/100ms
clock_gettime 125.984k i/100ms
-------------------------------------------------
Time.now 2.343M (± 7.1%) i/s - 11.670M
clock_gettime 4.979M (± 0.8%) i/s - 24.945M
Comparison:
clock_gettime: 4979393.8 i/s
Time.now: 2342986.8 i/s - 2.13x slower
Another benefit of using Process.clock_gettime() is that we can simplify
the code a bit since it can give timestamps in nanoseconds out of the
box.
In any case if just want the value which is always ‘gitlab’
require 'benchmark/ips'
Project.first # To load database things
GitlabIssueTrackerService.first # To load database things
Benchmark.ips do |x|
x.config(:time => 5, :warmup => 2)
x.report("current") do
Project.new.default_issue_tracker.to_param
end
x.report("") do
'gitlab'
end
x.compare!
end
Calculating -------------------------------------
current 4.000 i/100ms
30.938k i/100ms
-------------------------------------------------
current 47.298 (±10.6%) i/s - 232.000
4.366M (±20.9%) i/s - 17.202M
Comparison:
: 4366456.0 i/s
current: 47.3 i/s - 92318.26x slower
* master: (352 commits)
Display last commit of deleted branch in push events (!4699)
add changelog
add missing attribute to attr_encrypted so it is fully backwards-compatible
Add "GitLab team members only" to diagram link
doc: note that .gitattributes uses default branch
use the conf lexer so we have highlighted comments
first draft of docs
support cgi style options, such as erb?parent=json
move the path alias to a more appropriate location
make #custom_language private
appease rubocop
add an alias for Snippet#path
appease rubocop
check the tag so that an instance will pass too
fix the spec, using project.change_head
Revert "bump the master sha for gitlab-test!9"
bump the master sha for gitlab-test!9
add custom highlighting via .gitattributes
Rename Licenses API to License Templates API
Check for conflict with wiki projects when creating a new project.
...
Improve validations and error handling in new CI config entries
## What does this MR do?
This MR improves validation in new CI config.
## Why was this MR needed?
With that it will be easier to handle errors during validation and post-processing.
## What are the relevant issue numbers?
This is a continuation of #15060
See merge request !4560
We assume that when someone adds a key for the configuration entry, but
does not provide a valid value, which causes entry to be `nil`, then
entry should be considered as the undefined one. We also assume this is
semantically correct, this is also backwards compatible with legacy CI
config processor.
See issue #18775 for more details.
Fix subsequent SAML sign ins
Fixes a bug when `auto_link_ldap_user` is `true` that causes SAML users to be unable to sign in a second time.
Fix the problem for https://gitlab.zendesk.com/agent/tickets/22546
See merge request !4718
Resolve "Destroying a project causes post_decline_request to be executed"
## What does this MR do?
Ensure we don't send "access request declined" to access requesters when a project is deleted.
## Are there points in the code the reviewer needs to double check?
I've created a service to decouple the notification sending from the AR model.
## Why was this MR needed?
Because there was an issue.
## What are the relevant issue numbers?
Fixes#18755, #18750.
## Does this MR meet the acceptance criteria?
- [x] No CHANGELOG needed.
- [x] Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !4744
Allow to fetch LFS from CI
## What does this MR do?
This adds support for fetching LFS object from CI jobs (mostly it's made for supporting GitLab CI).
## What is left?
- [x] Write tests covering a new authorization mechanism
cc @grzesiek @marin
See merge request !4465
This commit builds on the groundwork in
ee008e300b1ec0abcc90e6a30816ec0754cea0dd, which refactored the backend
so the same code could be used for new dropdowns. In this commit its
used for templates for the `.gitlab-ci.yml` files.
* master: (345 commits)
use rails root join
fixed a couple of errors spotted in production
Fix RangeError exceptions when referring to issues or merge requests outside of max database values
Fix bug in `WikiLinkFilter`.
Small frontend code fixes and restore 8a2d88f commit
Warn about admin privilege to disable GitHub Webhooks
Listing GH Webhooks doesn't stop import process for non GH admin users
fixup! updated docs for api endpoint award emoji
Update CHANGELOG
Ensure Todos counters doesn't count Todos for projects pending delete
Add endpoints for award emoji on notes
Sort API endpoints and implement feedback
Add endpoints for Award Emoji
Fixed issue with assignee dropdown not selecting correctly
Removed update method Re-structured controller spec Renamed issuable param to issuable_id
Fix clibpoard buttons on "Check out branch" modal.
Track method call times/counts as a single metric
Cache todo counters (pending/done)
Fix a 'wrong number of arguments' error
Added missing mount point for Sidekiq Metrics API, after it got lost on rebase.
...
When using #XYZ in Markdown text, if XYZ exceeds the maximum value of a signed 32-bit integer, we
get an exception when the Markdown render attempts to run `where(iids: XYZ)`. Introduce a method
that will throw out out-of-bounds values.
Closes#18777
Previously we'd create a separate Metric instance for every method call
that would exceed the method call threshold. This is problematic because
it doesn't provide us with information to accurately get the _total_
execution time of a particular method. For example, if the method
"Foo#bar" was called 4 times with a runtime of ~10 milliseconds we'd end
up with 4 different Metric instances. If we were to then get the
average/95th percentile/etc of the timings this would be roughly 10
milliseconds. However, the _actual_ total time spent in this method
would be around 40 milliseconds.
To solve this problem we now create a single Metric instance per method.
This Metric instance contains the _total_ real/CPU time and the call
count for every instrumented method.
Allow customising of queries used for `update_column_in_batches`
This MR makes two changes to `add_column_with_default` and `update_column_in_batches`:
1. `add_column_with_default` no longer wraps the entire set of updates in a single transaction, preventing any locks from sticking around for the duration of the entire transaction
2. `update_column_in_batches` now takes a block which can be used to customise the queries. This uses Arel as messing with raw SQL strings is a total pain
In !4381 there's a need for updating existing rows/columns in a table in batches using a custom `WHERE` condition. Without the changes in this MR this would not be possible.
See merge request !4680
This changes update_column_in_batches to ensure it always updates all
rows now. These changes also allow for an extra SELECT query to be
removed, nor does it use the row count for determining offsets and the
likes; instead it's only used to determine the batch size.
* master: (189 commits)
Update CHANGELOG for !4659
Center the header logo for all Devise emails
Add previews for all customized Devise emails
Customize the Devise `unlock_instructions` email
Customize the Devise `reset_password_instructions` email
Customize the Devise `password_change` emails
Use gitlab-git 10.2.0
Use Git cached counters on project show page
Fix indentation scss-lint errors
Added title attribute to enties in tree view Closes#18353
Banzai::Filter::ExternalLinkFilter use XPath
Reduce queries in IssueReferenceFilter
Use gitlab_git 10.1.4
Fixed ordering in Project.find_with_namespace
Fix images in emails
Banzai::Filter::UploadLinkFilter use XPath
Turn Group#owners into a has_many association
Make project_id nullable
...
This follows a standard `ActiveModel` pattern of creating a custom
validators. We use `ActiveModel::EachValidator` here that reuses methods
provided by `LegacyValidationHelpers`.
We will remove `LegacyValidationHelpers` on some point in the future, at
the later stages of CI configuration refactoring. It may be possible
to rewrite custom validators to use format like:
`validates :config, array_of: String`
Currently the RegexpError can be raised when processing next stage which leads to 500 in different places of code base.
This adds early check that regexps used in only and except are valid.
This ensures that whatever locks are acquired aren't held onto until the
end of the transaction (= after _all_ rows have been updated). Timing
wise there's also no difference between using a transaction and not
using one.
By passing a block to update_column_in_batches() one can now customize
the queries executed. This in turn can be used to only update a specific
set of rows instead of simply all the rows in the table.
Add environments and deployments
This MR is a continuation of https://gitlab.com/gitlab-org/gitlab-ce/issues/17009.
The current implementation is as follow:
1. We have two new tables: `environments` and `deployments`.
2. We have a new tab: `Environments` under `Pipelines` where you can see all you environments and add a new one.
3. We add a new option to `.gitlab-ci.yml` to track where we should create a deployment for environment.
4. If environment in `.gitlab-ci.yml` is specified it will create a deployment. **If environment does not exist it will be created.** (this got changed)
5. The deployment is always successful and shows the time of the action, in that case a build that presumably should do deployment. In the future we could extend deployment with statuses: success, failure. We could extend deployments with information that this is partial or full deployment.
6. User have to create environments that he will track first.
7. User can remove environments.
8. User can retry/rollback past deployment (in that case we retry past build). The new build when succeeds it will create a new deployment.
9. Currently environment have only one parameter: `name`. In the future it should have: `variables`, `credentials` and possibly `runners` and maybe other resources.
10. Currently deployment have this parameters: `sha`, `ref`, `deployable (in this case a build)`, `user (who triggered a deployment)`, `created_at`.
The `.gitlab-ci.yml`:
```
deploy to production:
stage: deploy
script: dpl travis...
environment: production
```
What needs to be done:
- [x] Write initial implementation
- [x] Improve implementation (@ayufan)
- [x] Write tests (@ayufan)
- [x] Improve UX of the forms (cc @markpundsack) - reviewed by @markpundsack
- [x] Improve implementation of the views (cc @jschatz1) - done by @iamphill
- [x] Write .gitlab-ci.yml documentation for `environments` - done by @ayufan
- [ ] Write user documentation (@ayufan and @markpundsack)
See merge request !4605
We can't do a lot with classes without names as we can't filter by them,
have no idea where they come from, etc. As such it's best to just ignore
these.
By default instrumentation will instrument public,
protected and private methods, because usually
heavy work is done on private method or at least
that’s what facts is showing
Generating the following tags
Grape#GET /projects/:id/archive
from Grape::Route objects like
{ :path => /:version/projects/:id/archive(.:format)
:version => “v3”,
:method => “GET” }
Use an instance variable to cache raw_path transformations.
This variable is only going to growth to the number of
endpoints of the API, not with exact different requests
We can store this cache as an instance variable because
middleware are initialised only once
Add global entry with before script to new CI config
## What does this MR do?
This MR adds a new entries to a new CI config class. It is next refactoring step after !4462.
See #15060
See merge request !4482
Paco Guzman
Rémy Coutable
Timothy Andrew
Dmitriy Zaporozhets
Valery Sizov
Grzegorz Bizon
Tomasz Maczukin
Stan Hu
James Lopez
Eric K Idema
Douwe Maan
Robert Speicher
Alejandro Rodríguez
Connor Shea
Patricio Cano
Z.J. van de Weg
Yorick Peterse
http://jneen.net/
Drew Blessing
Kamil Trzcinski
Douglas Barbosa Alexandre
Sean McGivern
Jacob Vosmaer