Security and safety improvements for gitlab-workhorse integration
Companion to https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/60
- Use a custom content type when sending data to gitlab-workhorse
- Verify (using JWT and a shared secret on disk) that internal API requests came from gitlab-workhorse
This will allow us to build features in gitlab-workhorse that require
more trust, and protect us against programming mistakes in the future.
This is designed so that no action is required for installations from
source. For omnibus-gitlab we need to add code that manages the shared
secret.
See merge request !5907
Smartly calculate real running time and pending time
## What does this MR do?
Try to smartly calculate the running time and pending time for pipelines, instead of just use wall clock time from start to end. The algorithm is based on:
> Suppose we have A, B, and C jobs:
> * A: from 1 to 3
> * B: from 2 to 4
> * C: from 6 to 7
> The processing time should be accumulated from 1 to 4, and 6 to 7, totally 4, excluding retires, and calculate on `%w[success failed running canceled]` jobs (if a job is not finished yet, assume it's `Time.now`)
## Are there points in the code the reviewer needs to double check?
I would actually like to test `Gitlab::Ci::PipelineDuration#process_segments`, but it's a private method right now and it's not very convenient to test it. Is there a way to test it without changing the original code too much? Note that I would like to avoid saving merged segments because it's not used and should be garbage collected.
## Screenshots:
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- Tests
- [x] Added for this feature/bug
## What are the relevant issue numbers?
Closes#18260, #19804
See merge request !6084
Pass dependencies to CI configuration nodes
## What does this MR do?
This MR makes it possible to pass dependencies to CI configuration nodes.
## What are the relevant issue numbers?
See #15060
## Does this MR meet the acceptance criteria?
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
See merge request !6009
* master: (414 commits)
Remove suggested colors hover underline
Fix markdown anchor icon interaction
Fix expiration date picker after update
Refactored code to rely less on IDs that could change
Move CHANGELOG entry for !5858 from 8.11 to 8.12
Hides merge request section in edit project when disabled
Fix a typo
Change minimum Unicorns required to two
Update memory requirements
Added `.term-bold` declaration.
Change the inline code to codeblocks for the new features doc guideline
Fix GitLab import button
Rename behaviour to behavior in bug issue template for consistency
Convert datetime coffeescript spec to ES6
Align add button on repository view
Update CHANGELOG with 8.11.4 entries.
removed null return - renamed 'placeTop' to 'placeProfileAvatarsToTop'
Refactor Ci::Build#raw_trace
Move CHANGELOG entry to a proper version
Change widths of content in MR pipeline tab
...
Conflicts:
lib/gitlab/ci/config/node/jobs.rb
* upstream/master: (289 commits)
Fix a typo
Change minimum Unicorns required to two
Update memory requirements
Change the inline code to codeblocks for the new features doc guideline
Update CHANGELOG with 8.11.4 entries.
removed null return - renamed 'placeTop' to 'placeProfileAvatarsToTop'
Change widths of content in MR pipeline tab
Add curve to generic commit status pipeline
Rubocop syntax 2.3
Some minor updates for upgrade guides for 8.12.
Remove inconsistent font weight for sidebar's labels
Replace play icon font with svg
Project tools visibility level
Added todo filter tests
Fixed project filtering
Review changes, simplified dropdown init
Removed select2 from todos feature spec
Removed inline JS and improved dropdown labels
Added type and action dropdowns, need to finalize by removing all inline and polishing off the selected dropdown states
Completed project filter dropdown, still need to move it from inline to ProjectSelect.js (or different)
...
Handle non-UTF-8 conflicts gracefully
## What does this MR do?
If a conflict file isn't in a UTF-8-compatible encoding, we can't resolve it in the UI.
## What are the relevant issue numbers?
Closes#21247.
See merge request !5961
Remove gitorious
## What does this MR do?
Remove gitorious as import source
## Are there points in the code the reviewer needs to double check?
Did I remove everything?
## Why was this MR needed?
This button yielded a 404
## What are the relevant issue numbers?
Closes#17062
/cc @JobV
See merge request !5866
These can't be resolved in the UI because if they aren't in a UTF-8
compatible encoding, they can't be rendered as JSON. Even if they could,
we would be implicitly changing the file encoding anyway, which seems
like a bad idea.
Fix line commenting for the initial commit
## What does this MR do?
Support line positions on the initial commit, where we can't compare because there's no parent commit.
## Are there points in the code the reviewer needs to double check?
I chose to use the blank SHA to represent the initial commit, but it could as easily be the same SHA. I just thought this was clearer.
## Why was this MR needed?
People couldn't add line comments to the initial commit!
## What are the relevant issue numbers?
Closes#20895.
## Screenshots (if relevant)
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added (N/A, regression)
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md) (N/A)
- [x] API support added (N/A)
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !5900
Render coverage badge using latest successful pipeline
## What does this MR do?
This MR make test coverage badge to report value for the latest successful pipeline, instead of the latest one, regardless the status.
Latest pipeline is often running, which makes coverage report inaccurate. Latest pipeline can be also the failed one, which may mean that not all stages got processed, therefore coverage report can be inaccurate as well.
This also improves coverage badge performance because it is not necessary to touch repository to get recent SHA on the branch.
## Why was this MR needed?
See #21013
## What are the relevant issue numbers?
Closes#21013
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !5862
GitLab Performance Monitoring is now able to track custom events not
directly related to application performance. These events include the
number of tags pushed, repositories created, builds registered, etc.
The use of these events is to get a better overview of how a GitLab
instance is used and how that may affect performance. For example, a
large number of Git pushes may have a negative impact on the underlying
storage engine.
Events are stored in the "events" measurement and are not prefixed with
"rails_" or "sidekiq_", this makes it easier to query events with the
same name triggered from different parts of the application. All events
being stored in the same measurement also makes it easier to downsample
data.
Currently the following events are tracked:
* Creating repositories
* Removing repositories
* Changing the default branch of a repository
* Pushing a new tag
* Removing an existing tag
* Pushing a commit (along with the branch being pushed to)
* Pushing a new branch
* Removing an existing branch
* Importing a repository (along with the URL we're importing)
* Forking a repository (along with the source/target path)
* CI builds registered (and when no build could be found)
* CI builds being updated
* Rails and Sidekiq exceptions
Fixesgitlab-org/gitlab-ce#13720
Fix attribute inclusion import/export config ignored in some cases
In the `import_export.yml` file we define the inclusion of some of the attributes. For some reason, this isn't working in certain cases - very unfortunate this includes `user`. This has been introduced in 8.10.3.
Related https://gitlab.com/gitlab-org/gitlab-ce/issues/20802
See merge request !1982
- Removed unnecessary column from `SpamLog`
- Moved creation of SpamLogs out of its own service and into SpamCheckService
- Simplified code in SpamCheckService.
- Moved move spam related code into Spammable concern
Other improvements:
- Ensure slash commands autocomplete doesn't break when noteable_type is not given
- Slash commands: improve autocomplete behavior and /due command
- We don't display slash commands for note edit forms.
- Add tests for reply by email with slash commands
- Be sure to execute slash commands after the note creation in Notes::CreateService
Signed-off-by: Rémy Coutable <remy@rymai.me>
- Return only slash commands that make sense for the current noteable
- Allow slash commands decription to be dynamic
Other improvements:
- Add permission checks in slash commands definition
- Use IssuesFinder and MergeRequestsFinder
- Use next if instead of a unless block, and use splat operator instead of flatten
Signed-off-by: Rémy Coutable <remy@rymai.me>
Some important things to note:
- commands are removed from noteable.description / note.note
- commands are translated to params so that they are treated as normal
params in noteable Creation services
- the logic is not in the models but in the Creation services, which is
the right place for advanced logic that has nothing to do with what
models should be responsible of!
- UI/JS needs to be updated to handle notes which consist of commands
only
- the `/merge` command is not handled yet
Other improvements:
- Don't process commands in commit notes and display a flash is note is only commands
- Add autocomplete for slash commands
- Add description and params to slash command DSL methods
- Ensure replying by email with a commands-only note works
- Use :subscription_event instead of calling noteable.subscribe
- Support :todo_event in IssuableBaseService
Signed-off-by: Rémy Coutable <remy@rymai.me>
- Add match line header to expected result for `File#sections`.
- Lowercase CSS colours.
- Remove unused `diff_refs` keyword argument.
- Rename `parent` -> `parent_file`, to be more explicit.
- Skip an iteration when highlighting.
DRY code + fix rubocop
Add more test cases
Append to changelog
DRY changes list
find_url service for merge_requests
use GET for getting merge request links
remove files
rename to get_url_service
reduce loop
add test case for cross project
refactor tiny thing
update changelog
This change simplifies a Pipeline processing by introducing a special new status: created.
This status is used for all builds that are created for a pipeline.
We are then processing next stages and queueing some of the builds (created -> pending) or skipping them (created -> skipped).
This makes it possible to simplify and solve a few ordering problems with how previously builds were scheduled.
This also allows us to visualise a full pipeline (with created builds).
This also removes an after_touch used for updating a pipeline state parameters.
Right now in various places we explicitly call a reload_status! on pipeline to force it to be updated and saved.
* upstream/master: (233 commits)
Fix awardable button mutuality loading spinners
Update CHANGELOG for 8.10.5
Clean up project destruction
Small refactor of doc/development/README.md
Avoid commit lookup on diff_helper
Removed extra newline from redis_spec.rb
Used cached value of project count to reduce DB load
Remove duplicate link_to statements
Mention add_column_with_default in downtime guide
Add missing space to generic badge template
Rename `run` task helper method to prevent conflict with StateMachine
Add a method in Project to return a cached value of total count of projects
spellcheck
Add svg guidelines to ui guide
Add Changelog entry for Grape upgrade [ci skip]
Fix Grape tests.
Retain old behavior
Update Grape from 0.13.0 to 0.15.0.
adds second batch of tests changed to active tense
fixes part1 of files to start using active tense
...
Use badge image template instead of using separate images
## What does this MR do?
Makes it possible to use template for badge instead of having multiple files.
## Are there points in the code the reviewer needs to double check?
We also have a deprecated badge in `controllers/ci/projects_controller.rb`. We decided to leave it until 9.0, so we still have images in `public/ci/` until 9.0.
## Why was this MR needed?
We are going to implement build coverage badge, and we do not want to store 101 SVG images for each percentage value.
## What are the relevant issue numbers?
#3714
## Screenshots (if relevant)
## Does this MR meet the acceptance criteria?
- [ ] ~~[CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added~~ (refactoring)
- [ ] ~~[Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~
- [ ] ~~API support added~~
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] ~~[Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)~~ (refactoring)
See merge request !5520
* master: (52 commits)
remove offending empty line
Namespace EnableDeployKeyService under Projects
Update version_sorter and use new interface for faster tag sorting
Avoid to show the original password field when password is automatically seted
Support pending invitation project members importing projects
Added concern for a faster "cache_key" method
Update templates
"This file is managed by gitlab-ctl. Manual changes will be erased!"
Remove legacy Ci::StaticModel we do not use anymore
Revert "Defend against 'Host' header injection"
Simplify feature introduction note
Add migration-related tips to the "Merge Request Guidelines" doc
Enable Style/SpaceAroundEqualsInParameterDefault cop
Enable Style/EmptyLinesAroundClassBody cop
Enable Style/EmptyLinesAroundModuleBody cop
Ensure we are looking for the right dropdown inside the form wrapper
Set for for labels and ID for dropdowns on create form
Fix .panel-title style
Refine selector for form submit button
Fix spelling. `braches` to `branches`
...
Enable some Rubocop cops related to new lines
## What does this MR do?
This MR enabled two additional Rubocop cops:
Keeps track of empty lines around block bodies.
`Style/EmptyLinesAroundBlockBody`
Keeps track of empty lines around method bodies.
` Style/EmptyLinesAroundMethodBody`
See merge request !5637
Stop 'git push' over HTTP early
Before this change we always let users push Git data over HTTP before
deciding whether to accept to push. This was different from pushing
over SSH where we terminate a 'git push' early if we already know the
user is not allowed to push.
This change let Git over HTTP follow the same behavior as Git over
SSH. We also distinguish between HTTP 404 and 403 responses when
denying Git requests, depending on whether the user is allowed to know
the project exists.
See merge request !5639
* master: (363 commits)
Added changelog item for issuable form dropdowns
Add 'run tests' docs from GDK
Bump gitlab_git to lazy load compare commits
Add examples to repository files API (!5465)
Ignore URLs starting with // (!5677)
Add failing test for #7032
Update timeago to shorter representation
Add missing DOWNTIME constant to the AddTimestampsToMembersAgain migration
Added guide about migrations and downtime
Update CHANGELOG for 8.10.4
Add a data migration to fix some missing timestamps in the members table (again)
Move abilities by subject class to a dedicated method
Remove unnecessary empty line after css var
Set consistency in list text height css
Add description to text/plain emails
Fix Rename `add_users_into_project` and `projects_ids`
fix spec
Underscore variable to camelCase
using shared path for project import uploads and refactored gitlab remove export worker
Structure the development documentation
...
Developer cannot push to protected branch when project is empty or he has not been granted permission to do so
This MR was created following !1979 and !1978Closes#14898
See merge request !1980
Move CI job config entries from legacy to new config
## What does this MR do?
This MR extracts jobs configuration logic from legacy CI config processor to the new code.
## What are the relevant issue numbers?
#15060
## Does this MR meet the acceptance criteria?
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
See merge request !5087
1. Remove `Project#developers_can_push_to_protected_branch?` since it
isn't used anymore.
2. Remove `Project#developers_can_merge_to_protected_branch?` since it
isn't used anymore.
1. The crux of this change is in `UserAccess`, which looks through all
the access levels, asking each if the user has access to push/merge
for the current project.
2. Update the `protected_branches` factory to create access levels as
necessary.
3. Fix and augment `user_access` and `git_access` specs.
Implement #3243 New Issue by email
So we extend Gitlab::Email::Receiver for this new behaviour,
however we might want to split it into another class for better
testing it.
Another issue is that, currently it's using this to parse project
identifier:
Gitlab::IncomingEmail.key_from_address
Which is using:
Gitlab.config.incoming_email.address
for the receiver name. This is probably `reply` because it's used
for replying to a specific issue. We might want to introduce another
config for this, or just use `reply` instead of `incoming`.
I'll prefer to introduce a new config for this, or just change
`reply` to `incoming` because it would make sense for replying to
there, too.
The email template used in tests were copied and modified from:
`emails/valid_reply.eml` which I hope is ok.
/cc @DouweM #3243
See merge request !3363
This reduces the overhead of the method instrumentation code primarily
by reducing the number of method calls. There are also some other small
optimisations such as not casting timing values to Floats (there's no
particular need for this), using Symbols for method call metric names,
and reducing the number of Hash lookups for instrumented methods.
The exact impact depends on the code being executed. For example, for a
method that's only called once the difference won't be very noticeable.
However, for methods that are called many times the difference can be
more significant.
For example, the loading time of a large commit
(nrclark/dummy_project@81ebdea5df)
was reduced from around 19 seconds to around 15 seconds using these
changes.
* upstream/master: (45 commits)
Replace reject_blocked with reject_blocked! in callbacks.
Fix Project#to_param to keep invalid project suitable for use in URLs
Update CHANGELOG
Add feature specs for edit project settings
Fix renaming repository when name contains invalid chars under settings
Change requests_profiles resource constraint to catch virtually any file
Allow skipping users in autocomplete
Fix typo in CHANGELOG
Update CHANGELOG
Respective cache is now expired when creating a new branch
Update CHANGELOG
Unify HTML format in static error pages
Make error pages responsive design
Move color-logic into HipchatService#HipchatService
Depened on exact version of SimpleCov when patched
Refactor spam validation to a concern that can be easily reused and improve legibility in `SpamCheckService`
Refactor `SpamCheckService` to make it cleaner and clearer.
Submit all issues on public projects to Akismet if enabled.
Submit new issues created via the WebUI by non project members to Akismet for spam check.
Upgrade Bullet from 5.0.0 to 5.2.0.
...
* upstream/master: (620 commits)
Added '*.js.es6 gitlab-language=javascript' to .gitattributes
Fix CI status icon link underline
Update CHANGELOG after 8.10.1
Add CHANGELOG
Add es6 gem
Instrument Nokogiri parsing methods
Fix backup restore
Use project ID in repository cache to prevent stale data from persisting across projects
Add iid to MR API response
`WikiPage` should have a slug even when not persisted.
ES6ify all the things!
Make fork counter always clickable (!5463)
Revert "Merge branch '17073-tagscontroller-index-is-terrible-response-time-goes-up-to-5-…"
Fix CHANGELOG
Add spec for dashes in paths
Fix Error 500 when creating Wiki pages with hyphens or spaces
Add links to the real markdown.md file for all GFM examples
Remove magic comments from Ruby files (!5456)
Ignore invalid trusted proxies in X-Forwarded-For header
remove search_id for label dropdown filter
...
* master: (183 commits)
Add a spec for #20079.
Skip repository storage path valitaions on test environment
Use Pathname to make the repository storage path validations more robust
Update to gitlab_git 10.4.1 and take advantage of preserved Ref objects
Change nav link snippet controller
Reduce min width of pipeline table
Retrieve rendered HTML from cache in one request
Explain CI_PROJECT_NAMESPACE better
Bump vmstat version to fix issues reporting on FreeBSD
Fix sha icon positioning on safari
Don't drop in DropAndReaddHasExternalWikiInProjects
Mobile view for commit status
Fix ci icons getting cut off
Update CHANGELOG
Extract helper methods to clean up RepositoryArchiveCleanUpService spec
Use Dir.mktmpdir instead of FileUtils.mkdir_p in the spec
Fix firefox rendering of SVGs
Fix icons on commits page and builds page
Add new fork SVG to fix weird styling of other SVGs
Bug fixes
...
Add support for inline videos in issue, MR and notes (on issue, commit, MR, and MR diff)
## What does this MR do?
It adds support for inline videos in issue, MR and notes (on issue, commit, MR, and MR diff). Most of the work was done by @hayesr in !3508 but a few improvements were still missing.
## Why was this MR needed?
To be able to play uploaded videos in GitLab!
## What are the relevant issue numbers?
Closes#4142.
## Screenshots
### Video players
-----
-----
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- Tests
- [x] Test `VideoLinkFilter`
- [x] Test in `spec/features/markdown_spec.rb`
- [x] Improve `spec/uploaders/file_uploader_spec.rb`
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !5215
Added checks for migration downtime
This adds a set of checks that check/list which migrations require downtime (or not). It also comes with a CI task that fails should a migration not be tagged properly.
Fixes#14545
See merge request !4911
These new checks can be used to check if migrations require downtime or
not (as tagged by their authors). In CI this compares the current branch
with master so migrations added by merge requests are automatically
verified.
To check the migrations added since a Git reference simply run:
bundle exec rake gitlab:db:downtime_check[GIT_REF]
* master: (321 commits)
Fix the Sentry spam from CSP violations by disabling it.
Limit git rev-list output count to one in forced push check
Ensure Owners are included in the scope for authorized_projects
Fix alignment of icons on project page
Fix ci_status_helper_spec to look for new SVGs
use 2.0.5, actually (2.0.4 was a bad release)
upgrade rouge to 2.0.4
Fix help page paths to make sure shortcuts and the UI help page work.
fixes an issue cause by a bad merge
Vertically align status icon within table
Add new icons for every CI status
Add global style for running icon
Align running icon in merge request
Add new running icon; add a bunch of styles to get svg to match existing fa icons
Improve code design
Fix broken builds_for_ref
Move when tests before to make it no conflict with manual-actions
Use value of `yaml_variables` and `when` from config_processor if undefined
Add CHANGELOG entry
CHANGELOG item
...
Conflicts:
lib/ci/gitlab_ci_yaml_processor.rb
spec/lib/ci/gitlab_ci_yaml_processor_spec.rb
added spec for avatar saver
avatar saver!
added avatar restorer spec
fix spec
added avatar restorer class
fix export service
fix warnings, added changelog
fix spec
some refactoring based on feedback
fixed a few issues after testing i/e avatar
WIP - trying to replicate UTF-8 error
fix spec
fixing encoding issue and another spec, to do with MR diffs
fix issue and spec failure
Add changelog and bumped up I/E version
fix spec based on feedback - omitted target project
James Lopez
Jacob Vosmaer (GitLab)
Rémy Coutable
Dmitriy Zaporozhets
Andrew Smith
Sean McGivern
Lin Jen-Shin
Valery Sizov
Grzegorz Bizon
Douwe Maan
Felipe Artur
Douglas Barbosa Alexandre
Robert Speicher
Paco Guzman
Z.J. van de Weg
Yorick Peterse
Douwe Maan
Robert Speicher
Kamil Trzcinski
Ruben Davila
Patricio Cano
Alejandro Rodríguez
Scott Le
Stan Hu
tiagonbotelho
Gabriel Mazetto
Gabriel Mazetto
Rémy Coutable
Timothy Andrew