Fix undefined error in CI linter
## What does this MR do?
This MR fixes undefined error in CI linter.
## Does this MR meet the acceptance criteria?
- [x] [Changelog entry](https://docs.gitlab.com/ce/development/changelog.html) added
- [x] Tests added for this feature/bug
## What are the relevant issue numbers?
Closes#24759
See merge request !7650
* upstream/master: (497 commits)
Use single quote for strings
Ue svg from SVGs object
Dont trigger CI builds [ci skip]
Revert "Test only migrations"
Add custom copy for each empty stage
Fetch only one revision
Highlight nav item on hover
Test only migrations
Fix migration paths tests
Scroll CA stage panel on mobile
Fix CSS declaration
administer to administrator
Move SVGs to JS objects for easy reuse
Improve deploy command message
No enough data to Not enough data
Keep the cookie name as before
Fix variable usage
Evalute time_ago method instead of printing it
Removed button styling from restricted visibility levels and added checkboxes with icons
Do not show overview message if there’s already CA data
...
This moves the logic of detecting special repository files (e.g. a
README or a Koding configuration file) to a single class:
Gitlab::FileDetector. Moving this logic into a single place allows this
to be re-used more easily.
This commit also changes Repository#gitlab_ci_yaml so that its cached
similar to other data (e.g. the Koding configuration file).
Make job script a required configuration entry
## What does this MR do?
This MR makes a job script a required configuration entry.
## Does this MR meet the acceptance criteria?
- [x] [Changelog entry](https://docs.gitlab.com/ce/development/changelog.html) added
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
## What are the relevant issue numbers?
Closes#24575
See merge request !7566
Use `Gitlab.config.gitlab.host` over `'localhost'`
Use `Gitlab.config.gitlab.host` over `'localhost'`
This would fix long standing failures running tests on
my development machine, which set `Gitlab.config.gitlab.host`
to another host because it's not my local computer. Now I
finally cannot withstand it and decided to fix them once and
for all.
See merge request !7562
add parsing support for incoming html email
## What does this MR do?
Fixes#18388 by adding support for parsing HTML email
## Are there points in the code the reviewer needs to double check?
The new class, Gitlab::Email::HTMLParser, which needs to translate the HTML content to text and also delete replies, as they are not necessarily in the correct format to be caught by EmailReplyParser. The solution I found that should work for any HTML-formatted email is to remove all `<table>` and `<blockquote>` tags. Actual `<table>` elements (to be interpreted by markdown) should already be encoded with e.g. `<table>` - the only failure mode is if there is an *actual* HTML table in the content itself, which we wouldn't be able to support easily anyways.
The gem `html2text` traverses the HTML tree and outputs text - and markdown in the case of HTML links or images.
See merge request !7397
This would fix long standing failures running tests on
my development machine, which set `Gitlab.config.gitlab.host`
to another host because it's not my local computer. Now I
finally cannot withstand it and decided to fix them once and
for all.
* upstream/master: (236 commits)
Mention Git strategy none
Remove ToC since it's now supported in the docs portal itself
Add 8.14 to versions with further additions to review apps
Add Limitations sections to environments and review apps docs
Add link to environments docs
Fix URL to review apps docs
Add a prerequisites section, add some links
Link to NGINX example project for the time being
Get rid most of the irrelevant sections
Add note about current limitation in $CI_BUILD_REF_NAME
Add an intro and an Overview section for Review Apps
WIP review apps
Add Review apps link to CI README
Add stop environment permissions and remove delete
Add note about auto-stopping of environments
Update CHANGELOG.md for 8.13.6
Finish "Stopping envs" and "Grouping similar envs" sections
Highlight first user autocomplete option
Remove ToC and fix headings in Markdown docs
Revert "Merge branch '22680-unlabel-limit-autocomplete-to-selected-items' into 'master'"
...
Now test events return the builds instead a list of pipelines to avoid calling pipeline.builds per each and get the info. Also, added missing fields/data, URLs, and fixed specs in events spec.
- Refactored cycle analytics class to extract DB logic
- Reuse logic in new events fetcher
- Started adding cycle analytics events class and spec (still not functional)
Fix Error 500 when creating a merge request that contains an image that was deleted and added
_Originally opened at !4816 by @stanhu._
- - -
## What does this MR do?
This MR fixes an Error 500 when creating a merge request that contains an image that was deleted and added. Before, when displaying the before and after image, the code would always retrieve the image from the parent commit. However, in a diff, this could cause two different problems:
The "before" image may not actually be the image you want to compare against (regression of #14327)
It may appear as though a file was modified when it was really just added during the diff
## Are there points in the code the reviewer needs to double check?
There may be a more elegant to fix this bug.
## What are the relevant issue numbers?
Closes#3893, gitlab-org/gitlab-ee#678
See merge request !7457
Omniauth auto link LDAP user falls back to find by DN when user cannot be found by uid
Unfortunately, SAML IDs can be an LDAP UID, DN, or something else entirely. UID and DN are most common, though. This adds a fallback scenario so we first try to find a matching LDAP user by UID, then by DN. This will fix a problem for the customer in https://gitlab.zendesk.com/agent/tickets/43298
See merge request !7002
Steps to reproduce:
1. Start with a repo with an image
2. Add a commit to delete the image
3. Add another commit to replace the image with another image
In a diff comparison, we really just compare about what the image was before the diff, not
the direct parent of the last commit. This MR fixes that.
Closes#3893, gitlab-org/gitlab-ee#678
Signed-off-by: Rémy Coutable <remy@rymai.me>
Improve naming convention in ci configuration module
## What does this MR do?
This MR improves the naming convention in CI configuration module to reflect the domain design better.
## What are the relevant issue numbers?
Related to #15060
See merge request !7448
Centralize all LDAP config logic in `GitLab::LDAP::Config`. Previously,
some logic was in the Devise initializer and it was not honoring the
`user_filter`. If a user outside the configured `user_filter` signed
in, an account would be created but they would then be denied access.
Now that logic is centralized, the filter is honored and users outside
the filter are never created.
* upstream/master: (3852 commits)
Grapify token API
Fix cache for commit status in commits list to respect branches
Grapify milestones API
Grapify runners API
Improve EeCompatCheck, cache EE repo and keep artifacts for the ee_compat_check task
Use 'Forking in progress' title when appropriate
Fix CHANGELOG after 8.14.0-rc1 tag
Update CHANGELOG.md for 8.14.0-rc1
Fix YAML syntax on CHANGELOG entry
Remove redundant rescue from repository keep_around
Remove redundant space from repository model code
Remove order-dependent expectation
Minor CHANGELOG.md cleanups
Add a link to Git cheatsheet PDF in docs readme
Grapify the session API
Add 8.13.5, 8.12.9, and 8.11.11 CHANGELOG
Merge branch 'unauthenticated-container-registry-access' into 'security'
Merge branch '23403-fix-events-for-private-project-features' into 'security'
Merge branch 'fix-unathorized-cloning' into 'security'
Merge branch 'markdown-xss-fix-option-2.1' into 'security'
...
Respect project visibility settings in the contributions calendar
This MR fixes a number of bugs relating to access controls and date selection of events for the contributions calendar
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/23403
See merge request !2019
Signed-off-by: Rémy Coutable <remy@rymai.me>
disable markdown in comments when referencing disabled features
fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/23548
This MR prevents the following references when tool is disabled:
- issues
- snippets
- commits - when repo is disabled
- commit range - when repo is disabled
- milestones
This MR does not prevent references to repository files, since they are just markdown links and don't leak
information.
See merge request !2011
Signed-off-by: Rémy Coutable <remy@rymai.me>
It was previously possible for invalid credential errors to go unnoticed
in this task. Users would believe everything was configured correctly and
then sign in would fail with 'invalid credentials'. This adds a specific
bind check, plus catches errors connecting to the server. Also, specs :)
Allow owners to fetch source code in CI builds
Due to different way of handling owners of a project, they were not allowed to fetch CI sources for project.
This adds a separate code path for handling owners, that are not admins.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/23437
See merge request !6943
Use optimistic locking
## What does this MR do?
Removes the usage of pessimistic locking in favor of optimistic which is way cheaper and doesn't block database operation.
Since this is very simple change it should be safe. If we receive `StaleObjectError` message we will reload object a retry operations in lock.
However, I still believe that we need this one: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7005 as this will reduce a load on Database and FS.
This changes a behavior from:
### Pesimistic locking (previous behavior)
#### For updating
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. enqueue: (use: transition :created -> :pending)
5. [state_machine] we are in state created, we can go to pending
6. [state_machine] ci_pipeline.status = created
7. [state_machine] ci_pipeline.save
8. [state_machine] after_transition: (if for success): PipelineSuccessWorker on Sidekiq
9. release DB lock
#### If no update is required
1. SELECT * FOR UPDATE (other updates wait on this)
2. we update ci_pipeline
3. latest_build_status
4. we are in pending, we can't transition to pending, because it's forbidden
5. release DB lock
### Optimistic locking (implemented by this MR)
#### For updating
1. latest_build_status
2. enqueue: (use `transition :created -> :pending`)
3. [state_machine] we are in state created, we can go to pending
4. [state_machine] ci_pipeline.status = created
5. [state_machine] ci_pipeline.save
6. [state_machine] [save] where(lock_version: ci_pipeline.lock_version).update_all(status: :created, updated_at: Time.now)
7. [state_machine] [save] unless we_updated_row then raise ObjectInconsistentError
#### If no update is required
1. we update ci_pipeline
2. latest_build_status
3. we are in pending, we can't transition to pending, because it's forbidden
## Why was this MR needed?
We have been seeing a number of problems when we migrated Pipeline/Build processing to Sidekiq. Especially we started seeing a lot of blocking queries.
We used a pessimistic locking which doesn't seem to be required. This effectively allows us to fix our issues with blocked queries by using more efficient method of operation.
## What are the relevant issue numbers?
Issues: https://gitlab.com/gitlab-com/infrastructure/issues/623 and https://gitlab.com/gitlab-com/infrastructure/issues/584, but also there's a bunch of Merge Requests that try to improve behavior of scheduled jobs.
cc @pcarranza @yorickpeterse @stanhu
See merge request !7040
This changes ProjectCacheWorker.perform_async so it only schedules a job
when no lease for the given project is present. This ensures we don't
end up scheduling hundreds of jobs when they won't be executed anyway.
Fixed all related specs and also changed the logic to handle edge cases. This includes exporting and exporting of group labels, which will get associated with the new group (if any) or they will become normal project labels otherwise.
Found other issues to do with not being able to import all labels at once in the beginning of the JSON - code was much simpler when we import all labels and milestones associated to a project first, then the associations will find the already created labels instead of creating them from the associations themselves.
Delete dynamic environments
- Adds "close environment" action to a merge request
- Adds tabs to environments list
- Adds close button to each environment row in environments list
- Replaces Destroy button with Close button inside an environment
- Adds close button to builds list inside an environment
#### Configuration
In order to enable stopping environments a valid `.gitlab-ci.yml` syntax has to be used:
```
review:
environment:
name: review/$app
on_stop: stop_review
stop_review:
script: echo Delete My App
when: manual
environment:
name: review/$app
action: stop
```
This MR requires that `stop_review` has to have: `when`, `environment:name` and `environment:action` defined.
The next MR after this one will verify that and enforce that these settings are configured.
It will also implicitly configure these settings, making it possible to define it like this:
```
review:
environment:
name: review/$app
on_stop: stop_review
stop_review:
script: echo Delete My App
```
Closes#22191
See merge request !6669
* upstream/master: (58 commits)
Update endpoint to username validator
change border color to variable
Add todo for deprecated user routes and more information about deprecation to changelog
Provide better error message to the user
Apply better hierarchy to markdown headers and issue/mr titles
Swapped button text manipulation outcomes for the toggle query
Fixed find file keyboard navigation
Update CHANGELOG for 8.12.7
Added download-button class and applied button margin
Enable activerecord_sane_schema_dumper for test
Updated logo from @luke
Fix broken specs on MySQL after https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6896
Fix Test Env (proper error handling when gitlab-shell is not clonned)
Fix randomly crashing spinach test for merge request
[Great spinach fix] Replace gsub with delete
Remove carriage returns from commit description as summary is on a newline and will always include carriage returns
Convert due_date_select.js filetype to es6.
Stop directly parsing due_date with Date.parse, prefer parsing implicitly.
Improve spec for pipeline metrics worker
Add Pipeline metrics worker
...
* upstream/master: (237 commits)
Grapify boards API
Add test, fix merge error
Use local assigns to get the dropdown title
Updated issuable dropdown titles
Added safety check for formatted values
Minor style improvement
Fixed conflict and corrected teaspoon test
Rename method in test
Moved ci_status environments logic to new action ci_envrionments_status and set up frontend polling
Refactor ci_status on MergeRequestController
Fix indenting error in HAML
Show what time ago a MR was deployed
Fixed missing links
Fixed missing links
Refactor merge requests revisions
Add link to update docs for source installations
Grapify todos API
Link to review apps example from docs
fix grafana_configuration.md move link
Do not run before_script, artifacts, cache in trigger_docs job
...
* upstream/master: (292 commits)
Deletes extra empty line breaking the build
Optimize the `award_user_list` helper spec
Fix typo and add he MWBS accronym for "Merge When Build Succeeds"
Added missing content and improved layout
ExpireBuildArtifactsWorker query builds table without ordering enqueuing one job per build to cleanup
Improve the contribution and MR review guide
Updates test in order to look for link
Make projects API docs match parameter style
Fix Event#reset_project_activity updates
Update user whitelist reject message
Call ensure_secret_token! in secret token test's before block since it would be called in an initializer.
Add a CHANGELOG for CacheMarkdownField
Enable CacheMarkdownField for the remaining models
Make search results use the markdown cache columns, treating them consistently
Use CacheMarkdownField for notes
Add markdown cache columns to the database, but don't use them yet
Update issue board spec
Link to Registry docs from project settings
Truncate long labels with ellipsis in labels page
Improve issue load time performance by avoiding ORDER BY in find_by call
...
Memoize Github::Shell's secret token
## What does this MR do?
`API::Helpers#secret_token` was reading the secret file on every invocation. This MR reads the file in the `gitlab_shell_secret_token.rb` initializer and saves it as a class variable at `Gitlab::Shell.secret_token`
## Are there points in the code the reviewer needs to double check?
- I'm not sure if the use of `cattr_accessor` is the best approach, or if should be moved into the `class << self` block?
- Should `API::Helpers#secret_token` be removed in favor of using `Gitlab::Shell.secret_token`?
## Why was this MR needed?
Performance optimization.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/22510
See merge request !6599
This commit adds a number of _html columns and, with the exception of Note,
starts updating them whenever the content of their partner fields changes.
Note has a collision with the note_html attr_accessor; that will be fixed later
A background worker for clearing these cache columns is also introduced - use
`rake cache:clear` to set it off. You can clear the database or Redis caches
separately by running `rake cache:clear:db` or `rake cache:clear:redis`,
respectively.
Improve asynchronous pipeline processing
## What does this MR do?
This MR improves asynchronous processing of pipeline.
## Why was this MR needed?
It eliminates some race conditions and improves performance.
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
- [x] Added for this feature/bug
- [x] All builds are passing
## What are the relevant issue / merge request numbers?
Related merge request: !6410
Extracted from !6411
See merge request !6650
Refactor Gitlab::Identifier
## What does this MR do?
This refactors `Gitlab::Identifier` so that it:
1. Has tests
2. Caches output in an instance variable to reduce queries
3. Uses only a single query to find a user by an SSH key, instead of 2
## Why was this MR needed?
This code was untested and would execute more SQL queries than needed.
See merge request !6680
This refactors Gitlab::Identifier so it uses fewer queries and is
actually tested. Queries are reduced by caching the output as well as
using 1 query (instead of 2) to find a user using an SSH key.
* upstream/master: (372 commits)
Enable Lint/StringConversionInInterpolation cop and autocorrect offenses
resolve duplicated changelog entry
credit myself 😄
change determine conditions
override subject method in devise mailer
follow the styleguide: Don't use parentheses around a literal
wrap subject with method subject
move spec back into shared example `an email sent from GitLab`
stub config settings in spec
remove empty line at block body end
remove extra entry
create new test in `spec/mailers/notify_spec.rb`
move changelog to 8.13
add configurable email subject suffix
Fixes sidebar navigation.
Convert "SSH Keys" Spinach features to RSpec
Enable import/export back for non-admins
Update gitlab-shell to 3.6.3
Updated artwork of empty group state.
Better empty state for Groups view.
...
Resolve "`Member.add_user`doesn't detect existing members that have requested access"
## What does this MR do?
This merge request handle the case when an access requester is added to a group or project (via the members page or the API).
In `Member.add_user`, if an access requester already exists, we simply accept their request (and set the `created_by`, `access_level` and `expires_at` attributes if given).
## Are there points in the code the reviewer needs to double check?
I've taken the opportunity to cleanup the whole `{Group,Project}Member.add_user*` methods since it was quite a mess.
## What are the relevant issue numbers?
Closes#21983
See merge request !6393
Prevent claiming associated model IDs via import
On the import side, we should be careful not to use any IDs as part of the JSON file that could have been manipulated.
Part of https://gitlab.com/gitlab-org/gitlab-ce/issues/20821
Things we already do (__before__ this fix):
1. Remove all primary keys
1. **Always** reassign some of the foreign keys, such as ALL project IDs and user IDs (so it would be difficult to impersonate or try to gain access to another project)
1. Ignore/reject attributes that do not exist in the model
1. If someone reassigns a foreign key `submodel_id`, and that object has another json as the submodel, the new submodel will reassign the `submodel_id` to the newly created submodel ID.
Things we should do:
1. Remove/nullify any other foreign keys that we don't reassign (checked this, and there aren't many, fortunately. In fact, I don't think much harm can be done at all - at the moment).
See merge request !1985
Do not regenerate the `lfs_token` every time `git-lfs-authenticate` is called
## What does this MR do?
Do not regenerate the `lfs_token` every time `git-lfs-authenticate` is called, instead return the saved token if one is present.
This was causing a lot of 401s, leading to 403s, as state in #22527
As it turns out, when pushing a lot of LFS objects, the LFS client was calling `git-lfs-authenticate` in the middle of the request again. This caused the `lfs_token` to be regenerated. The problem lies in that the LFS client was not aware of this change, and was still using the old token. This caused all subsequent requests to fail with a 401 error.
Since HTTP Auth is protected by Rack Attack, this 401s where immediately flagged and resulted in the IP of the user being banned.
With this change, GitLab returns the value stored in Redis, if one is present, thus if the LFS client calls `git-lfs-authenticate` again during the request, the auth header will remain unchanged, allowing all subsequent requests to continue without issues.
## What are the relevant issue numbers?
Fixes#22527
cc @SeanPackham @jacobvosmaer-gitlab
See merge request !6551
Changes include:
- Ensure Member.add_user is not called directly when not necessary
- New GroupMember.add_users_to_group to have the same abstraction level as for Project
- Refactor Member.add_user to take a source instead of an array of members
- Fix Rubocop offenses
- Always use Project#add_user instead of project.team.add_user
- Factorize users addition as members in Member.add_users_to_source
- Make access_level a keyword argument in GroupMember.add_users_to_group and ProjectMember.add_users_to_projects
- Destroy any requester before adding them as a member
- Improve the way we handle access requesters in Member.add_user
Instead of removing the requester and creating a new member,
we now simply accepts their access request. This way, they will
receive a "access request granted" email.
- Fix error that was previously silently ignored
- Stop raising when access level is invalid in Member, let Rails validation do their work
Signed-off-by: Rémy Coutable <remy@rymai.me>
* No need to re-fetch issues from GH to read their labels, the labels
are already there from the index request.
* No need to look up labels on the database for every application, so we
cache them.
Use base SHA for patches and diffs
## What does this MR do?
Switch from using 'start SHA' to 'base SHA' for patches and diffs
## Are there points in the code the reviewer needs to double check?
## Why was this MR needed?
Makes the downloaded patches and diffs on the merge request page match the frontend-rendered "changes" in these scenarios:
* Unpatched gitlab-workhorse, downloading patchsets of open MRs (https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/68)
* Unpatched gitlab-workhorse, downloading diffs of open and merged MRs
* Patched gitlab-workhorse, downloading patchsets of merged merge requests
## What are the relevant issue numbers?
Closes#22229
See merge request !6435
This commit changes the revisions used for diffs. The current behaviour is
to show all changes between current tip of master and tip of the MR, rather
than matching the output of the web frontend (which just shows the changes
in the MR). Switching from start_sha to base_sha fixes this.
Z.J. van de Weg
Valery Sizov
Lin Jen-Shin
Sean McGivern
Ahmad Sherif
Douwe Maan
James Lopez
Kamil Trzciński
Grzegorz Bizon
Stan Hu
Rémy Coutable
Tomasz Maczukin
Yorick Peterse
Felipe Artur
Andrew Smith
http://jneen.net/
Drew Blessing
Patricio Cano
Robert Speicher
tiagonbotelho
Douwe Maan
Elan Ruusamäe
Jacob Vosmaer
Gabriel Mazetto
Douglas Barbosa Alexandre
Sean McGivern
Phil Hughes
Justin DiPierro
Nick Thomas
Paco Guzman
Robert Speicher
Markus Koller