Parse GFM references after sanitizing
Parse GFM references - labels, issues, MRs, etc. - after calling the HTML Pipeline `SanitizationFilter` so that we can use non-whitelisted attributes like `style`. See #2188.
See merge request !1745
Revert "Preload lib/"
This reverts commit 5511a731bc.
The original commit added this because it also enabled threadsafety, a change which was itself later reverted in 66d6c80966, but this got left behind.
I don't fully understand the reasoning behind it so if I'm wrong, please tell me.
My reasoning for reverting it is that it messes with Rails' (and by extension Spring's) class reloading during development. When I was working in `lib/gitlab/markdown` and had to stop and restart the server every time I made a change, I didn't know it at the time, but this was why. That was a huge pain point.
If it's needed for production perhaps we could add a `if Rails.env.production?` clause around it so that it doesn't mess with development.
See merge request !1758
Support configurable attachment size in Application Settings page
### What does this MR do?
This MR provides the ability to configure the maximum size of an attachment inside a note. A parameter has been added to the Application Settings page.
### Are there points in the code the reviewer needs to double check?
What should be done with the legacy note attachment validation? I added code to make the validation work with the configurable setting. I could see an issue where an admin lowers the limit from 10 megabytes to 5 megabytes, which could cause an existing model to be invalid.
### Why was this MR needed?
We often have attachments that exceed 10 MB, and it would be nice to be able to override the defaults.
### What are the relevant issue numbers / [Feature requests](http://feedback.gitlab.com/)?
See Issue #1258
### Screenshots
Before:
After:
See merge request !407
Import GitHub, Bitbucket or GitLab.com projects owned by authenticated user into current namespace.
Addresses #1347.
Untested since I'm in a bit of a hurry. Will definitely have time to test and add unit tests before the 7.10 release :)
See merge request !481
Don't allow username to end in period.
The current behavior doesn't do username referencing and mentioning in sentences like "I discussed with with @douwe." since `douwe.` is matched as a username.
Addresses private issue https://dev.gitlab.org/gitlab/gitlabhq/issues/2174.
See merge request !438
Parse GFM references - labels, issues, MRs, etc. - after calling the
HTML Pipeline `SanitizationFilter` so that we can use non-whitelisted
attributes like `style`.
Reduce Rack Attack false positives causing 403 errors during HTTP authentication
### What does this MR do?
This MR reduces false positives causing `403 Forbidden` messages after HTTP authentication.
A Git client may attempt to access a repository without a password. If it receives a 401 error, the client often will try again, this time supplying a password. The problem is that `grack_auth.rb` considers a blank password an authentication failure and increases a Redis counter each time this happens. With enough requests, an IP can be banned temporarily even though previous attempts may have been successful. This leads users to see `403 Forbidden` errors until the ban times out (default: 1 hour).
To reduce the chance of a false positive, this MR resets the counter upon a successful authentication from an IP.
In addition, this MR logs when a user has been banned and introduces the ability to disable Rack Attack via a config variable.
### Are there points in the code the reviewer needs to double check?
rack-attack v4.2.0 doesn't support the ability to clear counters out of the box, so `rack_attack_helpers.rb` includes a number of monkey patches to make it work. It looks like this functionality may be added in v4.3.0. I've also sent pull requests to rack-attack to add the functionality necessary to delete a key.
Each time an authentication is successful, the Redis counter for that IP is cleared. I deemed it better to clear the counter than to allow for blank passwords, since the latter seems like a security risk.
### Why was this MR needed?
It was quite difficult to figure out why users were seeing `403 Forbidden`, which is why the log message was added. Users were getting a lot of false positives when accessing repositories with HTTPS. Including the username in the HTTPS URL (e.g. `https://username@mydomain.com/account/repo.git`) caused authentication failures because while the git client provided the username, it left the password blank, leading to an authentication failure.
### What are the relevant issue numbers / [Feature requests](http://feedback.gitlab.com/)?
See Issue #1171https://github.com/kickstarter/rack-attack/issues/113
See merge request !392
Fix nested task lists
When nesting task list items, the parent item is wrapped in a `<p>` tag. Update the task list parser to handle these paragraph wrappers.
cc @sytse
See merge request !413
Replace commits calendar with contributions calendar
* count opening of issues and merge requests
* dont trigger git repository - use events from database
* count pushes instead of commits for faster and easier counting
* much-much faster since does not affected by repository size
See merge request !420
Return a `SafeBuffer` instead of a `String` from the `#gfm_with_options`
method so that Rails doesn't escape our markup.
Also add `<span>` to the sanitization whitelist to avoid breaking syntax
highlighting in code blocks.
Disable reference generation in preformatted/code blocks
### Summary
If a user adds text in code or preformatted text via Markdown or HTML that contains `#XXX`, the system adds a note that issue `XXX` was mentioned. This is particularly annoying because we often list gdb backtrace dumps into our issues, and many issues get mentioned as a result. For example:
```
(gdb) bt
#0 0x00000000004004c4 in second () at main.cc:6
#1 0x00000000004004d2 in first () at main.cc:11
#2 0x00000000004004dd in main () at main.cc:17
(gdb)
```
### Steps to reproduce
1. In an issue, write the above text using Markdown or HTML tags (e.g. `<code>`, `<pre>`).
2. Observe that [issue 1](https://gitlab.com/gitlab-org/gitlab-ce/issues/1) and [issue 2](https://gitlab.com/gitlab-org/gitlab-ce/issues/2) have a note that says they were mentioned.
### Expected behavior
Everything enclosed in the code blocks should be ignored as references.
### Observed behavior
Issues get referenced unnecessarily.
### Fix
I've made `reference_extractor.rb` strip out HTML and Markdown blocks before processing. I considered running the raw text through the entire Markdown processor, but this seems overkill and perhaps could lead to some unintended side effects.
See merge request !365
* count opening of issues and merge requests
* dont trigger git repository - use events from database
* much-much faster since does not affected by repository size
The Net::LDAP::Filter.escape function can not be used to escape the DN name because the backslash is required to escape special chars in the DN name. This leads to the error message "Access denied for your LDAP account." and prevents the user from logging in to gitlab.
Example DN:
CN=Test\, User,OU=Organization,DC=Company
CN=Test User,OU=Organization,DC=Company
http://www.ietf.org/rfc/rfc4514.txt
Fix invalid Atom feeds when using emoji, horizontal rules, or images
This is a fix for issues #880, #723, #1113.
Markdown must be rendered to XHTML, not HTML, when generating summary content for Atom feeds. Otherwise, content-less tags like *img* and *hr* are not terminated and make the Atom XML invalid. Such tags are generated when issue descriptions, merge request descriptions, comments, or commit messages use emoji, horizontal rules, or images.
To pass this option through from the relevant Haml templates to the proper place in the `gfm()` method, a new method `gfm_with_options()` is introduced. It reuses the options dictionary passed to `markdown()` and interprets options `xhtml` and `parse_tasks` from it (the latter was a convenient replacement for `gfm_with_tasks()`). `xhtml` is already interpreted by Redcarpet::Render::HTML, but that alone was not sufficient, because the post-processing in `gfm()` would convert its XHTML tags back to HTML.
I found no way of passing additional optional options to the existing `gfm()` method without requiring updates to existing callers and without getting in the way of the existing optional arguments, but maybe someone who knows more about Ruby than I can think of one.
Thorough review appreciated since this is the first time I have used Ruby.
See merge request !344
Fixes issues #880, #723, #1113: Markdown must be rendered to XHTML, not HTML, when generating summary content for Atom feeds. Otherwise, content-less tags like <img> and <hr>, generated when issue descriptions, merge request descriptions, comments, or commit messages use emoji, horizontal rules, or images, are not terminated and make the Atom XML invalid.
Restricted visibility levels - bug fix and new feature
This allows admin users to override restricted visibility settings when creating and updating projects and snippets, and moves the restricted visibility configuration from gitlab.yml to the web UI. See #1903.
## Move configuration location
I added a new section to the application settings page for restricted visibility levels. Each level has a checkbox, styled with Bootstrap to look like a toggle button. A checked box means that the level is restricted. I added a glowing text shadow and changed the background color for checked buttons because the default styles made it hard to distinguish between checked and unchecked. This image shows the new section with the "Public" box checked:
## Allow admins to override
To allow admin users to override the restricted visibility levels, I had to remove the `visibility_level` validation from the `Project` class. The model doesn't know about the `current_user`, which should determine whether the restrictions can be overridden. We could use the creator in the validation, but that wouldn't work correctly for projects where a non-admin user is the creator and an admin tries to change the project to a restricted visibility level.
The `Project::UpdateService` and `Project::CreateService` classes already had code to determine whether the current user is allowed to use a given visibility level; now all visibility level validation is done in those classes. Currently, when a non-admin tries to create or update a project using a restricted level, these classes silently set the visibility level to the global default (create) or the project's existing value (update). I changed this behavior to be more like an Active Model validation, where using a restricted level causes the entire request to be rejected.
Project and personal snippets didn't have service classes, and restricted visibility levels weren't being enforced in the model or the controllers. The UI disabled radio buttons for restricted levels, but that wouldn't be difficult to circumvent. I created the `CreateSnippetService` and `UpdateSnippetService` classes to do the same restricted visibility check that the project classes do. And since I was dealing with snippet visibility levels, I updated the API endpoints for project snippets to allow users to set and update the visibility level.
## TODO
* [x] Add more tests for restricted visibility functionality
cc @sytse @dzaporozhets
See merge request !1655
Execute hooks and services when branch or tag is created or deleted through web interface.
Fixes#2095.
Split up into commits to make it easier to see why what was changed :)
See merge request !1692
Generate valid json
This patch helps to be compatible to other programing languages as it improves the validation of hook data. It seems only ruby can handle 'nil' as value while other json decode function will fatal.
See merge request !182
Add new service classes to create and update project and personal
snippets. These classes are responsible for enforcing restricted
visibility settings for non-admin users.
Supports four different event types all bundled under the "note" event type:
- comments on a commit
- comments on an issue
- comments on a merge request
- comments on a code snippet
Increase timeout for Git-over-HTTP requests.
Fixes#2081 and https://gitlab.com/gitlab-org/gitlab-ce/issues/232.
Normal web requests are bound by the `Rack::Timeout` timeout of 60 seconds, while Grack Git-over-HTTP requests are only bound by Unicorn's timeout which is now set to 1 hour, which should be plenty.
The omnibus package should be updated to no longer use `unicorn['worker_timeout']` for the Unicorn timeout, but to set the `Slowpoke.timeout`.
See merge request !1619
Fix namespace in merge request url building
Changes in 42387b733b now require namespace specification and broke abc69c8905.
There are additional helper functions in c530ca00b0, but this seemed easier not to rely on them.
See merge request !363
Don't leak information about private project existence via Git-over-SSH/HTTP.
Fixes#2040 and https://gitlab.com/gitlab-org/gitlab-ce/issues/343.
Both `Grack::Auth` (used by Git-over-HTTP) and `Api::Internal /allowed` (used by gitlab-shell/Git-over-SSH) now return a generic "Not Found" error when the project exists but the user doesn't have access to it.
See merge request !1578
Fix merge request URL passed to Webhooks
If you look at the data structure passed to Webhooks, you will see:
`"url"=>nil`
I don't think any of the Webhooks or services are using this yet, so right now nothing so far depends upon this value being correct.
See merge request !352
1) Adds a DB migration for all services to toggle on push, issue, and merge events.
2) Upon an issue or merge request event, fire service hooks.
3) Slack service supports custom messages for each of these events. Other services
not supported at the moment.
4) Label merge request hooks with their corresponding actions.
Improve error messages when file editing fails
Give more specific errors in API responses and web UI flash messages when a file update fails. See #1479.
Instead of returning false from `Gitlab::Satellite::Files::EditFileAction#commit!` when a `Grit::Git::CommandFailed` error is raised, now `#commit!` raises a different error depending on whether the failure happened during checkout, commit, or push.
@dzaporozhets Please let me know if you want to change the HTTP status codes or the error messages in `Files::UpdateService`
cc @sytse
See merge request !1569
Fix note attachments XSS and access control
Replaces the reverted #1528, as proposed in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/434, as discussed with @dzaporozhets and as summarized in #2032.
@marin Could you take a look at the nginx config and apply it to Omnibus once this gets merged?
See merge request !1553
Robert Speicher
Douwe Maan
Dmitriy Zaporozhets
Vinnie Okada
Dmitriy Zaporozhets
Jacob Vosmaer
Stan Hu
Marin Jankovski
nicklegr
Douwe Maan
Michael Alt
Kamil Trzcinski
Vinnie Okada
Hannes Rosenögger
Christian Walther
Nicole Cordes
Valery Sizov
Marin Jankovski
Jeroen van Baarsen
Nicolas Bouilleaud
