This solves a problem where commits populated with Gitaly were not equal
to commits populated with Rugged. This is because Gitaly may not return
all fields of a commit for optimizations purposes, which resulted in
false negatives when comparing the same commit (commits with the same
sha) with different sources.
It seems that bad things happen when two gRPC stubs share one gRPC
channel so let's stop doing that. The downside of this is that we
create more gRPC connections; one per stub.
- Previously, we sorted commits by date, which seemed to work okay.
- The one edge case where this failed was when multiple commits have the same
commit date (for example: when a range of commits are cherry picked with a
single command, they all have the same commit date [and different author
dates]).
- Commits with the same commit date would be sorted arbitrarily, and usually
break the network graph.
- This commit solves the problem by both sorting by date, and by sorting
topographically (parents aren't displayed until all their children are
displayed)
- Include review comments from @adamniedzielski
A more detailed explanation is present here:
https://gitlab.com/gitlab-org/gitlab-ce/issues/30973#note_28706230
- We upgraded `rugged` to 0.25.1.1 in !10286 for %9.1
- Prior to this upgrade, the default sort order for commits returned by
`Gitlab::Git::Repository#find_commits` was `Rugged::SORT_DATE`, which the
graph relied on.
- While upgrading `rugged`, the MR also changed this default to
`Rugged::SORT_NONE`, which broke commit ordering in the graph.
- This commit adds an option to `Gitlab::Git::Repository#find_commits` to sort
by date, and changes the graph builder `Network::Graph` so it explictly
requests the `:date` sort order
This is analogous to `git log -- foo bar baz`, but not the same as
`Gitlab::Git::Repository#log(path: 'foo bar baz')`, which would run `git
log -- 'foo bar baz'`.
- `raise "string"` raises a `RuntimeError` - no need to be explicit
- Remove top-level comment in the `RevList` class
- Use `%w()` instead of `%w[]`
- Extract an `environment_variables` method to cache `env.slice(*ALLOWED_VARIABLES)`
- Use `start_with?` for env variable validation instead of regex match
- Validation specs for each allowed environment variable were identical. Build them dynamically.
- Minor change to `popen3` expectation.
- Don't define "allowed environment variables" in two places.
- Dispatch to different arities of `Popen.open` without an if/else block.
- Use `described_class` instead of explicitly stating the class name within a
- spec.
- Remove `git_environment_variables_validator_spec` and keep the validation inline.
The list of environment variables in `Gitlab::Git::RevList` need to be validate
to make sure that they don't reference any other project on disk.
This commit mixes in `ActiveModel::Validations` into `Gitlab::Git::RevList`, and
validates that the environment variables are on the level (using a custom
validator class). If the validations fail, the force push is still executed
without any environment variables set.
Add specs for the validation using shared examples.
1. Starting version 2.11, git changed the way the pre-receive flow works.
- Previously, the new potential objects would be added to the main repo. If the
pre-receive passes, the new objects stay in the repo but are linked up. If
the pre-receive fails, the new objects stay orphaned in the repo, and are
cleaned up during the next `git gc`.
- In 2.11, the new potential objects are added to a temporary "alternate object
directory", that git creates for this purpose. If the pre-receive passes, the
objects from the alternate object directory are migrated to the main repo. If
the pre-receive fails the alternate object directory is simply deleted.
2. In our workflow, the pre-recieve script (in `gitlab-shell) calls the
`/allowed` endpoint, which calls out directly to git to perform
various checks. These direct calls to git do _not_ have the necessary
environment variables set which allow access to the "alternate object
directory" (explained above). Therefore these calls to git are not able to
access any of the new potential objects to be added during this push.
3. We fix this by accepting the relevant environment variables
(GIT_ALTERNATE_OBJECT_DIRECTORIES, GIT_OBJECT_DIRECTORY) on the
`/allowed` endpoint, and then include these environment variables while
calling out to git.
4. This commit includes (whitelisted) these environment variables while making
the "force push" check. A `Gitlab::Git::RevList` module is extracted to
prevent `ForcePush` from being littered with these checks.
Allow specifying protected branches using wildcards
Closes#18627
# Tasks
- [ ] #18627!4665 Allow specifying protected branches using wildcards
- [x] Find existing usages of protected branches
- Protecting branches
- `ProtectedBranchesController` is used to mark a branch protected/unprotected
- `API::Branches` can be used to mark a branch protected/unprotected
- Enforcing branch protection
- `Gitlab::GitAccess` has helpers (`can_push_to_branch?`, `check`) that are used to deny pushes if a branch is protected
- Over SSH: `gitlab-shell` receives a push, and calls `/allowed` on the GitLab API, which calls `GitAccess.check`
- Over HTTP:
- `gitlab-workhorse` receives the request, and forwards it to rails
- Rails (in the `GitHttpController#git-recieve-pack`) runs basic checks (is the user logged in, not protected branch checks) and returns ok with `GL_ID` and `RepoPath`
- `gitlab-workhorse` looks at the response, and calls the relevant `gitlab-shell` action from `git-http/handlePostRPC`
- Rest of this flow is the same as the SSH flow above
- [x] Implementation
- [x] Backend
- [x] Change `project#protected_branch?` to look at wildcard protected branches
- [x] Change `project#developers_can_push_to_protected_branch?`
- [x] Change `project#open_branches`
- [x] Better error message when creating a disallowed branch from the Web UI
- [x] Frontend
- [x] Protected branches page should allow typing out a wildcard pattern
- [x] Add help text explaining the use of wildcards
- [x] Show matching branches for each protected branch
- [x] ~~On the index page~~
- [x] On a show page
- [x] Index?
- [x] Can't have the "last commit" column for wildcard protected branches
- [x] Fix / write tests
- [x] What happens if a hook is missing in dev?
- [x] Refactor
- [x] Test workflows
- Create a branch matching a wildcard pattern
- Push to a branch matching a wildcard pattern
- Force push to a branch matching a wildcard pattern
- Delete a branch matching a wildcard pattern
- [x] Test using Web UI
- [x] Test over SSH
- [x] Test over HTTP
- [x] Test as developer and master
- [x] Investigate performance
- [x] Test with a large number of protected branches / branches
- [x] Paginate list of protected branches
- [x] ~~Possibly rewrite `open_branches`~~
- [x] Add `iid`s to existing `ProtectedBranch`es
- [x] Add documentation
- [x] Add CHANGELOG entry
- [x] Add screenshots
- [x] Make sure [build](2f753e3ed2/builds) passes
- [x] Assign to endboss for review
- [x] Address @DouweM's comments
- [x] `protected_branch_params`
- [x] `exact_match` instead of `explicit_match`
- [x] When would self.name be blank?
- [x] Move `protected_branches.each` to a partial
- [x] Move `matching_branches.each` to a partial
- [x] If the branch is in @matching_branches, it's not been removed
- [x] move this regex to a method and memoize it
- [x] `commit_sha` directly for exact matches
- [x] Number of matches for wildcard matches, with a link
- [x] Wait for [build](43f9ce0e88/builds) to pass
- [x] Respond to @DouweM's comments
- [x] Don't use iid
- [x] Controller should use `@project.protected_branches.new`
- [x] move the memoization to `def wildcard_regex`
- [x] render with `collection: @protected_branches`
- [x] Wait for [build](f7beedf122/builds) to pass
- [x] Wait for @DouweM's review
- [x] Wait for @jschatz1's review
- [x] Respond to @jschatz1's comments
- [x] Use the new dropdown style
- [x] description should be moved to the description section without the styling
- [x] Protect button should be disabled when no branch is selected
- [x] Update screenshots
- [x] Merge conflicts
- [x] Make sure [build](20f3cfe8d5/builds) passes
- [ ] Revisit performance, possibly with staging/production data
- [ ] Get a dump of staging / run against staging live
- [ ] Get SSH access to staging
- [ ] Wait for review/merge
# Screenshots
## Creating wildcard protected branches
### Using the `GLDropdown` component
## Enforcing wildcard protected branches
### From the Web UI
### Over SSH
### Over HTTPS
## Listing matching branches
See merge request !4665
Note: This feature was developed independently on master while this was
in review. I've removed the conflicting bits and left the relevant
additions, mainly a test for `Gitlab::Git::Hook`. The original commit
message follows:
1. `gitlab-shell` outputs errors to `stderr`, but we weren't using this
information, prior to this commit. Now we capture the `stderr`, and
display it in the flash message when branch creation fails.
2. This can be used to display better errors for other git operation
failures with small tweaks.
3. The return value of `Gitlab::Git::Hook#trigger` is changed from a
simple `true`/`false` to a tuple of `[status, errors]`. All usages
and tests have been updated to reflect this change.
4. This is only relevant to branch creation _from the Web UI_, since SSH
and HTTP pushes access `gitlab-shell` either directly or through
`gitlab-workhorse`.
5. A few minor changes need to be made on the `gitlab-shell` end. Right
now, the `stderr` message it outputs is prefixed by "GitLab: ", which
shows up in our flash message. This is better removed.
Sean McGivern
Douwe Maan
Douwe Maan
Jacob Vosmaer
Alejandro Rodríguez
Ahmad Sherif
Rémy Coutable
Timothy Andrew
Tiago Botelho
Kim "BKC" Carlbäcker
James Lopez
Robert Speicher
mhasbini
Sean McGivern
Adam Niedzielski
Mark Fletcher
Oswaldo Ferreira
George Andrinopoulos
Z.J. van de Weg
Robert Schilling
Robert Speicher
Valery Sizov
Patricio Cano
Valery Sizov
Dmitriy Zaporozhets
Sato Hiroyuki
akzhan