Commit Graph

2134 Commits

Author SHA1 Message Date
Yorick Peterse be3b878443 Track method call times/counts as a single metric
Previously we'd create a separate Metric instance for every method call
that would exceed the method call threshold. This is problematic because
it doesn't provide us with information to accurately get the _total_
execution time of a particular method. For example, if the method
"Foo#bar" was called 4 times with a runtime of ~10 milliseconds we'd end
up with 4 different Metric instances. If we were to then get the
average/95th percentile/etc of the timings this would be roughly 10
milliseconds. However, the _actual_ total time spent in this method
would be around 40 milliseconds.

To solve this problem we now create a single Metric instance per method.
This Metric instance contains the _total_ real/CPU time and the call
count for every instrumented method.
2016-06-17 13:09:55 -04:00
Yorick Peterse ed5f17cc73 Merge branch 'secure-request-uris' into 'master'
Filter out sensitive parameters of metrics data

See merge request !4748
2016-06-17 16:43:14 +00:00
Robert Speicher 8dccfb4a9c Merge branch 'update-column-in-batches-where' into 'master'
Allow customising of queries used for `update_column_in_batches`

This MR makes two changes to `add_column_with_default` and `update_column_in_batches`:

1. `add_column_with_default` no longer wraps the entire set of updates in a single transaction, preventing any locks from sticking around for the duration of the entire transaction
2. `update_column_in_batches` now takes a block which can be used to customise the queries. This uses Arel as messing with raw SQL strings is a total pain

In !4381 there's a need for updating existing rows/columns in a table in batches using a custom `WHERE` condition. Without the changes in this MR this would not be possible.

See merge request !4680
2016-06-17 16:36:22 +00:00
Paco Guzman 2e552c6bf0 Filter out sensitive parameters of metrics data 2016-06-17 18:14:25 +02:00
Yorick Peterse c1e756c242
Fix update_column_in_batches to update all rows
This changes update_column_in_batches to ensure it always updates all
rows now. These changes also allow for an extra SELECT query to be
removed, nor does it use the row count for determining offsets and the
likes; instead it's only used to determine the batch size.
2016-06-17 17:50:38 +02:00
Douwe Maan 9b7fd74805 Merge branch 'feature/project-export' into 'master'
Export project functionality

This is a MR for the export functionality of https://gitlab.com/gitlab-org/gitlab-ce/issues/3050, which adds the ability to export single projects.

- [x] members
- DB data
  - [x] issues
  - [x] issue comments
  - [x] merge requests
  - [x] merge request diff
  - [x] merge request comments
  - [x] labels
  - [x] milestones
  - [x] snippets
  - [x] releases
  - [x] events
  - [x] commit statuses
  - [x] CI builds
- File system data
  - [x] Git repository
  - [x] wiki
  - [x] uploads
  - [ ] ~~CI build traces~~
  - [ ] ~~CI build artifacts~~
  - [ ] ~~LFS objects~~
- DB configuration
  - [x] services
  - [x] web hooks
  - [x] protected branches
  - [x] deploy keys
  - [x] CI variables
  - [x] CI triggers

See merge request !3114
2016-06-17 15:35:27 +00:00
James Lopez 2d4556c5d2 a few changes based on MR feedback 2016-06-17 15:47:00 +02:00
Kamil Trzcinski aef6214c42 Validate only and except regexp
Currently the RegexpError can be raised when processing next stage which leads to 500 in different places of code base.
This adds early check that regexps used in only and except are valid.
2016-06-17 11:52:22 +02:00
James Lopez b3a0a1c315 fixed merge conflicts on UI branch 2016-06-16 15:01:27 +02:00
James Lopez 2a747d386d fixed merge conflicts 2016-06-16 14:07:49 +02:00
James Lopez 778d72664f Merge branch 'master' of gitlab.com:gitlab-org/gitlab-ce into feature/project-export
# Conflicts:
#	app/models/ci/pipeline.rb
2016-06-16 13:04:00 +02:00
James Lopez 452c076a34 Revert "squashed merge and fixed conflicts"
This reverts commit 13e37a3ee5.
2016-06-16 12:59:07 +02:00
James Lopez 13e37a3ee5 squashed merge and fixed conflicts 2016-06-16 12:12:28 +02:00
James Lopez 1b8a107325 fix wiki stuff 2016-06-16 10:59:36 +02:00
James Lopez 5087e10fe9 fix specs 2016-06-16 09:01:09 +02:00
Douwe Maan af493ccf36 Merge branch 'remove-shell-env' into 'master'
Get rid of Gitlab::ShellEnv

Remove obsolete code we once needed for Grack and satellites.

See merge request !4673
2016-06-15 16:41:13 +00:00
James Lopez 4bde59341f lots of refactoring again based on feedback. Changed the UI slightly and also fixed a small bug 2016-06-15 17:31:00 +02:00
Yorick Peterse 816c453558
Don't update columns in batches in a transaction
This ensures that whatever locks are acquired aren't held onto until the
end of the transaction (= after _all_ rows have been updated). Timing
wise there's also no difference between using a transaction and not
using one.
2016-06-15 17:04:07 +02:00
Yorick Peterse 8966263e0c
Customizing of update_column_in_batches queries
By passing a block to update_column_in_batches() one can now customize
the queries executed. This in turn can be used to only update a specific
set of rows instead of simply all the rows in the table.
2016-06-15 16:44:44 +02:00
Rémy Coutable be09845914 Merge branch 'environments-and-deployments' into 'master'
Add environments and deployments

This MR is a continuation of https://gitlab.com/gitlab-org/gitlab-ce/issues/17009.

The current implementation is as follow:
1. We have two new tables: `environments` and `deployments`.
2. We have a new tab: `Environments` under `Pipelines` where you can see all you environments and add a new one.
3. We add a new option to `.gitlab-ci.yml` to track where we should create a deployment for environment.
4. If environment in `.gitlab-ci.yml` is specified it will create a deployment. **If environment does not exist it will be created.** (this got changed)
5. The deployment is always successful and shows the time of the action, in that case a build that presumably should do deployment. In the future we could extend deployment with statuses: success, failure. We could extend deployments with information that this is partial or full deployment.
6. User have to create environments that he will track first.
7. User can remove environments.
8. User can retry/rollback past deployment (in that case we retry past build). The new build when succeeds it will create a new deployment.
9. Currently environment have only one parameter: `name`. In the future it should have: `variables`, `credentials` and possibly `runners` and maybe other resources.
10. Currently deployment have this parameters: `sha`, `ref`, `deployable (in this case a build)`, `user (who triggered a deployment)`, `created_at`.

The `.gitlab-ci.yml`:
```
deploy to production:
  stage: deploy
  script: dpl travis...
  environment: production
```

What needs to be done:
- [x] Write initial implementation
- [x] Improve implementation (@ayufan)
- [x] Write tests (@ayufan)
- [x] Improve UX of the forms (cc @markpundsack) - reviewed by @markpundsack
- [x] Improve implementation of the views (cc @jschatz1) - done by @iamphill 
- [x] Write .gitlab-ci.yml documentation for `environments` - done by @ayufan
- [ ] Write user documentation (@ayufan and @markpundsack)

See merge request !4605
2016-06-15 13:48:09 +00:00
Jacob Vosmaer c32e61251e Get rid of Gitlab::ShellEnv 2016-06-15 14:59:37 +02:00
Z.J. van de Weg b2df118561 Random selection now also works for MySQL 2016-06-15 12:10:41 +02:00
James Lopez fc5f6943f8 yay finally importing working with the new services structure 2016-06-14 21:41:40 +02:00
James Lopez ff44198e17 few fixes after refactoring the whole UI stuff 2016-06-14 21:03:14 +02:00
James Lopez b53ed84843 adapted current services stuff to use new project import, plus fixes a few issues, updated routes, etc... 2016-06-14 20:32:19 +02:00
Yorick Peterse ab91f1226f
Filter out classes without names in the sampler
We can't do a lot with classes without names as we can't filter by them,
have no idea where they come from, etc. As such it's best to just ignore
these.
2016-06-14 18:09:06 +02:00
James Lopez 1d4243f5dd Merge branches 'feature/project-export-ui-experimental' and 'feature/project-import' of gitlab.com:gitlab-org/gitlab-ce into feature/project-export-ui-experimental 2016-06-14 16:40:13 +02:00
James Lopez 862b359b9a fix merge issue 2016-06-14 16:37:41 +02:00
James Lopez 9ecebaaea1 adding notifications stuff and more refactoring for exporting projects 2016-06-14 16:31:03 +02:00
Yorick Peterse 0ca7b3ba37 Merge branch '18449-instrument-grape-endpoints' into 'master'
Instrument Grape API endpoints

See merge request !4587
2016-06-14 14:29:55 +00:00
Yorick Peterse f558bf0de4 Merge branch '18527-instrument-private-methods' into 'master'
Instrument private methods and instance private methods

See merge request !4639
2016-06-14 14:28:40 +00:00
Kamil Trzcinski 006b650988 Merge remote-tracking branch 'origin/master' into environments-and-deployments
# Conflicts:
#	db/schema.rb
2016-06-14 16:19:29 +02:00
Paco Guzman dadc531353 Instrument private/protected methods
By default instrumentation will instrument public,
protected and private methods, because usually
heavy work is done on private method or at least
that’s what facts is showing
2016-06-14 15:17:51 +02:00
Kamil Trzcinski 975e388751 Merge remote-tracking branch 'origin/master' into environments-and-deployments
# Conflicts:
#	lib/ci/gitlab_ci_yaml_processor.rb
2016-06-14 14:53:10 +02:00
Kamil Trzcinski 3f5819fb7a Merge remote-tracking branch 'origin/master' into artifacts-expire-date 2016-06-14 14:49:59 +02:00
James Lopez fe370b1c39 new export stuff and view 2016-06-14 14:28:30 +02:00
Yorick Peterse 4b964011cf Merge branch '18528-cpu-time-instrumentation' into 'master'
Measure CPU time for instrumented methods

See merge request !4640
2016-06-14 11:39:19 +00:00
Paco Guzman 509082bafb Instrument Grape Endpoint with Metrics::RackMiddleware
Generating the following tags

Grape#GET /projects/:id/archive

from Grape::Route objects like

{ :path => /:version/projects/:id/archive(.:format)
  :version => “v3”,
  :method => “GET” }

Use an instance variable to cache raw_path transformations.
This variable is only going to growth to the number of 
endpoints of the API, not with exact different requests

We can store this cache as an instance variable because 
middleware are initialised only once
2016-06-14 13:06:46 +02:00
Kamil Trzcinski e8f09f02bf Validate environment name with regex 2016-06-14 13:04:21 +02:00
Rémy Coutable 47cdb69921 Merge branch 'refactor/ci-config-add-global-entry' into 'master'
Add global entry with before script to new CI config

## What does this MR do?

This MR adds a new entries to a new CI config class. It is next refactoring step after !4462.

See #15060

See merge request !4482
2016-06-14 10:53:26 +00:00
Paco Guzman 120fbbd487 Measure CPU time for instrumented methods 2016-06-14 12:49:31 +02:00
James Lopez 3f7ed55011 lots of refactoring to reuse import service 2016-06-14 12:47:07 +02:00
Grzegorz Bizon 30e946ce8a Validate ci config entry value before processing nodes 2016-06-14 11:28:20 +02:00
James Lopez 1ea44ee750 Merge branches 'feature/project-export-ui-experimental' and 'feature/project-import' of gitlab.com:gitlab-org/gitlab-ce into feature/project-export-ui-experimental 2016-06-14 11:06:22 +02:00
James Lopez 279412f90a updated relation_factory based on MR feedback 2016-06-14 10:55:24 +02:00
Grzegorz Bizon 59eeec3ff8 Make method that composes ci config entry private 2016-06-14 10:48:51 +02:00
James Lopez 7779457907 fix merge 2016-06-14 10:20:47 +02:00
James Lopez 9ff1990422 Merge branches 'feature/project-export' and 'feature/project-import' of gitlab.com:gitlab-org/gitlab-ce into feature/project-import 2016-06-14 10:18:03 +02:00
James Lopez f6ed7c8ff8 missed line break 2016-06-14 10:15:20 +02:00
James Lopez 4020b0f55f few changes based on MR feedback 2016-06-13 21:18:26 +02:00