Adds permission checks to the metrics_dashboard endpoint. Users
with role of Reporter or above should have access to view the
metrics for a given project.
This commits adds support for metrics dashboards
for embedding. If the flag 'embedded' is provided
to the environments/id/metrics_dashboard endpoint,
the response will be suitable for embedding in
issues or other content.
This is a precursor for support for embedding
metrics in GFM.
Opts to cache a full list of cached dashboards
to better manage removing items from the cache.
This also allows dashboards to be stored in the
cache that don't necessarily correspond to a
single dashboard yml.
Error classes associated with individual stages of
dashboard processing tend to have very long names.
As dashboard post-processing includes more steps,
we will likely need to handle more error cases.
Refactoring to have all errors inherit from a specific
base class will help accommodate this and keep the code
more readable.
Adds a new stage to dashboard processesing step for the
EnvironmentsController::metrics_dashboard endpoint.
Allows the front end to avoid generating the endpoint
unitutive string mutations.
In some cases (during worker start) it's possible that
Puma.stats returns an empty hash for worker's last status. In
that case we just skip sampling of the worker until these
stats are available.
This adds ruby and unicorn instrumentation. This was originally
intended in 11.11 but due to performance concerns it was reverted. This
new commit foregoes the sys-proctable gem was causing performance issues
previously.
Updates the EnvironmentController#metrics_dashboard endpoint
to support a "dashboard" param, which can be used to specify
the filepath of a dashboard configuration from a project
repository. Dashboard configurations are expected to be
stored in .gitlab/dashboards/.
Updates dashboard post-processing steps to exclude custom
metrics, which should only display on the system dashboard.
This adds back ruby_memory_bytes for backwards compatibility, limits
code duplication, cleans up unused methods, and limits the unicorn
worker sampling scope.
This updates monitor docs to reflect the new ruby and unicorn metrics as
well as making it so we fetch process start time via the proc table
instead of via CLOCK_BOOTTIME
This adds new metrics for monitoring unicorn. These metrics include
process_cpu_seconds_total, process_start_time_seconds, process_max_fds,
and unicorn_workers.
On reload, references to Metrics within classes in the Gitlab::Metrics
module fail. Update all references to ::Gitlab::Metrics so that
constant lookup finds the right module in development. This fix should
not impact production.
* Use a simple counter for sampler duration instead of a histogram.
* Use a counter to collect GC time.
* Remove unused objects metric.
* Cleanup metric names to match Prometheus conventions.
* Prefix generic GC stats with `gc_stat`.
* Include worker label on memory and file descriptor metrics.
In MR https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15834 we
removed use of the data produced by the Allocations Gem. However, we
never removed the code that just enables tracking of allocations. In
this commit we remove all remaining traces of this Gem.
There seem to be a lot of cases where the suffix of an action (e.g.
".html") is set to bogus data, such as "*/*" or entire URLs. This can
increase cardinality of our metrics, and isn't very useful for
monitoring and filtering. To work around this, we enforce a whitelist
containing a few suffixes we actually care about. Suffixes not supported
will be grouped under the action without a suffix. This means that a
request to "FooController#bar.jpeg" will be assigned to
"FooController#bar".
Certain controllers (e.g. Doorkeeper::TokensController) don't expose the
method "request_format". This commit changes
Gitlab::Metrics::WebTransaction so we don't rely on this method, instead
using the underlying code this method uses.
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/46412
The method "content_type" on a controller does not always return the
correct content type. On the other hand, the method "request_format"
does _and_ immediately returns a Symbol (e.g. :json) instead of a
mime-type name (e.g. application/json). With these changes metrics
should again report their action names correctly.
Fixes https://gitlab.com/gitlab-com/infrastructure/issues/3499
The Sidekiq exporter logs were mixing with the normal Sidekiq logs. In order
to support structured logging in Sidekiq, we either need to split this data
out or convert the exporter to produce structured logs. Since Sidekiq job
processing is fundamentally different information from Web server traffic,
it seems cleaner to move the metrics traffic into a separate file, where they
can be parsed by a different filter if needed.
Relates to #20060
```
metric = @_metrics_provider_cache&.[](name)
return metric if metric
``
acccessor that only slightly slower (<20ns) than direct class variable access
Reduce cardinality of some of GitLab's Prometheus metrics and fix observed duration reporting.
Closes#41045
See merge request gitlab-org/gitlab-ce!15881
* upstream/master: (671 commits)
Make rubocop happy
Use guard clause
Improve language
Prettify
Use temp branch
Pass info about who started the job and which job triggered it
Docs: add indexes for monitoring and performance monitoring
clearer-documentation-on-inline-diffs
Add docs for commit diff discussion in merge requests
sorting for tags api
Clear BatchLoader after each spec to prevent holding onto records longer than necessary
Include project in BatchLoader key to prevent returning blobs for the wrong project
moved lfs_blob_ids method into ExtractsPath module
Converted JS modules into exported modules
spec fixes
Bump gitlab-shell version to 5.10.3
Clear caches before updating MR diffs
Use new Ruby version 2.4 in GitLab QA images
moved lfs blob fetch from extractspath file
Update GitLab QA dependencies
...
i.e.
Using compare and swap we update the expires_at value.
The thread that actually is able to update this value will also set
the cache holding method_call enabled state
On GitLab.com, InfluxSampler#sample_objects appears to take 1.2 s or so to
iterate through 1059 objects. This had led to delays of a couple hundred
milliseconds in processing in the main thread. Remove this code since it's not
really being used.
Closesgitlab-com/infrastructure#3250
* upstream/master: (170 commits)
support ordering of project notes in notes api
Redirect to an already forked project if it exists
Reschedule the migration to populate fork networks
Create fork networks for forks for which the source was deleted.
Fix item name and namespace text overflow in Projects dropdown
Minor backport from EE
fix link that was linking to `html` instead of `md`
Backport epic tasklist
Add timeouts for Gitaly calls
SSHUploadPack over Gitaly is now OptOut
fix icon colors in commit list
Fix star icon color/stroke
Backport border inline edit
Add checkboxes to automatically run AutoDevops pipeline
BE for automatic pipeline when enabling Auto DevOps
I am certainly weary of debugging sidekiq but I don't think that's what was meant
Ensure MRs always use branch refs for comparison
Fix issue comment submit button disabled on GFM paste
Lock seed-fu at the correct version in Gemfile.lock
Improve indexes on merge_request_diffs
...
* upstream/master: (126 commits)
Update VERSION to 10.3.0-pre
Update CHANGELOG.md for 10.2.0
default fill color for SVGs
ignore hashed repos (for now) when using `rake gitlab:cleanup:repos`
Use Redis cache for branch existence checks
Update CONTRIBUTING.md: Link definition of done to criteria
Use `make install` for Gitaly setups in non-test environments
FileUploader should check for hashed_storage?(:attachments) to use disk_path
Set the default gitlab-shell timeout to 3 hours
Update composite pipelines index to include "id"
Use arrays in Pipeline#latest_builds_with_artifacts
Fix blank states using old css
Skip confirmation user api
Custom issue tracker
Revert "check for `read_only?` first before seeing if request is disallowed"
add `#with_metadata` scope to remove a N+1 from the notes' API
Fix promoting milestone updating all issuables without milestone
Batchload blobs for diff generation
check for `read_only?` first before seeing if request is disallowed
use `Gitlab::Routing.url_helpers` instead of `Rails.application.routes.url_helpers`
...
* upstream/master: (507 commits)
Add dropdowns documentation
Convert migration to populate latest merge request ID into a background migration
Set 0.69.0 instead of latest for codeclimate image
De-duplicate background migration matchers defined in spec/support/migrations_helpers.rb
Update database_debugging.md
Update database_debugging.md
Move installation of apps higher
Change to Google Kubernetes Cluster and add internal links
Add Ingress description from official docs
Add info on creating your own k8s cluster from the cluster page
Add info about the installed apps in the Cluster docs
Resolve "lock/confidential issuable sidebar custom svg icons iteration"
Update HA README.md to clarify GitLab support does not troubleshoot DRBD.
Update license_finder to 3.1.1
Make sure NotesActions#noteable returns a Noteable in the update action
Cache the number of user SSH keys
Adjust openid_connect_spec to use `raise_error`
Resolve "Clicking on GPG verification badge jumps to top of the page"
Add changelog for container repository path update
Update container repository path reference
...
* upstream/master: (1723 commits)
Resolve "Editor icons"
Refactor issuable destroy action
Ignore routes matching legacy_*_redirect in route specs
Gitlab::Git::RevList and LfsChanges use lazy popen
Gitlab::Git::Popen can lazily hand output to a block
Merge branch 'master-i18n' into 'master'
Remove unique validation from external_url in Environment
Expose `duration` in Job API entity
Add TimeCop freeze for DST and Regular time
Harcode project visibility
update a changelog
Put a condition to old migration that adds fast_forward column to MRs
Expose project visibility as CI variable
fix flaky tests by removing unneeded clicks and focus actions
fix flaky test in gfm_autocomplete_spec.rb
Use Gitlab::Git operations for repository mirroring
Encapsulate git operations for mirroring in Gitlab::Git
Create a Wiki Repository's raw_repository properly
Add `Gitlab::Git::Repository#fetch` command
Fix Gitlab::Metrics::System#real_time and #monotonic_time doc
...
Ryan Cobb
Aleksei Lipniagov
Sarah Yasonik
Sean McGivern
Bob Van Landuyt
Jan Provaznik
Ash McKenzie
Stan Hu
Nick Thomas
Ben Kochie
Lin Jen-Shin
gfyoung
William George
Mark Chao
Yorick Peterse
Pawel Chojnacki
Mario de la Ossa
Takuya Noguchi
🙈 jacopo beschi 🙉
Oswaldo Ferreira
Douwe Maan
Jacopo
micael.bergeron
Grzegorz Bizon
Rémy Coutable