Move Gitlab::SidekiqMonitor to namespace Gitlab::SidekiqDaemon::Monitor
- Class name and file name change
- File path change to lib/gitlab/sidekiq_daemon/monitor.rb
- Update class usage/reference in other files, including documentation
This makes:
- very shallow `Middleware::Monitor` to only request tracking
of sidekiq jobs,
- `SidekiqStatus::Monitor` to be responsible to maintain persistent
connection to receive messages,
- `SidekiqStatus::Monitor` to always use structured logging
and instance variables
This adds a middleware to track all threads
for running jobs.
This makes sidekiq to watch for redis-delivered notifications.
This makes be able to send notification to interrupt
running sidekiq jobs.
This does not take into account any native code,
as `Thread.raise` generates exception once the control gets
back to Ruby.
The separate measure should be taken to interrupt gRPC, shellouts,
or anything else that escapes Ruby.
This will help identify Sidekiq jobs that invoke excessive number of
filesystem access.
The timing data is stored in `RequestStore`, but this is only active
within the middleware and is not directly accessible to the Sidekiq
logger. However, it is possible for the middleware to modify the job
hash to pass this data along to the logger.
This change is a fairly straightforward refactor to extract the tracing
and correlation-id code from the gitlab rails codebase into the new
LabKit-Ruby project.
The corresponding import into LabKit-Ruby was in
https://gitlab.com/gitlab-org/labkit-ruby/merge_requests/1
The code itself remains very similar for now.
Extracting it allows us to reuse it in other projects, such as
Gitaly-Ruby. This will give us the advantages of correlation-ids and
distributed tracing in that project too.
Sidekiq jobs frequently spawn long-lived child processes to do work.
In some circumstances, these can be reparented to init when sidekiq is
terminated, leading to duplication of work and strange concurrency
problems.
This commit changes sidekiq so that, if run as a process group leader,
it will forward `INT` and `TERM` signals to the whole process group. If
the memory killer is active, it will also use the process group when
resorting to `kill -9` to shut down.
These changes mean that a naive `kill <pid-of-sidekiq>` will now do the
right thing, killing any child processes spawned by sidekiq, as long as
the process supervisor placed it in its own process group.
If sidekiq isn't a process group leader, this new code is skipped.
The Correlation ID is taken or generated from received X-Request-ID.
Then it is being passed to all executed services (sidekiq workers
or gitaly calls).
The Correlation ID is logged in all structured logs as `correlation_id`.
The sidekiq memory killer prints warnings like
gitlab-sidekiq[8245]: 2017-09-27T11:31:43.192Z 8245 TID-ov9l6b4e0 WARN:
this thread will shut down PID 8245 - Worker ProjectWebHookWorker -
JID-dd3b1c23a653deec5e575ab4in 900 seconds
There is a space missing between the JID and 'in'.
Log sidekiq arguments as json
Logging the sidekiq job arguments as a ruby literal structure
is akward. Specially when parsing the logs.
JSON is a standar format.
See merge request !3735
This makes the memory growth-triggered Sidekiq restarts more reliable by
reducing the chance that Sidekiq ends up in a state where it is not
accepting new jobs but also not shutting down: SIGKILL is more likely to
work than SIGTERM.
It looks like SIGTERM may not be enough to shut down a Sidekiq process
when its RSS has gotten too big. This change will allow us to experiment
with sending SIGKILL instead of SIGTERM to Sidekiq processes on
gitlab.com.
When enabled, this middleware allows Sidekiq to detect that its RSS has
exceeded a maximum value, triggering a graceful shutdown. This
middleware should be combined with external process supervision that
will restart Sidekiq after the graceful shutdown, such as Runit.
GitLab Bot
Qingyu Zhao
Andrew Newdigate
Kamil Trzciński
Stan Hu
Shinya Maeda
Ryan Cobb
Nick Thomas
Douwe Maan
gfyoung
🙈 jacopo beschi 🙉
Jacob Vosmaer (GitLab)
Benjamin Drung
Rémy Coutable
James Lopez
Jacob Vosmaer
David