This change is a fairly straightforward refactor to extract the tracing
and correlation-id code from the gitlab rails codebase into the new
LabKit-Ruby project.
The corresponding import into LabKit-Ruby was in
https://gitlab.com/gitlab-org/labkit-ruby/merge_requests/1
The code itself remains very similar for now.
Extracting it allows us to reuse it in other projects, such as
Gitaly-Ruby. This will give us the advantages of correlation-ids and
distributed tracing in that project too.
Sidekiq jobs frequently spawn long-lived child processes to do work.
In some circumstances, these can be reparented to init when sidekiq is
terminated, leading to duplication of work and strange concurrency
problems.
This commit changes sidekiq so that, if run as a process group leader,
it will forward `INT` and `TERM` signals to the whole process group. If
the memory killer is active, it will also use the process group when
resorting to `kill -9` to shut down.
These changes mean that a naive `kill <pid-of-sidekiq>` will now do the
right thing, killing any child processes spawned by sidekiq, as long as
the process supervisor placed it in its own process group.
If sidekiq isn't a process group leader, this new code is skipped.
The Correlation ID is taken or generated from received X-Request-ID.
Then it is being passed to all executed services (sidekiq workers
or gitaly calls).
The Correlation ID is logged in all structured logs as `correlation_id`.
The sidekiq memory killer prints warnings like
gitlab-sidekiq[8245]: 2017-09-27T11:31:43.192Z 8245 TID-ov9l6b4e0 WARN:
this thread will shut down PID 8245 - Worker ProjectWebHookWorker -
JID-dd3b1c23a653deec5e575ab4in 900 seconds
There is a space missing between the JID and 'in'.
Log sidekiq arguments as json
Logging the sidekiq job arguments as a ruby literal structure
is akward. Specially when parsing the logs.
JSON is a standar format.
See merge request !3735
This makes the memory growth-triggered Sidekiq restarts more reliable by
reducing the chance that Sidekiq ends up in a state where it is not
accepting new jobs but also not shutting down: SIGKILL is more likely to
work than SIGTERM.
It looks like SIGTERM may not be enough to shut down a Sidekiq process
when its RSS has gotten too big. This change will allow us to experiment
with sending SIGKILL instead of SIGTERM to Sidekiq processes on
gitlab.com.
When enabled, this middleware allows Sidekiq to detect that its RSS has
exceeded a maximum value, triggering a graceful shutdown. This
middleware should be combined with external process supervision that
will restart Sidekiq after the graceful shutdown, such as Runit.
Andrew Newdigate
Nick Thomas
Kamil Trzciński
Douwe Maan
gfyoung
🙈 jacopo beschi 🙉
Jacob Vosmaer (GitLab)
Benjamin Drung
Rémy Coutable
Stan Hu
James Lopez
Jacob Vosmaer
David