6.3 KiB

Raw Blame History

stage	group	info
Systems	Distribution	To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments

Debugging tips (FREE SELF)

Sometimes things don't work the way they should. Here are some tips on debugging issues out in production.

Starting a Rails console session

Troubleshooting and debugging your GitLab instance often requires a Rails console.

Your type of GitLab installation determines how to start a rails console. See also:

GitLab Rails Console Cheat Sheet.

Enabling Active Record logging

You can enable output of Active Record debug logging in the Rails console session by running:

ActiveRecord::Base.logger = Logger.new($stdout)

This shows information about database queries triggered by any Ruby code you may run in the console. To turn off logging again, run:

ActiveRecord::Base.logger = nil

Disabling database statement timeout

You can disable the PostgreSQL statement timeout for the current Rails console session by running:

ActiveRecord::Base.connection.execute('SET statement_timeout TO 0')

This change only affects the current Rails console session and is not persisted in the GitLab production environment or in the next Rails console session.

Output Rails console session history

If you'd like to output your Rails console command history in a format that's easy to copy and save for future reference, you can run:

puts Readline::HISTORY.to_a

Using the Rails runner

If you need to run some Ruby code in the context of your GitLab production environment, you can do so using the Rails runner. When executing a script file, the script must be accessible by the git user.

For Omnibus installations

sudo gitlab-rails runner "RAILS_COMMAND"

# Example with a two-line Ruby script
sudo gitlab-rails runner "user = User.first; puts user.username"

# Example with a ruby script file (make sure to use the full path)
sudo gitlab-rails runner /path/to/script.rb

For installations from source

sudo -u git -H bundle exec rails runner -e production "RAILS_COMMAND"

# Example with a two-line Ruby script
sudo -u git -H bundle exec rails runner -e production "user = User.first; puts user.username"

# Example with a ruby script file (make sure to use the full path)
sudo -u git -H bundle exec rails runner -e production /path/to/script.rb

Mail not working

A common problem is that mails are not being sent for some reason. Suppose you configured an SMTP server, but you're not seeing mail delivered. Here's how to check the settings:

Run a Rails console.
Look at the ActionMailer delivery_method to make sure it matches what you intended. If you configured SMTP, it should say :smtp. If you're using Sendmail, it should say :sendmail:
```
irb(main):001:0> ActionMailer::Base.delivery_method
=> :smtp
```
If you're using SMTP, check the mail settings:
```
irb(main):002:0> ActionMailer::Base.smtp_settings
=> {:address=>"localhost", :port=>25, :domain=>"localhost.localdomain", :user_name=>nil, :password=>nil, :authentication=>nil, :enable_starttls_auto=>true}
```
In the example above, the SMTP server is configured for the local machine. If this is intended, you may need to check your local mail logs (for example, /var/log/mail.log) for more details.
Send a test message via the console.
```
irb(main):003:0> Notify.test_email('youremail@email.com', 'Hello World', 'This is a test message').deliver_now
```
If you do not receive an email and/or see an error message, then check your mail server settings.

Advanced Issues

For more advanced issues, gdb is a must-have tool for debugging issues.

The GNU Project Debugger (GDB)

To install on Ubuntu/Debian:

sudo apt-get install gdb

On CentOS:

sudo yum install gdb

rbtrace

GitLab 11.2 ships with rbtrace, which allows you to trace Ruby code, view all running threads, take memory dumps, and more. However, this is not enabled by default. To enable it, define the ENABLE_RBTRACE variable to the environment. For example, in Omnibus:

gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"}

Then reconfigure the system and restart Puma and Sidekiq. To run this in Omnibus, run as root:

/opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/rbtrace

Common Problems

Many of the tips to diagnose issues below apply to many different situations. We use one concrete example to illustrate what you can do to learn what is going wrong.

GitLab: API is not accessible

This often occurs when GitLab Shell attempts to request authorization via the internal API (for example, http://localhost:8080/api/v4/internal/allowed), and something in the check fails. There are many reasons why this may happen:

Timeout connecting to a database (for example, PostgreSQL or Redis)
Error in Git hooks or push rules
Error accessing the repository (for example, stale NFS handles)

To diagnose this problem, try to reproduce the problem and then see if there is a Unicorn worker that is spinning via top. Try to use the gdb techniques above. In addition, using strace may help isolate issues:

strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt

If you cannot isolate which Unicorn worker is the issue, try to run strace on all the Unicorn workers to see where the /internal/allowed endpoint gets stuck:

ps auwx | grep puma | awk '{ print " -p " $2}' | xargs  strace -ttTfyyy -s 1024 -o /tmp/puma.txt

The output in /tmp/puma.txt may help diagnose the root cause.

6.3 KiB Raw Blame History