Ruby apps under runit: notes to self

I was recently migrating nearly ten-year-old Ruby software from an Ubuntu 10:04 VM to something more recent. It took me way longer than I expected, due to two major snags.

Snag one: Bundler and per-user runsvdir

The apps in their original Lucid Lynx setting were run from a bash script that resembled this:

while (/bin/true); do
  (bundle exec 'thin -C config_foo.yml -R config.ru start' 2>&1)
  echo "Server died at `date`. Respawning.." >&2
  sleep 2
done

The app stayed up for years, so I guess this unsophisticated approach was good enough. For ease of deployment (and improved logging, see point #2 below), I decided to move all ruby web apps on this VPS to a user-local runit supervision tree.

In practice, this means that there’s a /etc/service/webuser-sv directory, containing a runit service which launches a nested runsvdir program as the webuser-sv user.

$ cat /etc/service/webuser-sv
#!/bin/sh
exec 2>&1
exec chpst -uwebuser runsvdir /home/webuser/service

Now, I can define all of webuser’s ruby apps as entries under /home/webuser/service/*, and have them supervised without bash hackery.

The snag was that the apps would crash with this error, but only when run as part of the runit supervision tree:

2019-06-25_11:27:37.51241 `/` is not writable.
2019-06-25_11:27:37.51244 Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.

But if I ran the runit run scripts by hand, the apps started up and worked correctly.

After a lot of false starts and pointless github issue rabbit-holes, I realized that the runsvdir process managing the user-local supervision tree was launched with chpst by the master runit supervisor. Specifying a UID for chpst with -u does not automatically mean that the profile for this user gets loaded. In particular, not even $HOME was configured in the runit supervisor environment.

Bundler needs ‘$HOME’ to be set, otherwise it gets confused.

Hence, my runit run files now look like this:

exec 2>&1
export RUBYOPT=-W0
export HOME=/home/web
export LANG=en_US.UTF-8
cd /home/web/webapp.1.com
exec bundle exec rackup -E production -s thin -o 0.0.0.0 -p 4567 config.ru

Duplication could be further removed by setting up an envdir, and running the user-level runsvdir with this envdir passed to chpst … but the above solution is good enough for today.

Snag two: Don’t ever redirect $stdin and $stderr

Runit has a wonderfully clean approach to logging, predating the 12-factor app, but very similar in spirit. Run your app in the foreground, and pipe its output to your log processor. This way applications themselves never need to concern themselves with logging frameworks and infrastructure.

Now ten years ago, when I launched these apps from bash scripts, they apps themselves definitely needed to know what to do with their logs. Hence, these two monstrous lines:

settings.configure do |s|
    $stdout = File.open("log/log_out.txt","a")
    $stderr = File.open("log/log_err.txt","a")

These two lines had me stumped as to why an app was silently crashing at startup with nothing in the runit logs. After removing the two lines I was able to see the real reason for the crash (some missing assets).

Sure, the exception had been logged all along to log/log_err.txt, but I’d completely forgotten to look there, expecting the logging to be handled by runit’s log facility.

Never redirect $stdout and $stdin inside the app, kids. Your future self will thank you.