Ruby apps under runit: notes to self
Today I learned (2019-06-25)
I was recently migrating nearly ten-year-old Ruby software from an Ubuntu 10:04 VM to something more recent. It took me way longer than I expected, due to two major snags.
Snag one: Bundler and per-user runsvdir
The apps in their original Lucid Lynx setting were run from a bash script that resembled this:
while (/bin/true); do (bundle exec 'thin -C config_foo.yml -R config.ru start' 2>&1) echo "Server died at `date`. Respawning.." >&2 sleep 2 done
The app stayed up for years, so I guess this unsophisticated approach was good enough. For ease of deployment (and improved logging, see point #2 below), I decided to move all ruby web apps on this VPS to a user-local runit supervision tree.
In practice, this means that there’s a
containing a runit service which launches a nested
runsvdir program as the
$ cat /etc/service/webuser-sv #!/bin/sh exec 2>&1 exec chpst -uwebuser runsvdir /home/webuser/service
Now, I can define all of
webuser’s ruby apps as entries under
/home/webuser/service/*, and have them supervised without bash hackery.
The snag was that the apps would crash with this error, but only when run as part of the runit supervision tree:
2019-06-25_11:27:37.51241 `/` is not writable. 2019-06-25_11:27:37.51244 Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.
But if I ran the runit
run scripts by hand, the apps started up and worked correctly.
After a lot of false starts and pointless github issue rabbit-holes, I realized
runsvdir process managing the user-local supervision tree was
launched with chpst by the master
runit supervisor. Specifying a UID for
-u does not automatically
mean that the profile for this user gets loaded. In particular, not even
$HOME was configured in the runit supervisor environment.
Bundler needs ‘$HOME’ to be set, otherwise it gets confused.
Hence, my runit run files now look like this:
exec 2>&1 export RUBYOPT=-W0 export HOME=/home/web export LANG=en_US.UTF-8 cd /home/web/webapp.1.com exec bundle exec rackup -E production -s thin -o 0.0.0.0 -p 4567 config.ru
Duplication could be further removed by setting up an
envdir, and running the
runsvdir with this envdir passed to
chpst … but the above
solution is good enough for today.
Snag two: Don’t ever redirect $stdin and $stderr
Runit has a wonderfully clean approach to logging, predating the 12-factor app, but very similar in spirit. Run your app in the foreground, and pipe its output to your log processor. This way applications themselves never need to concern themselves with logging frameworks and infrastructure.
Now ten years ago, when I launched these apps from bash scripts, they apps themselves definitely needed to know what to do with their logs. Hence, these two monstrous lines:
settings.configure do |s| $stdout = File.open("log/log_out.txt","a") $stderr = File.open("log/log_err.txt","a")
These two lines had me stumped as to why an app was silently crashing at startup with nothing in the runit logs. After removing the two lines I was able to see the real reason for the crash (some missing assets).
Sure, the exception had been logged all along to
log/log_err.txt, but I’d
completely forgotten to look there, expecting the logging to be handled by
runit’s log facility.
Never redirect $stdout and $stdin inside the app, kids. Your future self will thank you.