Continuing on with yesterday’s theme of posts about Amazon Elastic Beanstalk bits and pieces I’ve discovered I’m going to talk about how to setup the simple monitoring tool Monit on your EC2 instance(s). In my case I’m using Monit to monitor the delayed_job service I mentioned in my last post, and restart it if it dies.
Monit isn’t installed on the instance by default so we’ll need to install the package. After that we just need to configure it to monitor delayed_job and make sure it’s running. This is actually pretty simple with just the standard
Here is my
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
Basically what we’re doing is:
- Using yum to install the monit package
- deploying a config file for monitoring delayed_job into
/etc/monit.d(more on this later)
- When you deploy the config script, if it already exists Elastic Beanstalk will create a backup. We need to delete this file if it exists because monit will load all files in
/etc/monit.dregardless of extension
- ensure the monit service is enabled and running
So the config file for monit is where the interesting stuff is. We’re configuring stuff in here that we’d probably normally put in core monit config, but in the interests of keeping the deployment script simple we do it all from one script.
What we’re doing with this config is:
- Setup our mailserver to use GMail SMTP to send the emails – I’m using a specific service account.
- Setup the monit httpd service so we can run
monit statuson the instance when debugging
- Setup the config for monitoring
delayed_job– more on this below.
- Last of all we setup an alert to be sent to our alerts email when there’s a status change for delayed_job
So the hardest part was getting the commands right. There were a few pages I found that gave me most of the answer, however not fully – for me it was failing for ages because I didn’t have the
BUNDLE_WITHOUT variable set when trying to run the script and I have some test specific dependencies.
The part that makes this more difficult is because monit runs your commands with an extremely cut down shell – all you get is basically the path, with no other environment variables. So what we’re doing is running our command through
/usr/bin/env to setup the environment (although I’m not sure that’s actually doing anything), and explicitly passing the RAILS_ENV. We also need to specify the user to run the commands as, otherwise the delayed_job service will start up as root – which is not good.
I could probably improve this script by refactoring the commands to use the Elastic Beanstalk environment like
$EB_CONFIG_APP_CURRENT rather than hardcoding
/var/app/current, but it’s working for now so I’ve left it alone – and as I said above, monit runs commands in a very cut down environment so getting those variables in might not be easy.
Anyway, that’s another piece of my Elastic Beanstalk puzzle. Pretty simple in hindsight, but probably took a good hour or two of fiddling around on the instance, trying various approaches, reading documentation and so on to finally get it all working reliably and repeatably.