Marcin's Musings

Random bits of stuff

Setting Up Monit on Elastic Beanstalk

| Comments

Continuing on with yesterday’s theme of posts about Amazon Elastic Beanstalk bits and pieces I’ve discovered I’m going to talk about how to setup the simple monitoring tool Monit on your EC2 instance(s). In my case I’m using Monit to monitor the delayed_job service I mentioned in my last post, and restart it if it dies.

Monit isn’t installed on the instance by default so we’ll need to install the package. After that we just need to configure it to monitor delayed_job and make sure it’s running. This is actually pretty simple with just the standard .config files.

Here is my .ebextensions/90monit.config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
packages:
  yum:
    monit: []
files:
  "/etc/monit.d/delayed_job":
    mode: "000644"
    owner: root
    group: root
    content: |
      set mailserver smtp.gmail.com port 587
      username "system@XXXXXX" password "XXXXXX" using tlsv1
      with timeout 30 seconds

      set httpd port 2812 and
        use address localhost  # only accept connection from localhost
        allow localhost        # allow localhost to connect to the server and

      check process delayed_job
        with pidfile /var/app/support/pids/delayed_job.pid every 2 cycles
        start program = "/usr/bin/env BUNDLE_WITHOUT=test:development RAILS_ENV=production /var/app/current/script/delayed_job --pid-dir=/var/app/support/pids start" as uid webapp and gid webapp
        stop program = "/usr/bin/env BUNDLE_WITHOUT=test:development RAILS_ENV=production /var/app/current/script/delayed_job --pid-dir=/var/app/support/pids stop" as uid webapp and gid webapp

      alert alerts@XXXXX with mail-format {
        from: system@XXXXXX
        reply-to: alerts@XXXXXX
        subject: XXXXXX $SERVICE $EVENT at $DATE
      }
commands:
  remove_bak:
    command: "rm /etc/monit.d/delayed_job.bak"
    ignoreErrors: true
service:
  sysvinit:
    monit:
      ensureRunning: true
      enabled: true

Basically what we’re doing is:

  • Using yum to install the monit package
  • deploying a config file for monitoring delayed_job into /etc/monit.d (more on this later)
  • When you deploy the config script, if it already exists Elastic Beanstalk will create a backup. We need to delete this file if it exists because monit will load all files in /etc/monit.d regardless of extension
  • ensure the monit service is enabled and running

So the config file for monit is where the interesting stuff is. We’re configuring stuff in here that we’d probably normally put in core monit config, but in the interests of keeping the deployment script simple we do it all from one script.

What we’re doing with this config is:

  • Setup our mailserver to use GMail SMTP to send the emails – I’m using a specific service account.
  • Setup the monit httpd service so we can run monit status on the instance when debugging
  • Setup the config for monitoring delayed_job – more on this below.
  • Last of all we setup an alert to be sent to our alerts email when there’s a status change for delayed_job

So the hardest part was getting the commands right. There were a few pages I found that gave me most of the answer, however not fully – for me it was failing for ages because I didn’t have the BUNDLE_WITHOUT variable set when trying to run the script and I have some test specific dependencies.

The part that makes this more difficult is because monit runs your commands with an extremely cut down shell – all you get is basically the path, with no other environment variables. So what we’re doing is running our command through /usr/bin/env to setup the environment (although I’m not sure that’s actually doing anything), and explicitly passing the RAILS_ENV. We also need to specify the user to run the commands as, otherwise the delayed_job service will start up as root – which is not good.

I could probably improve this script by refactoring the commands to use the Elastic Beanstalk environment like $EB_CONFIG_APP_CURRENT rather than hardcoding /var/app/current, but it’s working for now so I’ve left it alone – and as I said above, monit runs commands in a very cut down environment so getting those variables in might not be easy.

Anyway, that’s another piece of my Elastic Beanstalk puzzle. Pretty simple in hindsight, but probably took a good hour or two of fiddling around on the instance, trying various approaches, reading documentation and so on to finally get it all working reliably and repeatably.

Comments