Cron monitoring with Blazer

Many apps have recurring jobs and we should track them. There are third-party services to do this, but I like to keep things simple and own my data.

I’ll show you how to keep track of cron jobs within your Rails application and report on them through Slack. We collect logs in your database, so you can use them for other things as well.

When done we can write this SQL:

SELECT * FROM job_logs WHERE name = 'ping_site' AND strftime('%Y-%m-%d', created_at) = CURRENT_DATE;

And get notifications like these:

Using this UI:

There are two steps to this:

1. We will build a logging endpoint into our Rails app
2. We will install the blazer gem for building checks and notifications

Logging endpoint

I’ve taken inspiration from Healthchecks.io:

Run our command and then curl to our logging endpoint:

some_command && curl -fsS -m 10 --retry 5 -o /dev/null https://our-domain.com/job_logs/:token/:name

We match a log to a job through the :name. Then we hash a secret and :name into a :token and use it to prevent people from pushing bogus logs to our database.

Let’s create the table we’ll use first:

# The migration

class CreateJobLogs < ActiveRecord::Migration[7.0]
  def change
    create_table :job_logs do |t|
      t.string :name, null: false

      t.string :status, null: false, default: 'success'

      t.string :error
      t.string :duration
      t.string :output

      t.timestamps
    end
  end
end

The fields :status, :error, :duration and :output might be useful in the future.

Now for the controller. We need it to check the token, and create a log record:

# Route - We use GET instead of POST
get '/job_logs/:token/:name', to: 'job_logs#create'

# The controller
class JobLogsController < ApplicationController
  def create
    @job_log = JobLog.new(job_log_params)

    return render json: { error: 'Invalid token' }, status: :unauthorized unless @job_log.valid_token?(params.fetch(:token))

    @job_log.save!
  end

  private

  def job_log_params
    params.permit(:name, :status, :error, :duration, :output)
  end
end

Our model uses the Rails app’s secret key base to generate a token and looks like this:

class JobLog < ApplicationRecord
  validates :name, presence: true
  validates :status, presence: true

  def token
    Digest::SHA1.hexdigest("#{name}-#{secret}")
  end

  def valid_token?(input_token)
    token == input_token
  end

  private

  def secret
    Rails.application.secrets.secret_key_base
  end
end

Let’s try it out:

curl -fsS -m 10 --retry 5 -o /dev/null http://localhost:3000/job_logs/123/ping_site
  curl: (22) The requested URL returned error: 401

# Let's generate a token:
bin/rails r "puts JobLog.new(name: 'ping_site').token"
  40f958deccb8ba624562a7c81e5609ab66a71b4a

curl -fsS -m 10 --retry 5 -o /dev/null http://localhost:3000/job_logs/40f958deccb8ba624562a7c81e5609ab66a71b4a/ping_site

Set up blazer checks

The blazer readme describes how to install it. I’ve mounted it to /blazer.

Take extra care to set up authentication to keep out prying eyes.

To get Slack notifications working, you will have to set the BLAZER_SLACK_WEBHOOK_URL and also set an action_mailer default_url option like so:

# config/environments/development.rb
config.action_mailer.default_url_options = { host: "localhost:3000" }

Let’s write a query to have a look at all our job logs so far:

If we want to run a job daily, we’ll have to truncate the records to their day. We create the following query:

SELECT * FROM job_logs WHERE name = 'ping_site' AND strftime('%Y-%m-%d', created_at) = CURRENT_DATE;

Now we create a check based on this query. Notice the Slack channel I specified:

Running cron

Finally set up a cron job to run blazer’s checking command:

rake blazer:run_checks SCHEDULE="1 day"

When this runs, Blazer will check if our query returns a result. If it doesn’t, it sends a Slack notification like so:

Once the check starts passing again, we will also get a notification:

Conclusion

There you have it, we have a neat, low-effort way to track cron jobs from within our app. On top of this, we could build extra features such as API throttling or reports.

Blazer enables you two to make something that otherwise requires quite a bit of coding. So think about what else you could do with it.

Enjoy!