Cron monitoring with Blazer

 · 
rails developer-tools

Many apps have recurring jobs and we should track them. There are third-party services to do this, but I like to keep things simple and own my data.

I'll show you how to keep track of cron jobs within your Rails application and report on them through Slack. We collect logs in your database, so you can use them for other things as well.

When done we can write this SQL:

SELECT * FROM job_logs WHERE name = 'ping_site' AND strftime('%Y-%m-%d', created_at) = CURRENT_DATE;

And get notifications like these:

Slack notification of check passing again

Using this UI:

Creating our check for today's logs

Query passing again

There are two steps to this:

  1. We will build a logging endpoint into our Rails app
  2. We will install the blazer gem for building checks and notifications

Logging endpoint

I've taken inspiration from Healthchecks.io:

Run our command and then curl to our logging endpoint:

some_command && curl -fsS -m 10 --retry 5 -o /dev/null https://our-domain.com/job_logs/:token/:name

We match a log to a job through the :name. Then we hash a secret and :name into a :token and use it to prevent people from pushing bogus logs to our database.

Let's create the table we'll use first:

# The migration

class CreateJobLogs < ActiveRecord::Migration[7.0]
def change
create_table :job_logs do |t|
t.string :name, null: false

t.string :status, null: false, default: 'success'

t.string :error
t.string :duration
t.string :output

t.timestamps
end
end
end

The fields :status, :error, :duration and :output might be useful in the future.

Now for the controller. We need it to check the token, and create a log record:

# Route - We use GET instead of POST
get '/job_logs/:token/:name', to: 'job_logs#create'

# The controller
class JobLogsController < ApplicationController
def create
@job_log = JobLog.new(job_log_params)

return render json: { error: 'Invalid token' }, status: :unauthorized unless @job_log.valid_token?(params.fetch(:token))

@job_log.save!
end

private

def job_log_params
params.permit(:name, :status, :error, :duration, :output)
end
end

Our model uses the Rails app's secret key base to generate a token and looks like this:

class JobLog < ApplicationRecord
validates :name, presence: true
validates :status, presence: true

def token
Digest::SHA1.hexdigest("#{name}-#{secret}")
end

def valid_token?(input_token)
token == input_token
end

private

def secret
Rails.application.secrets.secret_key_base
end
end

Let's try it out:

curl -fsS -m 10 --retry 5 -o /dev/null http://localhost:3000/job_logs/123/ping_site
curl: (22) The requested URL returned error: 401

# Let's generate a token:
bin/rails r "puts JobLog.new(name: 'ping_site').token"
40f958deccb8ba624562a7c81e5609ab66a71b4a

curl -fsS -m 10 --retry 5 -o /dev/null http://localhost:3000/job_logs/40f958deccb8ba624562a7c81e5609ab66a71b4a/ping_site

Set up blazer checks

The blazer readme describes how to install it. I've mounted it to /blazer.

Take extra care to set up authentication to keep out prying eyes.

To get Slack notifications working, you will have to set the BLAZER_SLACK_WEBHOOK_URL and also set an action_mailer default_url option like so:

# config/environments/development.rb
config.action_mailer.default_url_options = { host: "localhost:3000" }

Let's write a query to have a look at all our job logs so far:

Setting up a check in Blazer

If we want to run a job daily, we'll have to truncate the records to their day. We create the following query:

SELECT * FROM job_logs WHERE name = 'ping_site' AND strftime('%Y-%m-%d', created_at) = CURRENT_DATE;

Creating our query in blazer

Now we create a check based on this query. Notice the Slack channel I specified:

Creating our check for today's logs

Running cron

Finally set up a cron job to run blazer's checking command:

rake blazer:run_checks SCHEDULE="1 day"

When this runs, Blazer will check if our query returns a result. If it doesn't, it sends a Slack notification like so:

Slack notification of failure

Once the check starts passing again, we will also get a notification:

Slack notification of check passing again

Conclusion

There you have it, we have a neat, low-effort way to track cron jobs from within our app. On top of this, we could build extra features such as API throttling or reports.

Blazer enables you two to make something that otherwise requires quite a bit of coding. So think about what else you could do with it.

Enjoy!

Comments