Zero downtime credential updates on Heroku.

When you use Rails credentials on Heroku, you probably have your key in your environment variables. Probably RAILS_MASTER_KEY. Let’s say you want to rotate this key. You re-encrypt your credentials, push the code (reboot 1) and then add the env-var (reboot 2). You could also flip these two steps around.

In between these two reboots, your app won’t work since you either have a new credentials file with the old key, or the old credentials file with a new key.

What I want to happen is:

1. Set the new key in a new environment variable: RAILS_MASTER_KEY_NEW
2. The rails app will boot again since Heroku reboots after each ENV update and see the new key
3. The rails app checks if the new key can decrypt the credentials, if not, it falls back to the old key

This is how you can do this:

Add this to your config/application.rb:

class Application < Rails::Application # Don't add this obviously

    # When we've defined RAILS_MASTER_KEY_NEW it means we are rotating the encryption key
    # for our credentials. What we want to do then is:
    # 1. Check if we can decrypt the current credentials file with the new key
    # 2. If we can, we will change RAILS_MASTER_KEY to equal RAILS_MASTER_KEY_NEW
    # 3. If not, we will fallback to the old key, thus leave RAILS_MASTER_KEY alone
    if ENV.fetch("RAILS_MASTER_KEY_NEW", false)
      require 'logger'

      cred_logger = Logger.new($stdout) # Rubocop wanted this. He is the boss.
      credential_path = Rails.root.join("config/credentials/#{Rails.env}.yml.enc")
      begin
        Rails.application.encrypted(credential_path, env_key: 'RAILS_MASTER_KEY_NEW').read
        ENV["RAILS_MASTER_KEY"] = ENV.fetch("RAILS_MASTER_KEY_NEW")
        cred_logger.info "application.rb: Using the new credential key, it works!"
      rescue ActiveSupport::MessageEncryptor::InvalidMessage => e
        cred_logger.info "application.rb: Using the old key"
      end
    end

    #...

That’s it.

This needs to be at the top because Rails will pretty quickly try to load the credentials. Some trial and error will probably reveal the sweet spot, but this works just fine here.

After successfully updating your credentials file, you can remove the new key again. Be sure to set the value of RAILS_MASTER_KEY to that of RAILS_MASTER_KEY_NEW though!