Resumable Long Jobs with Active Job Continuations in Rails 8.1

A long import used to be a gamble. You enqueue it, a deploy lands, the worker is killed, and the job starts over from row one. Rails 8.1 gives that job a memory. Active Job continuations break the work into named steps and, after a restart, pick up at the last step that finished.

This showed up because of how deploys actually work. Kamal gives a job container about thirty seconds to shut down. That is plenty of time for a mailer and nowhere near enough for a backfill. Before 8.1 the honest options were "make the job smaller" or "hope nobody deploys." Continuations are the third option: the job is allowed to stop, and it is allowed to continue.

What a continuation is

You include ActiveJob::Continuable and describe perform as a sequence of steps. A finished step is not run again. An interrupted step can keep a cursor, so the next attempt starts where the last one stopped instead of at the beginning of the method.

class ProcessImportJob < ApplicationJob
  include ActiveJob::Continuable

  def perform(import_id)
    @import = Import.find(import_id)

    step :initialize do
      @import.initialize
    end

    step :process do |step|
      @import.records.find_each(start: step.cursor) do |record|
        record.process
        step.advance! from: record.id
      end
    end

    step :finalize
  end

  private
    def finalize
      @import.finalize
    end
end

There are two ways to write a step. A block is inline, which is what initialize and process use. A symbol, step :finalize, calls a method of the same name. Use the method form when the step is long enough that you do not want it buried inside perform.

The important line is step.advance! from: record.id. That stores a cursor. On the next run, step.cursor is that id, and find_each(start:) skips the rows already handled. Without the cursor, a restart of :process would walk the whole table again even though earlier rows had been saved.

Where this earns its keep

The jobs that benefit are the ones that are long because of volume, not because of a single slow call. Imports, backfills, report builds, and "touch every account" migrations all look the same: thousands of small, safe units of work, and one process that is not safe to kill.

A job that does one external charge and then writes a row does not need this. If that job dies, you want a retry of the whole thing, with idempotency on the charge. Continuations are for work you can checkpoint.

They also change how you think about deploys. A worker that can stop at a step boundary is a worker you can ship during business hours. You still want the step itself to be short. A single step that holds a database lock for twenty minutes will still be killed, and the cursor will not save you if you never called advance!.

How to split the work

A useful step is something you can explain in one sentence and re-run without corrupting data.

• Initialize should be cheap and idempotent. Creating the bookkeeping row, marking the import as running, clearing a previous partial result.
• Process should be the loop, and it should checkpoint often. Advancing once per row is fine when the row is expensive. Advancing every few hundred rows is fine when the row is a simple update. The cost of the checkpoint has to stay smaller than the cost of repeating the work.
• Finalize should be the part you only want once: sending the "import finished" mail, flipping the status, enqueueing the next job. Because earlier steps are skipped on retry, finalize runs after the data is actually done.

If process is not idempotent, the cursor is a lie. Writing the row and then crashing before advance! means the next attempt will write it again. Do the write in a way that is safe to repeat, or advance only after the write commits.

What we still watch in production

Continuations do not replace a queue. Solid Queue or Sidekiq still owns retries, concurrency, and dead jobs. What changes is the shape of the retry. Look at jobs that retry more than once and check whether they are making progress. A cursor that never moves is a step that errors before advance!. A job that finishes :process and then fails in :finalize on every attempt is a bug in finalize, not a reason to reprocess the file.

Also check the shutdown window. Thirty seconds is the Kamal default, not a law. If a single record takes longer than the grace period, no checkpoint inside the loop will run. Shrink the unit of work, or raise the stop timeout for that service on purpose.

If you are upgrading into this

Rails 8.1 is a small step from 8.0, and continuations are opt-in. You do not have to rewrite the queue. Pick the one job that hurts when it restarts, wrap it, and deploy that. The rest of the app can stay as it is.

If the job in question is the one nobody wants to touch because the last retry corrupted a batch of rows, that is the kind of work we take on at Rails Agency. We would rather checkpoint it than tell you to stop deploying.