Durable workflows

A long-running or human-in-the-loop workflow can pause durably and continue later — possibly in another process — without re-running completed, already-paid-for work. Three small primitives compose over the existing run persistence (no step-graph engine, no replay log, no scheduler):

• checkpoint(name) { … } runs its block once and stores the json-serializable result under name in the run’s state. On a later run/resume of the same run, a present checkpoint returns the stored value without re-running the block. This is the tool that makes resume cheap and side-effect-safe.
• suspend!(reason:, resume_key: nil) pauses the run: it marks the run "suspended" (a non-failure outcome, distinct from "failed") and returns it to the caller — Workflow.run does not raise. Call it outside a checkpoint.
• Workflow.resume(run_id, input = {}) (sync) and Workflow.resume_later(run_id, input = {}) (enqueued) continue a suspended run, feeding input in as #resume_input.

The basic loop

class DocumentApproval < Nexo::Workflow
  def call(payload)
    document = checkpoint(:fetch) { fetch_expensive(payload[:id]) } # paid for once

    # `resume_input` is {} on the first pass, so we pause; on resume the host
    # feeds { approved: true }, so we fall through and publish.
    suspend!(reason: "awaiting approval") unless resume_input[:approved]

    checkpoint(:publish) { publish!(document) }
    { done: true }
  end
end

run = DocumentApproval.run(id: 42)   # reaches suspend!, returns
run.status                            # => "suspended"
run.suspend_reason                    # => "awaiting approval"  (AR store)
run.state["fetch"]                    # => the fetched document (checkpoint persisted)

# ...later, once a human approves — possibly in another process:
resumed = DocumentApproval.resume(run.id, approved: true)
resumed.status                        # => "done"  (the :fetch block did NOT re-run)

A host UI lists paused runs with the suspended scope and inspects them with the readers (Nexo ships no controllers/views — the UI is your app’s job):

Nexo::WorkflowRun.suspended            # scope: all paused runs
run.suspended?                          # => true
run.suspend_reason                      # => "awaiting approval"
run.checkpoint_result(:fetch)           # => the stored :fetch value, or nil

For a durable, cross-process resume from a background job, enqueue it — the job carries the run id plus the (json-safe) resume input; the payload still lives on the run:

# The resume input is a positional Hash (queue: is the only keyword), so pass it
# as { approved: true } — bare approved: true would bind as an unknown keyword.
DocumentApproval.resume_later(run.id, { approved: true }, queue: :nexo)

Live example

The full offline approval flow is runnable in the repo:

ruby -Ilib examples/approval_workflow.rb

View examples/approval_workflow.rb on GitHub →

Parallel checkpoints — checkpoint_all

When several checkpoints are independent (no step depends on another’s result), run them concurrently with checkpoint_all(name => callable, …) instead of a sequence of checkpoint calls. It fans the pending steps out through Nexo.concurrent — all in flight at once — and persists each step as it completes (not the batch as a whole), so a resume after a partial failure only re-runs the steps that never landed:

class BuildDashboard < Nexo::Workflow
  def call(payload)
    data = checkpoint_all(
      account: -> { fetch_account(payload[:id]) },   # these two run
      usage:   -> { fetch_usage(payload[:id]) }      # concurrently
    )
    { report: render(data[:account], data[:usage]) }
  end
end

checkpoint_all returns a Hash keyed by the original names you passed (data[:account]), with values read back from state — the same shape whether a value came from this pass or a prior one. Each newly-completed step also surfaces a "checkpoint"-typed event on the run’s event log (data is the step name only, never the value — so a dashboard can show batch progress without the event log carrying large or sensitive results). Steps already present in state are skipped silently and emit nothing.

Bound the batch by how many keys you pass — there is no separate rate knob; every pending step goes in flight. Because it drives Nexo.concurrent, checkpoint_all needs the async gem only when something is actually pending — an all-persisted pass returns the prior values directly without touching concurrency. The same restrictions as checkpoint apply: values must be json-serializable, a step must not be named after a reserved state key (__suspend__/__approval__/__buffer_events__ — raises Nexo::Error before any step runs), and do not call suspend! inside a step (undefined — unsupported).

Known trade-off: per-step persistence, not an atomic batch. checkpoint_all is not transactional. If step B raises after step A persisted, A stays in state, B is absent, the run goes "failed", and the exception propagates through the workflow’s normal failure path (Nexo.concurrent’s “first failure re-raises, the rest stop” — it is not rescued away). A subsequent execute of the same run re-submits only the still-missing names — A is skipped, B re-runs. Do not treat a batch as all-or-nothing.

Durable agent approval — :approve

The example above suspends at an explicit suspend! the workflow author placed. The :approve mode adds the durable, cross-process sibling of :ask for the case where a run_agent-driven agent hits a permission gate mid-loop and you want that to pause the run for a human, not run unchecked and not block a worker. Declare the agent under the :approve mode:

class Scribe < Nexo::Agent
  model   ENV.fetch("NEXO_MODEL")
  sandbox :local
  permissions :approve        # every gated capability needs a human decision
end

class ApprovedWrite < Nexo::Workflow
  sandbox :local
  agent   Scribe
  def call(_p) = { content: run_agent("Write 'hi' to notes.txt").content }
end

The loop is: :approve gate with no decision → Nexo::ApprovalRequired → run_agent suspends → host renders the pending call → resume(approved:) threads the decision back through the gate.

run = ApprovedWrite.run                       # agent reaches the write gate, run suspends
run.status                                     # => "suspended"
run.state["__suspend__"]["reason"]             # => "approval: notes.txt"
run.state["__approval__"]                      # => { "capability" => "write",
                                               #      "tool" => "notes.txt", "args" => nil }
# "args" carries the tool call arguments only for an MCP-tool approval; a sandbox
# capability gate (write/shell/fetch/search) records "args" => nil — the pending
# call is identified by "capability" + "tool".

# ...a human approves — possibly in another process (resume_later for the AR store):
resumed = ApprovedWrite.resume(run.id, approved: true)
resumed.status                                 # => "done" (the gate allowed the write)

• Nexo::ApprovalRequired is a signal, distinct from Permissions::Denied: Denied means “no, adapt” (tools rescue it into {error:}); ApprovalRequired means “pause and ask a human”, so tools must not rescue it — it propagates out of the tool loop and out of Agent#prompt, where run_agent catches it.
• Undecided ⇒ suspend, approved: false ⇒ deny. The default stays safe: an unresolved approval never silently allows, and a denial on resume makes the tool return {error:} (the model adapts) — the run still finishes "done", without the gated effect, never "failed".

• Scope which actions need approval with the same ask_when predicate as :ask (aliased approve_when: for readability) — unset means every gated action needs a decision; a falsey predicate auto-allows without one:

  Nexo::Permissions.new(mode: :approve,
    approve_when: ->(cap, detail) { cap == :write && detail.to_s.start_with?("/protected") })
  

• Synchronous :ask is untouched. :ask (in-process on_ask) is still the right choice with a human at the keyboard during a synchronous run; :approve is its durable, cross-process sibling for run_later/resume_later.

Caveats — read before relying on it

• Re-entry, not replay. On resume the agent re-drives #call from the top; a non-idempotent tool call before the approval gate re-runs on resume (agent tool calls generally aren’t checkpointable). Put approval gates early, or after the expensive work is already checkpointed by the workflow.
• One approval per suspend cycle, global decision. The {approved:} answers whichever gate the re-driven agent hits first. A second gate after an approved first one simply suspends again — the next resume decides it. There is no per-tool decision granularity in v1.
• Cross-process approval needs the ActiveRecord store + ActiveJob (like all durable resume). In-process resume works with the Memory store; a Memory run does not survive the process.
• Branch depends on upstream ruby_llm. This works because ruby_llm’s tool loop lets a tool execute exception propagate out of chat.ask (verified, 1.16.0). If a future ruby_llm swallows tool exceptions, tool-triggered approval would be constrained — a genuine upstream dependency, stated plainly.

Live example

The live approval-agent flow is runnable in the repo:

NEXO_LIVE=1 NEXO_MODEL=… ruby -Ilib examples/approval_agent.rb

View examples/approval_agent.rb on GitHub →

The state column ships with fresh installs. Apps installed before this feature add it with an additive migration:

rails g nexo:state
rails db:migrate

Honest resume semantics — read this before relying on resume

Resume re-enters #call from the top — Ruby has no transparent continuation capture, so this is not replay:

• Everything outside a checkpoint re-runs on resume. Only checkpoint-guarded work is skipped (its stored result is returned). Wrap every expensive step and every side effect in a checkpoint; the idempotency of the non-checkpointed code is your responsibility.
• A crash inside a checkpoint re-runs that checkpoint on resume (at-least-once for the in-flight step) — so a checkpoint’s side effect should tolerate being retried.
• Checkpoint values must be json-serializable — they round-trip the store exactly like result/events.
• Cross-process resume needs the ActiveRecord store. A run suspended under the in-memory store resumes only in-process (which is what the test suite exercises); a run that must survive the process needs the AR store with a shared database.
• Never suspend! inside a checkpoint block (undefined — unsupported), and never name a checkpoint "__suspend__" (reserved for the suspend metadata) or "__approval__" (reserved for the pending approval call) — both are keys Nexo stores in state.

There is no distinct "resumed" status: resume re-enters execute, so a host sees the existing suspended → running → done (or suspended again) transitions over the usual nexo.workflow.status notifications. The boot reconcile_interrupted! sweep leaves "suspended" runs untouched — an intentional pause is never mistaken for an orphaned "running" run.

Next steps