Durable workers: the company must survive a process restart

A customer does not care whether the agent process restarted. If a quote was approved yesterday, the company should still know whether it was sent today. If the worker disappears after asking a provider to send it, starting another worker must not create a second quote email.

This is where an agent prototype becomes a workflow system. The model decides what work to propose. A durable execution layer records the obligations, schedules attempts and preserves the evidence needed to resume. I am treating that layer as a distinct part of this internal R&D platform.

A job is an obligation with a lifecycle

Use a persisted job identity, tenant, business subject, action identity, state, next eligible time and attempt metadata. Distinguish READY, LEASED, WAITING_EXTERNAL, WAITING_REVIEW, DONE and NEEDS_ATTENTION. The public lab implements the smaller READY/RUNNING/DONE subset, with lease fields on a running job, so the concurrency contract stays easy to inspect.

Do not encode waiting for a customer response as a worker sleeping for three days. Persist the waiting condition and release the worker. A reply, a timer or an operator decision creates another eligible step. Long waits should consume storage, not a running model session.

from company_lab.store import Store, Conflict

store = Store()
store.enqueue("studio", "send-quote-017")
epoch_a = store.claim("studio", "send-quote-017", "worker-a", now=100, ttl=30)
assert epoch_a == 1
assert store.claim("studio", "send-quote-017", "worker-b", now=110) is None
epoch_b = store.claim("studio", "send-quote-017", "worker-b", now=131)
assert epoch_b == 2
try:
    store.finish("studio", "send-quote-017", "worker-a", epoch_a, now=132)
    raise AssertionError("old worker should have been fenced out")
except Conflict:
    pass
store.finish("studio", "send-quote-017", "worker-b", epoch_b, now=132)

A lease limits how long a worker owns a job. The incrementing epoch is a fencing token: an old worker cannot mark the new worker's job complete. In a real service, use a consistent server-side clock for lease comparisons and renew leases only while progress is healthy.

Fencing the queue does not fence the outside world

The old worker might still have an HTTP request in flight. A database fence stops stale writes to the job table; it cannot recall an email or cancel a provider operation that has already been accepted. External dispatch therefore needs its own stable operation identity, request digest and receipt state.

Persist an operation intent before dispatch. If the provider times out, retain UNKNOWN rather than pretending the operation failed. Reconcile using the provider's supported lookup or idempotency mechanism. A provider without those capabilities requires a more conservative recovery path, possibly human investigation.

The public connector simulator demonstrates this distinction. Its idempotency behaviour is deliberately explicit. It is not a promise that every email, banking or CRM API has the same semantics.

Claim work without making every worker wait

For a PostgreSQL-backed queue, a short transaction can claim an eligible job using FOR UPDATE SKIP LOCKED. This is a proposed production schema; the public lab uses SQLite.

WITH candidate AS (
  SELECT id FROM jobs
  WHERE state = 'READY' AND next_attempt_at <= now()
  ORDER BY priority DESC, next_attempt_at, id
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
UPDATE jobs AS j
SET state = 'LEASED', lease_epoch = lease_epoch + 1,
    lease_until = now() + interval '30 seconds', worker_id = :worker
FROM candidate AS c
WHERE j.id = c.id
RETURNING j.*;

A separate bounded recovery query returns expired leases to eligible work. Include tenant scheduling policy so one busy account cannot consume the entire fleet. SKIP LOCKED is useful for queue consumers because they can take another row; it is not a general solution for obtaining a consistent business report. PostgreSQL SELECT reference.

Use an outbox to connect state and scheduling

Suppose quote acceptance updates the opportunity but the process crashes before creating the delivery job. Without a durable handoff, the company has accepted work that nobody will start. Write the state change and an outbox event in the same database transaction. A relay can publish that event repeatedly; consumers deduplicate by business identity.

The outbox does not remove the need for retries. It makes the event recoverable. Monitor the age of the oldest unpublished event and the number of attempts, rather than only checking whether the relay process is alive. A running process can still be stuck on one malformed record.

Timers need business meaning

A payment reminder should be scheduled against an invoice and its current due date. If the terms change, the timer must re-read that record. Do not retain an old due date inside a generated paragraph and treat it as an enduring instruction.

Store scheduled_for, reason, subject_version and cancellation_condition. When the timer fires, refresh the balance, suppression state and customer context. A scheduled action is permission to reconsider the task, not permission to ignore intervening facts.

Use exponential backoff with jitter for transient failures, a provider-specific ceiling and a maximum elapsed recovery window. Permanent validation errors go to review immediately. Never retry authentication failures indefinitely while quietly growing the queue.

Where a workflow engine fits

Temporal's workflow model persists execution history and reconstructs workflow state through deterministic replay; external operations belong in activities. That is useful when a business process waits across days and needs explicit recovery. A graph framework's checkpointing solves a related but narrower problem: preserving agent execution state. Neither substitutes for the business approval and receipt contracts. Temporal workflows, LangGraph persistence.

I would choose the durable engine after the first complete loop establishes its failure modes. The acceptance test is concrete: terminate the worker after provider acceptance, restart it, and observe one customer-visible effect with a reconciled receipt. A successful restart that sends twice is still a failed business operation.