FIELD · Capacity and platforms · 4 min read
Recover durable work after a regional outage
Restore task state, identity, artifacts and external-effect reconciliation.
Arunank Sharan · Kuzushi Labs ·
In this article
This worked application uses explicit planning assumptions to demonstrate FIELD. Quantities below are inputs, calculated implications or proposed targets; they are not reported customer results. The engagement decision remains conditional on the evidence described.
Planning basis and engagement boundary
| Item | Basis for this application |
|---|---|
| Demand and context | Five tasks/s; ten-minute interruption; eight tasks/s processing capacity after recovery |
| First delivery boundary | Restore task state, identity, artifacts and external-effect reconciliation |
| Acceptance to agree and test | Timed regional drill covers dependencies and real business state |
| Exclusion or constraint | Application replicas alone are not evidence of regional recovery. |
F — Frame the outcome
Restore useful agent work after regional failure while preserving the agreed acknowledged-state and external-action contract.
Value measure: Verified task recovery time, acknowledged-state loss and unresolved action backlog.
I — Investigate the workflow
Separate RPO/RTO for state, intents, receipts and artifacts; determine whether old-region workers can still reach external systems.
First evidence task: Leave an old worker connected to the tool while transferring ownership to the recovery region.
E — Engineer the solution
Use replication trade-offs, fenced authority, operation reconciliation and surviving execution capacity; calculate backlog drain after recovery.
The implementation assessment below makes the authority, workload, resource and failure assumptions concrete. Its numbers are calculations from the stated scenario, not a vendor performance guarantee.
Contract
Set RPO/RTO separately for task state, action intent/results, artifacts and analytics. Decide whether zero loss of acknowledged actions is mandatory and whether cross-region synchronous coordination is acceptable. Define permitted data locations and surviving model/tool capacity.
Implementation plan
Use an authoritative replicated task/action store with a supported failover protocol; workers acquire fenced ownership. Persist effect intent before execution and use stable operation keys with external services. Store artifacts durably and keep a replay/reconciliation path. The scheduler activates only in a region with valid authority.
Failure semantics
With asynchronous replication, recent acknowledged local state may be absent after region loss; that is an RPO exposure, not solved by starting workers elsewhere. Synchronous quorum/replication can strengthen acknowledged durability at latency/partition-availability cost. Fencing must reach the protected resource or effect mediator; a stale old-region worker may still reach an external tool.
Sizing
If arrival is 5 tasks/s and recovery takes 10 minutes, 3,000 tasks accumulate if accepted durably. At resumed capacity 8/s with continuing 5/s arrivals, ideal drain takes 1,000s≈16m40s. If a cold model fleet takes additional minutes to load, that extends usable recovery. Capacity for state recovery alone is insufficient.
Partition perturbation
Prevent both regions from committing authoritative transitions independently. Clients retry the same logical operation, not a new action. For remote effects whose status is unknown, reconcile provider state before resume; mark unresolved cases for review. Failback is another ownership/data migration with validation, not simply switching DNS back.
Drill
In isolation, partition rather than merely stop the old region; leave a stale worker able to contact the test tool. Verify no conflicting commits, no duplicate effect within the stated contract, bounded backlog and measured RPO/RTO. Recover from backup as well as replica failover.
L — Launch with evidence
Drill a partition, not just a stopped region. Measure useful task recovery, acknowledged loss and duplicate/unknown effects against the agreed scope.
Critical decision condition: Two regions acting with conflicting authority or unverified external outcomes block resume.
Attach raw artifacts, the tested configuration, population, denominator, date and limitations to the release record. Missing or stale evidence holds the dependent decision. A favorable value forecast cannot offset a failed critical condition.
D — Develop ownership and reuse
Service and business owners accept recovery/degraded behavior; customer operators rehearse failback and reconciliation.
Candidate reusable asset: Region-failover action reconciliation contract.
Commercial decision and next commitment
Accountable decision owner: Service incident owner with affected business owners.
The initial commitment is the bounded journey above, with the stated exclusions. Confirm access, customer staffing and acceptance ownership during discovery; price and schedule unresolved dependencies explicitly. A new source, tenant class, language, action or operating envelope requires an impact review rather than silent scope growth.
Approve only the next exposure level supported by the evidence. The next useful investment is: leave an old worker connected to the tool while transferring ownership to the recovery region.
Method and references
Apply the FIELD method, economic model and evidence rules. Methodological and technical influences are listed in the source register.