Durable Execution for AI Agents: How to Make Long‑Running Work Reliable

Short answer: Durable execution for AI agents means the agent’s plan and side effects survive restarts, network splits, flaky tools, and human wait states without losing correctness. We structure the agent’s loop as a durable workflow, isolate side effects into idempotent tools, checkpoint state at every decision, and use replay to debug and evaluate. Temporal‑style runtimes or managed workflow services provide timers, signals, and deterministic orchestration that notebooks and simple queues lack. Human‑in‑the‑loop becomes a first‑class wait state with SLAs, not an ad‑hoc pause. This is how we close the hype‑vs‑production gap: agents that actually finish the job, days later, exactly once.

Durable execution for AI agents: definition and scope

Most agent demos assume a fast, uninterrupted loop: reason, call a tool, reason again, done. Production agents do not live in that world. They coordinate long‑running work across APIs, users, and systems with variable latency and failure modes.

Durable execution means three concrete guarantees:

• Progress survives failure. If a worker crashes or a deploy rolls out, the agent resumes from the last safe checkpoint without repeating unintended side effects.
• Exactly‑once intent. External effects (emails, tickets, payments, file writes) happen once, even if we retry internally many times.
• Auditability and replay. We can reconstruct what the agent decided, which tools it called, what came back, and why it picked the next step.

For AI agents, these guarantees sit on top of nondeterministic components (LLMs, third‑party APIs). We do not try to make the LLM deterministic. We make the orchestration deterministic and we record nondeterministic outputs as events. That gives us reliable progress and honest forensics.

The failure modes that break naive agents

When teams take a prototype to production, the same patterns of breakage recur. We design durable execution to neutralize them.

• Process death and deployments. Ephemeral workers die. Without persisted state and timers, a multi‑hour agent quietly vanishes mid‑run.
• Network splits and flaky APIs. Tool calls take minutes, time out, succeed late, or succeed twice. Naive retries create duplicates and inconsistent state.
• LLM nondeterminism. Sampling produces different tool calls on retry, making post‑failure resumption unsafe unless we capture decisions.
• Human latency. Approval steps pause for hours or days. If the agent’s state lives in RAM, it’s gone by the time someone clicks Approve.
• Clock and scheduling drift. Cron‑based pauses and sleeps do not persist across restarts and lack durable handles; timers get lost.
• Schema drift. Tool payloads evolve. Without versioned state and migrations, old runs become unreadable and unreplayable.
• Partial side effects. A step writes data to one system and fails to write to another. There is no compensation and no ledger to reconcile.

Durable execution treats each of these as first‑class concerns: time, idempotency, human wait states, and compensations all live in the orchestrator, not in scattered try/catch blocks.

Patterns that make agents durable

The goal is simple: make the orchestration deterministic, make side effects idempotent, and make time durable. Here are the patterns we lean on.

Sagas and compensating actions

Agents rarely have ACID transactions across services. We use the Saga pattern: break work into steps with forward actions and compensations. If step N+1 fails irrecoverably, we run compensations for steps 1..N to restore a consistent business state.

• Forward actions: create ticket, write record, send email.
• Compensations: close/revert ticket, delete record or flag as void, send correction email.
• Ledger: store which compensations are still eligible; compensations must be themselves idempotent.

Sagas keep the agent’s external world consistent even when tool calls misfire or human approvals deny a change after partial execution.

Idempotent tools and deduplication

We wrap every side effect behind a tool interface that accepts an idempotency key and returns a stable external ID. If our orchestrator retries the tool, the provider either returns the original success or does nothing.

• Idempotency keys per logical action, not per attempt.
• Natural keys where possible (e.g., “invoice for order-123”).
• Dedup caches on our side when upstream services lack idempotency; we map our logical keys to provider responses.
• Versioned tools so we can evolve payloads without corrupting replay of older runs.

Idempotency is the difference between safe retries and duplicate side effects. We never ship without it.

Checkpointing state and decisions

We persist the agent’s state after every meaningful step: the plan, the selected tool, the arguments, the raw and parsed outputs, and any human inputs. We do not rely on the LLM’s scratchpad to recover context.

• Run record: a structured log of steps with timestamps, prompts, responses, tool calls, and artifacts.
• Step outcomes: success, retry scheduled, compensation scheduled, waiting for signal, aborted.
• Data separation: store PII or secrets outside the general transcript and fetch by reference when needed; encrypt at rest.
• Schema version: capture the state schema version at each step to support migrations.

With checkpoints, a restart resumes from the last committed decision. We avoid re‑asking the LLM to decide the same thing twice unless we explicitly want to.

Replay determinism and event sourcing

We treat nondeterministic outputs as events. On a cold replay, we feed the recorded events back into the workflow rather than regenerating them. That gives us step‑by‑step determinism without freezing the model.

• Record once every nondeterministic outcome: LLM outputs, random IDs, time‑derived values, external API responses.
• Pure orchestration code that uses recorded events during replay; the same code executes in live runs, reading from an event log.
• Selective re‑execution when we intentionally want a different outcome (e.g., a policy now forbids a previously allowed tool). We mark the event invalid and continue from that point.

Replay lets us debug with surgical precision, run offline evals over real histories, and reproduce a production bug without guessing at prompts or timing.

Durable time: timers, heartbeats, and leases

Async work needs a durable notion of time and ownership. We do not sleep threads; we set timers and heartbeats in the orchestrator.

• Durable timers wake the workflow after minutes, hours, or days. Timers survive restarts and deployments.
• Heartbeats from activities (tool wrappers) let us cancel or retry long calls and detect stuck workers.
• Leases on external resources prevent two workers from acting on the same item concurrently.

With durable time, the agent can pause for a user reply or a vendor backoff window without tying up compute or losing its place.

Human‑in‑the‑loop as a first‑class signal

Approvals and clarifications are not exceptions. We model them as signals that unblock a waiting state with SLAs and escalation paths.

• Wait states with deadlines. If the deadline passes, auto‑escalate, auto‑deny, or route to fallback.
• Artifact diffs for review (e.g., proposed PR changes, email drafts, payment details) to enable quick, safe approvals.
• Access control on who can approve which actions; approvals become signed events in the run record.

Durable human steps prevent the classic stall: an agent waiting forever for a button click that never comes.

Backoff, circuit breakers, and budgets

When tools misbehave, we fail fast and protect the system.

• Exponential backoff with jitter and ceilings per tool.
• Circuit breakers to open when error rates spike; route to queued review flows.
• Error budgets per agent capability. If the budget is exhausted, we degrade gracefully or disable risky actions.

These guard the agent’s reliability and contain blast radius during vendor outages or schema changes.

Choosing an execution substrate

We need a runtime that persists workflow state, provides durable timers and signals, and executes orchestration code deterministically across failures. Several options exist; the right choice depends on your stack and operational constraints.

• Temporal: code‑first durable workflows with strong guarantees around determinism, timers, signals, and replay. Great fit for long‑running agents. Requires operating the service or using a managed offering, plus discipline around workflow code evolution.
• AWS Step Functions: managed workflows with service integrations. Good for AWS‑centric stacks and shorter logical flows. Less natural for iterative LLM tool loops, but viable with careful state design and external activities.
• Azure Durable Functions / GCP Workflows: language‑integrated durable patterns in their ecosystems. Similar trade‑offs to Step Functions.
• Zeebe/Camunda: BPMN‑centric engine with strong process modeling and scaling. Useful when you already model business processes visually and can map agent states to BPMN.
• Argo Workflows: container‑oriented DAGs on Kubernetes. Best for batch and data tasks; can support agent steps with custom controllers but lacks out‑of‑the‑box deterministic replay semantics.
• Framework‑embedded persistence: some agent frameworks and libraries provide graph execution and checkpointing. Useful to start, but verify guarantees around timers, idempotency, and cross‑version replay before betting on them.

We avoid ad‑hoc mixes of queues, cron jobs, and a database table of “runs” unless we are ready to rebuild half a workflow engine. It looks easy, then quietly consumes months in edge cases: lost timers, double deliveries, inconsistent retries, and untraceable deadlocks.

The decision criteria we apply:

• Language fit with your core services and team skills.
• Deterministic replay model and migration story for workflow code changes.
• Timers and signals that survive restarts and provide strong delivery guarantees.
• Operational posture: managed service vs self‑hosted; latency, throughput, and observability.
• Integration friction with your secrets, identity, and VPC boundaries.

Migration path: from prototype to durable production

Most teams already have a working prototype agent. Good. We keep the behavior but change the skeleton so it survives real world timing and failures. This is the path we use.

1. Freeze the business‑critical path. Write down the minimal set of actions the agent must perform to deliver value. Everything else can be deferred.
2. Externalize side effects into tools. Wrap every call that changes the world (tickets, emails, payments, repositories) behind a tool boundary with idempotency keys.
3. Add a run record. Introduce a structured log for prompts, tool calls, outputs, and human inputs. Keep PII and secrets referenced, not inlined.
4. Introduce a durable orchestrator. Move the loop (plan‑act‑observe) into a workflow engine that supports timers and signals. Keep tools as activities that can retry and heartbeat.
5. Checkpoint after each decision. Persist plan, chosen tool, arguments, and outputs. Resume from checkpoints on failure rather than re‑asking the LLM.
6. Model human steps as signals. Replace ad‑hoc blocking calls with durable waits, deadlines, and escalation rules. Expose artifacts for review.
7. Make tools idempotent and safe by default. Add idempotency keys, timeouts, backoff, and circuit breakers. Log stable external IDs for audit.
8. Wire observability. Add trace IDs across the workflow, LLM calls, and tools. Emit metrics for success rate, retries, wait durations, and compensation rate.
9. Enable replay and sandboxes. Store nondeterministic events. Build a replay harness that can run a production history offline against the latest code.
10. Define SLAs and error budgets. Establish targets for latency, completion rate, and maximum duplicate side effects. Automate degradation when budgets are hit.
11. Plan for schema and code evolution. Version state, tools, and prompts. Write migration shims for in‑flight runs before deploying incompatible changes.

Common pitfalls to avoid:

• Skipping idempotency because a provider “rarely” duplicates requests.
• Storing everything in the transcript instead of modeling structured state and artifacts.
• Using sleeps for hours‑long waits instead of durable timers.
• Bundling tool calls inside the LLM step so you cannot retry or compensate separately.
• Hardcoding prompt changes without versioning, making replay impossible and audits inconclusive.

Run the migration in slices. Start with one capability end‑to‑end, prove durability, then expand to adjacent flows.

How Moai Team approaches this

We build agents that reach production and stay there. Durable execution is not an afterthought in our projects; it is where we spend engineering attention after the first demo clicks. Our approach is pragmatic: keep what works, replace the skeleton under it, and ship with safety rails on day one.

• Scoping to the critical path. We cut the capability to the bone so we can design idempotent tools and compensations for the few actions that matter.
• Harness engineering. We formalize the agent’s loop as a deterministic workflow with state checkpoints, timers, and signals. We avoid hidden state in memory and isolate side effects.
• Tool hardening. We wrap external systems with idempotency, retry policies, backoff, and circuit breakers. We add observability and stable external IDs for auditability.
• Replay‑first debugging and evals. We record nondeterministic outcomes and build a replay harness to reproduce issues and measure upgrades before we ship them.
• Human‑in‑the‑loop by design. We make approvals and clarifications durable, with deadlines and escalation. We expose compact artifacts for fast review.
• Governance and SLAs. We define error budgets, rollout gates, and degradation modes. If a budget is hit, the agent backs off or flips to a safer path automatically.
• Integration depth. We connect to the actual systems of record and move data safely across identity, secrets, and compliance boundaries. We do not ship shadow databases.

The result is simple to state and hard to fake: an agent that completes long‑running work exactly once, with a paper trail and a rollback plan. That is the gap between a demo and a production system. That is the gap we close.

Frequently Asked Questions

What is the difference between durable execution and a queue with retries?

A queue with retries replays an entire task when something fails. Durable execution persists the state machine of the work, so we resume from the last safe step, not from the beginning. It also gives you durable timers, human signals, and exactly‑once semantics for side effects through idempotent tools. Queues are great for independent, short tasks. Long‑running agents need a workflow runtime.

Do I need Temporal to get durable execution for AI agents?

No. Temporal is a strong fit, but you can deliver durable execution with managed options like AWS Step Functions, Azure Durable Functions, or GCP Workflows, and even with BPM engines like Zeebe/Camunda. The key is to meet the requirements: persisted state, durable timers and signals, deterministic orchestration, and a replay story. We choose the substrate that fits your stack and ops posture.

How do you keep determinism when an LLM is nondeterministic?

We make the orchestrator deterministic and treat every nondeterministic outcome as a recorded event. On replay, we feed the recorded events back instead of regenerating them. Prompt text and tool schemas are versioned. That lets us reproduce any past run exactly while still allowing the live system to sample freely.

How do you add human approval steps without stalling the agent?

Model approvals as durable wait states with deadlines. Use timers to auto‑escalate or choose a safe default when the deadline passes. Package artifacts for fast review and constrain who can approve which actions. The agent sleeps in storage, not on a thread, and wakes when a signal arrives or a timer fires.

What does “exactly‑once” really mean for external systems?

We approximate exactly‑once by designing idempotent tools and deduplication. Each logical action uses a stable idempotency key and we store the resulting external ID. If a retry happens, the provider returns the same result or we detect the prior success and skip. This eliminates duplicate emails, double tickets, and repeated writes.

Can we bolt durable execution onto our existing prototype without a rewrite?

Often yes. Start by extracting side effects into idempotent tools, adding a run record, and moving the orchestration loop into a workflow. Keep your prompts and tool schemas but version them. Migrate one capability end‑to‑end, learn from it, then expand.

Have a prototype that stalls in the wild or a roadmap that needs durable execution? Talk to us at Moai Team — contacts. We will map the shortest path from demo to production.