Inside Opus · Part 2 of 13
This series follows a hypothetical supplier-invoice exception workflow. An Integration node retrieves invoice and purchase-order data, an Agent node extracts and interprets fields, Code nodes check arithmetic and approval thresholds, a Human Task handles exceptions, and a final Integration node records the outcome. Part 1, “A piece of work through Opus”, is the map.
Three things go wrong at once
It is late afternoon. The invoice case has reached the point where the purchase order cannot be found, so the Human node has been dispatched and the reviewer will not look at it until tomorrow morning. At the same time, the finance system’s API is returning errors for a few minutes, so the Integration node that was going to fetch the supplier’s payment history has failed on its first attempt. And the arithmetic check, which needs neither the reviewer nor the finance system, is ready to run now.
A workflow engine has to hold all three situations at once. It must let the arithmetic check finish, retry the integration without losing what has already been done, and keep the case alive overnight without treating it as an active job. If it gets any of these wrong, the failure is not dramatic. It is a case that quietly stalls, or a duplicate write when someone restarts it.
The requirement
The executor must keep the state of the case correct while its tasks run in parallel, fail, retry and wait, and it must do this for many cases at the same time. That is the durable-execution problem, and it is well understood. The question for this article is not whether Opus needed durable execution. It is why we built the executor ourselves rather than adopting one.
Dependencies are not resources
The first separation the executor makes is between what a task depends on and what it needs to run.
The workflow graph tells the execution manager which nodes are ready: a node is ready when the nodes that produce its inputs have completed. In the invoice case, the arithmetic check depends only on the extracted fields, so it is ready as soon as the Agent node returns. The payment-history lookup depends on the supplier being identified, so it is also ready. Once the missing purchase order is established, human review is independently ready as well. The final Integration node waits for the required check results and the reviewer’s decision.
Ready nodes are then dispatched to whichever worker type they need. The execution manager does not run tasks itself. It plans, dispatches and records.
Following the execution manager
A case is initiated from an application, the API or the MCP server. The execution manager is the coordination point: it evaluates readiness, produces events and applies security checks. Each node type has its own queue and its own pool of workers: Agent, Code, Integration and Human. Completed work returns through a worker output queue, and the manager uses the returned outputs to decide what is ready next.
For the invoice, the sequence looks like this:
The first Integration node is dispatched to the integration queue and picked up by an integration worker.
Its output makes the Agent node ready. That goes to the agent queue.
The Agent node’s output makes the arithmetic Code node and the payment-history Integration node ready. With the missing purchase order established, the Human is independently ready too. The manager dispatches each ready branch without making it wait for the others.
The Code node finishes. The Integration node fails and is scheduled for retry according to the node’s settings.
On the human branch, the dispatched task reaches the reviewer through a human review gateway, which also handles reviews coming back from third-party review platforms. This branch does not wait for the Code or Integration node to finish.
The queue-per-type design has a practical consequence: agent workers, code workers and integration workers can be scaled independently. This lets operators allocate capacity separately and reduces direct competition between model-heavy tasks and deterministic checks. Shared-resource contention and capacity planning still matter.
Waiting without holding a worker
The overnight wait is the part most worth explaining. The Human Task has been dispatched, and the reviewer might take fifteen hours. Holding a worker open for that time would be wasteful, and holding thousands of them for thousands of cases would be impossible.
The executor instead persists the case’s state and releases the worker. We call this hibernation. When the review comes back through the gateway, an event tells the execution manager that the Human node has completed, and the manager resumes the case from persisted state. The same mechanism handles other asynchronous returns, such as an integration that calls back later.
What hibernation saves is specific. The case occupies no task worker while it waits. It does still occupy the persistence layer, the queues that will carry its resume event, and the supporting services that keep the case addressable. “Preserved with no active worker” is an accurate description. “Zero resources” is not.
The timeline shows when work runs; the architecture below shows which services carry each hand-off.
The controls
Each configurable behaviour in the executor answers a specific operational problem.
Control: Retries and delays
The problem it answers: A dependency is temporarily unavailable, as with the finance API above.
Control: Priority
The problem it answers: Some cases matter more than others when workers are contended.
Control: Failure behaviour
The problem it answers: Whether one node’s failure should stop the case, skip a branch or route to review.
Control: Input and output type checks
The problem it answers: A node returned something the next node cannot consume.
Control: Scoped caching
The problem it answers: A node with the same input and definition has already produced an output that can be reused, within the permitted scope.
Control: Loop mode
The problem it answers: A node should run once per item of a list, in parallel.
Control: Multi-route flow
The problem it answers: A case can follow several routes without a chain of decision nodes.
Control: Change history and observability
The problem it answers: Someone needs to know what ran, in which version, and why.
The caching control deserves one caveat. Matching input and definition fingerprints makes an output a candidate for reuse; it does not establish that reuse is valid. Reuse is appropriate only when the permitted scope, current authorization, output validity and side-effect requirements allow it. The workflow designer must account for freshness, external dependencies and actions that still need to occur. Determinism alone is insufficient, and willingness to reuse an agent’s answer does not remove those conditions. Caching is a setting, not a default assumption.
The ownership decision
An established durable-orchestration engine is a serious alternative, and I would not argue against one for a team whose application fits its execution model. Those engines have years of operational hardening behind their retry semantics, their persistence and their tooling.
We chose to own the executor because the execution semantics are where our other subsystems attach. Node types map to worker pools with different resource profiles. The described authorization design includes checks at the manager and the workers. Human review is a first-class node type with its own gateway rather than an external wait. Configured node traces provide evidence for the review and learning subsystems, subject to the applicable capture, audit, access and retention settings. Each of those integrations is possible with an external engine, but each would be an adapter maintained at the seam, and the seams are where the linked-lifecycle argument from Part 1 is either kept or lost.
The cost is real. Owning the executor means owning reliability, recovery, replay and the operational tooling around them. It does not by itself make Opus faster or more resilient than the alternatives; that would be a claim about measured throughput and recovery behaviour, and I am not making it here.
Durable state also needs to be read precisely. It means a case can be resumed correctly after a failure. It does not, on its own, make the final write to the finance system safe to repeat. That is the integration’s responsibility: the node that records the outcome has to be written so that a retried write cannot post the invoice twice, and the executor’s retry behaviour assumes that it is.
Next
The executor can dispatch an Agent node and collect its output. What happens inside that node, between the input payload and the output, is a different system with its own loop. That is Part 3.