Inside Opus · Part 9 of 13
This series follows a hypothetical supplier-invoice exception workflow. Its Agent node extracts fields from invoices under a defined schema. Part 3 mentioned the node’s model assignment; this article explains where it comes from. Part 1, “A piece of work through Opus”, is the map.
The same task, thousands of times
The extraction node in the invoice workflow does one thing: read an invoice and available purchase-order data, and produce a structured record matching a fixed output schema. It will do that thing thousands of times a month. The task definition does not change between invoices. The schema does not change. The acceptable accuracy does not change.
Given that, deciding on every single invocation which model should run the task is wasted work. It is also unpredictable: a routing decision made from the shape of an individual prompt can vary from request to request, and the finance team would rather know what is running.
The requirement
For a task whose requirements are stable, the model decision should be made once, with enough evidence to trust it, recorded where the runtime can find it, and revisited when something material changes. The runtime should be fast and predictable by default and should have a way to escalate the difficult cases.
Known task structure
The reason this is possible in Opus is that the task is known before any inference happens. The workflow tells us the node’s task class (extract fields, in this case; others are classify, summarize into a schema, translate, validate, redact, escalate), its input and output schemas, its constraints, and where it sits among other nodes. The Work Knowledge Graph stores this as durable metadata on the node.
So the first step of routing is to parse the workflow graph, find the nodes that carry inference, and classify each by complexity: is it easy, bounded or complex, given its risk, reasoning depth, context size and quality requirements? Field extraction against a fixed schema is typically bounded. Interpreting an ambiguous supplier reference against a vendor master is harder, and a workflow designer might split it into a separate node for exactly that reason.
Comparing routes before scale
For each inference-bearing node, candidate models are benchmarked on representative task data. The candidates span tiers: small language models, open-weight models hosted through the cloud provider, and proprietary frontier models. The test set is built from the node’s actual task definition and schemas: generated input-output pairs, supplemented with human-reviewed traces from real cases where those exist. An output judge grades results, and the comparison covers accuracy, latency, cost and failure modes.
The approved choice must meet the task’s configured thresholds for accuracy, latency and quality. The comparison also presents model quality and full cost, including the expected cost of errors, so the team can approve a route on task-specific evidence. Within a workflow, the extraction node can have a higher accuracy bar than a summarization node, with the cost that implies. Part 10 explains how Remodel makes those recommendation trade-offs visible.
I am going to leave the evaluation methodology at that level here. How the evidence is generated, how uncertainty is expressed and when fine-tuning a smaller model is worth it are the subject of Part 10.
Recording the route
The approved route is written back into the Work Knowledge Graph as a dispatch policy attached to the node. It is a design-time artifact, in the same sense that the node’s schema is: versioned with the workflow, visible to designers, and the thing the runtime reads.
That is the central design choice. Model selection becomes part of the workflow’s definition rather than a property of each request.
Keeping the runtime simple
When a case runs, the executor retrieves the node’s approved route and sends the inference to that model tier. It does not re-run selection. The path is: retrieve the node definition and its cognitive context, retrieve the route, invoke the model through the inference queue, receive the response.
Optionally, a lightweight quality gate checks the response for confidence, anomalies, policy signals and output quality. These are inexpensive checks, not a second model comparison. If the gate raises a flag, the case follows an approved fallback path: escalation to a permitted model, or to human review. Escalation remains subject to the approved model and data-handling policy; a difficult case does not expand the allowed inference boundary. Execution telemetry (latency, cost, whether fallback occurred, quality signals) is recorded back to the graph.
The word “dynamic” fits only the escalation. The route itself is chosen at build time and applied as the default; the runtime’s dynamic behaviour is what happens when the gate fires.
The difficult tail, and change
A stable route must not become a permanent assumption. Several mechanisms prevent that.
Escalation handles the individual hard case. The extraction node’s small model handles the routine invoice; the one with a handwritten annotation and a crossed-out total goes to a frontier model permitted by its inference policy, or to the reviewer.
Monitoring watches the aggregate. If fallback frequency rises, or quality signals drop, or cost drifts, that is evidence that task data, prompts, policies or model performance have shifted.
Workflow changes invalidate the route. If a designer changes the extraction node’s schema or prompt, the route was approved for a task that no longer exists as specified, and re-benchmarking is due.
Model changes do too. New candidates appear; existing ones are updated or deprecated.
Scheduled refresh re-benchmarks on a cadence, monthly for example, regardless of whether anything visibly changed.
For customers running a single-tenant deployment, there is also the option of a fine-tuning pipeline that trains a small model against the node’s benchmark, producing a new candidate for the next refresh. Part 10 covers when that is worth doing.
The alternative, and the cost
A single capable model for everything is the simplest arrangement. Nothing to benchmark, nothing to maintain, and for a workflow with few invocations it may cost less in total than the evaluation would.
Dynamic selection on every request is the right tool when requests are heterogeneous or poorly specified: an open-ended assistant receiving arbitrary questions cannot know the task class in advance. It pays per request for the flexibility.
Opus’s build-time approach applies when the task is known and repeated, which is the normal condition for a workflow node. It moves the selection work out of the execution path and makes the decision inspectable. The costs are up-front evaluation for every inference-bearing node, and ongoing maintenance of routes as workflows, models and data change. Whether the saving on production calls outweighs the evaluation cost depends on volume, and it is a measurement rather than a given: per-node win rate against a fixed frontier baseline, fallback rate, and cost saved net of the benchmarking are the numbers that settle it.
Next
The route is only as good as the evidence that approved it. What makes that evidence strong enough, how is uncertainty expressed, and when does a fine-tuned small model belong among the candidates? Part 10 is about Remodel.