Inside Opus · Part 10 of 13
This series follows a hypothetical supplier-invoice exception workflow. Part 9 explained that the extraction node’s model is chosen at build time and recorded as a route. This article is about the evaluation that justifies the choice. Part 1, “A piece of work through Opus”, is the map.
Two candidates, two kinds of wrong
The extraction node is being evaluated. In this hypothetical comparison, one candidate, a small model, produces perfectly valid structured output every time: every field present, every type correct, the schema satisfied. It also, in something like one case in ten, records the service-period start as the invoice date. A second candidate, a frontier model, also produces schema-conforming outputs, gets the dates right almost always, and costs many times more per call.
Schema conformance alone cannot distinguish the two outputs: it misses the date error. A recommendation that only reports “accuracy” prefers the second and ignores the cost. A useful recommendation has to expose both differences, say how confident it is in each, and let the team decide how much a wrong date is worth avoiding.
The requirement
The evaluation must be specific to the actual task, must separate checks that can be made deterministically from judgements of quality, must express its uncertainty, must present cost and quality together, and must be reproducible. When it recommends training a smaller model, it must include the full cost of doing so. And it must protect itself from its own shortcuts.
Profile the actual task
Remodel sits on each Agent node and starts by reading what the node actually is: its prompts, its input and output schemas, its knowledge domain, and the policies and constraints that apply. This is the task profiler, and it means the evaluation is of this extraction task, with these fields and this schema, not of “extraction” in general.
That specificity is the point. A generic benchmark ranking tells you which model is better at a public test suite. It does not tell you which model confuses the service period with the invoice date on your suppliers’ documents.
Choose candidates and cases
The candidate selector filters a catalogue of frontier models, open-weight models, off-the-shelf small models and fine-tuned checkpoints, taking into account which models this user, workspace and organization have preferred before.
Evaluation cases come from two sources. Generated input-output pairs are produced from the task definition and schema. Human-reviewed traces from real cases are added where they exist. The second source is the more valuable one: a reviewer’s correction of the invoice date is exactly the kind of case that exposes the first candidate’s weakness. The first source is what makes evaluation possible before the workflow has much history.
Structural checks and quality judgements are different things
The auto-evaluation harness runs each candidate on the cases and grades the output in two separate ways.
Deterministic validators check conditions that can be specified exactly: does the output conform to the schema, are the required fields present, are the types right, do the line items sum to the total? If trusted reference labels are available, exact comparisons can also check meaning-bearing fields, such as whether the extracted invoice date matches the expected date. These checks give definite answers to the specified conditions.
A model judge compares each candidate’s output against a reference produced by a stronger teacher model. It assesses qualities that are not fully captured by exact rules or available trusted labels: whether a supplier has been resolved correctly, or an ambiguous reference interpreted sensibly. Checking the invoice date is a semantic evaluation; it can use an exact comparison with a trusted label, a model judge or human review, depending on the evidence available.
Neither schema compliance nor agreement with a model-generated reference establishes correctness in any universal sense. Valid formatting is not accuracy; the first candidate proves that. And agreement with a reference is only as good as the reference. If the teacher model also confuses the dates on a particular layout, the judge will reward candidates that make the same mistake. Teacher outputs and judge preferences can carry errors into the evaluation, and the human-reviewed traces are the corrective.
Make the recommendation interpretable
Before any mathematics, here is the recommendation in plain terms. For each candidate, Remodel reports how well it fits the task and how sure it is of that figure. It reports what the candidate costs per call. It combines those with two things the team supplies: how much a wrong answer costs them (a misposted invoice has a real cost, in rework and in supplier relations), and how much they are willing to pay for a unit of quality improvement. From that it ranks the candidates and names a few useful picks: the recommended one, the best quality, the best value and the cheapest viable.
For the two candidates above, the outcome depends on the numbers the team supplies. If a wrong date costs little because a reviewer catches it before it affects payment, the small model may be recommended. If wrong dates flow through to payment, the frontier model or a fine-tuned alternative may be.
Technical inset. Task fit is reported with bootstrap confidence intervals, so limited evidence is visible in the uncertainty estimate. More representative cases generally improve precision; sample count alone does not determine interval width or remove dataset bias. Ranking is on a probabilistic Pareto frontier of quality against coupled cost, where coupled cost includes the per-call price and the expected cost of being wrong. The willingness-to-pay parameter is the exchange rate between the two axes. The engine is a pure function of these inputs, so the same inputs always give the same ranking and the ranking can be explained by pointing at them.
When to fine-tune
Remodel offers a path to fine-tune a small model for the task. Here, fine-tuning means a customer-directed training run for an approved task, using data and model outputs under terms that permit that use. It does not confer permission for a provider to train its general models on customer data; that is governed separately by the applicable agreement and configuration. The conditions under which task-specific fine-tuning is worth doing are narrower than the offer might suggest.
The task should be bounded and recurring: field extraction against a fixed schema qualifies; open-ended interpretation does not. There must be enough suitable data. There must be enough usage that a lower per-call cost adds up to something. And the comparison must include everything: training compute, evaluation of the result, serving the checkpoint, and maintaining it when the task or the data drifts. A fine-tuned model that saves a fraction of a cent per call on a workflow that runs a few hundred times a month is not a saving.
When those conditions hold, the evaluation run does double duty. The teacher model’s reference outputs, used under terms that permit it, become training labels. A held-back split of the cases becomes the test set. The Fine-Tuning Studio trains the candidate, evaluates it on the held-back set, and registers the checkpoint, which re-enters the candidate pool as a new small-model option for the next evaluation.
Protect the evaluation, close the loop
Two disciplines keep this honest.
Datasets are versioned, and training and held-out evaluation are kept separate. A checkpoint that was trained on its own test set will report a task fit it does not have.
A new checkpoint gets no shortcut. It goes through the same acceptance process as any other candidate before the route in Part 9 is updated to use it.
Implementation inset. One deployment option places Remodel inside the customer’s own account on a governed data platform. In that design, evaluation datasets, graded results and telemetry are held in application schemas within the account. Provider access depends on the platform’s permissions, the application configuration and any customer-approved support arrangement; account placement alone does not establish an access guarantee. Evaluation compute runs in the account, while any external teacher-model or fine-tuning call must follow the customer’s approved egress and data-use policy. The platform’s zero-copy cloning can capture a run’s starting dataset without an initial physical copy of all its data. A clone is still a writable, independent object: reproducibility requires the run and other actors to leave it unchanged, enforced through the relevant access controls. This is a deployment option whose permissions and operating controls must be specified for the customer.
The alternative, and the cost
Generic benchmark rankings, a manually chosen model or further prompt work are adequate starting points, and for a low-volume workflow they may be the right stopping point too.
Remodel makes the task-specific evidence and the improvement process explicit. The costs are data preparation, grading, compute for evaluation and training, and calibration: the cost-of-error and willingness-to-pay defaults have to be fitted to outcomes over time, and judge agreement with human reviewers has to be checked. A recommendation is only as trustworthy as its intervals, the lineage of the dataset behind it, and, for a fine-tuned checkpoint, its held-out results measured rather than assumed.
Next
A better model, a better prompt or a better agent is still a change to a production component that several workflows depend on. How does it become available for use, and who decides where? Part 11.