Inside Opus · Part 13 of 13
This series follows a hypothetical supplier-invoice exception workflow. Part 11 followed a component to activation; Part 12 described the environment it runs in. This closing article is about how that environment itself is changed. Part 1, “A piece of work through Opus”, is the map.
One release, two owners
A release of Opus is ready. It has to reach an environment we operate for one customer on AWS, and an environment a second customer’s platform team operates for themselves on Azure. The first customer expects us to deploy it. The second customer’s change-management process requires their own engineer to review and promote anything that reaches production, on their schedule, and they do not want us to have hands on their cluster.
Both are legitimate. The engineering question is how to serve both from one release process, so that the two environments do not drift into two products.
The requirement
Releases must be reviewable and repeatable across clouds and across ownership models. Changes to the platform (clusters, databases, networking) and changes to the workloads (Opus services) must be able to move at different rates under different owners. And the customer who operates their own environment must get the same release as everyone else, delivered in a form their process can consume.
Two kinds of change
The first separation is between platform and workload.
Platform changes are infrastructure: a new node pool, a database version, a network rule, the cluster tooling. They change rarely, carry high blast radius, and are owned by platform engineers.
Workload changes are the Opus services themselves: a new version of the execution manager, a fix in the collaboration service. They change often and are owned by the product release process.
Conflating them means every service release becomes an infrastructure change review, or every infrastructure change ships on a service cadence. Separating them means each has its own repository, its own owners and its own approval.
Desired state, declared
The pattern described here is GitOps. Configuration managed by this process declares the intended environment state in a source-controlled repository. A controller in the cluster continuously compares the live environment with the declaration and attempts to reconcile differences. Routine desired-state changes are reviewed commits. Emergency access outside that path is an exception that must be justified and recorded under the deployment’s operating policy.
This gives three things. The repository records configuration changes, while the associated review records identify approvals. The controller attempts to make the environment converge on the declaration and reports reconciliation status. And the same release pattern can be applied to another cluster using its environment-specific configuration, which is what makes the two-customer problem tractable.
The pattern
Our deployment pattern uses two open-source controllers in this arrangement.
FluxCD is the reconciliation controller. It watches configured Git repositories, applies the declared resources to the relevant Kubernetes clusters, and reports the outcome of its attempts to reconcile desired state.
Crossplane provides declarative cloud-resource provisioning. Storage, databases, networking, load balancers and managed services are defined as Kubernetes-managed resources, so the same declaration style that describes a workload also describes the cloud infrastructure it needs, on AWS or Azure.
The repositories reflect the platform-versus-workload split. A platform infrastructure repository holds cluster bootstrap, tooling and Crossplane resource definitions. A workloads repository holds the application deployments, with one workload definition per environment.
The managed path
In the managed arrangement described here, our platform engineers commit to the infrastructure repository. FluxCD, bootstrapped into the customer’s cluster, monitors that repository and deploys the cluster tooling. Crossplane provisions the cloud resources the declaration calls for. Workload releases are committed to the workloads repository, and FluxCD deploys them into the cluster. Our engineers can bootstrap, provision, deploy and monitor the environment from a central operations layer. Routine desired-state changes are reviewable commits that the customer can inspect; monitoring and other operational actions are separate from those commits.
Each tenant can have its own repository, its own cloud account or subscription, its own cluster and its own Crossplane-managed resources, all following this same pattern. That is the isolation option mentioned in Part 12.
The customer-operated path
In the customer-operated arrangement described here, the path changes hands at a specific point. We do not commit to their repository and we do not reconcile their cluster. Instead, each Opus release is supplied to the customer as a release package. Their platform engineer reviews it, commits the desired configuration to their own GitOps repository, and promotes it through their environments using their own process and their own GitOps tooling, which may be FluxCD or another controller of their choosing. Crossplane, if they use it, provisions resources in their own account.
What we see of that environment is governed by the arrangement the customer permits. Central visibility is preserved where they allow it; where they do not, the customer retains full operational control, cloud ownership and their tenancy boundary, and deployment authority remains with the customer.
The two paths deliver the same release. They differ in who commits, who approves and who reconciles.
Back to the invoice
I want to end where the series began, with the invoice, and trace one improvement all the way through.
In Part 6, a reviewer corrected an invoice date, and the correction became a workspace-scoped learning. That improvement needed no Opus platform software release: it passed the learning promotion gate, lives in scoped memory and reached later agents through the cognitive layer. Continuing the hypothetical example, suppose a route refresh finds a cheaper model that meets the task thresholds and is approved after comparing quality and full cost, using the approach in Part 9. That also needs no Opus platform software release: the route is a design-time artifact in the Work Knowledge Graph, updated through task-specific evaluation, versioning and approval.
Suppose the evaluation described in Part 10 also supports a fine-tuned small model for the extraction task. In Part 11’s hypothetical continuation, an engineer builds a new extractor version around it, takes it through source control and the pipeline, publishes it, and the exception workflow’s team activates it. That is a component release, but not an environment release; the platform does not change.
Now suppose the fine-tuned model is to be served inside the trust boundary, because the customer has adopted the private-inference option from Part 8. That requires GPU capacity that the environment does not have. This is a platform change: a new node pool declared in the infrastructure repository. In the managed environment, our engineers use Crossplane and FluxCD to provision and reconcile it. In the customer-operated environment, the customer’s engineers review and promote the change through their chosen infrastructure and GitOps tooling. Then the model-serving workload is released through the workloads path, the checkpoint is registered, and an evaluated, approved route is updated to use it.
Four improvements, four different mechanisms, and only the last one required a change to the cloud infrastructure and model-serving workload in this example. Knowing which kind of change you are making is most of the discipline.
One distinction to hold onto: reverting a commit requests the previous declared configuration, and the controller attempts to reconcile the environment to it. A resource transition can fail or be unsupported, so rollback may require planning and operator intervention. Restoring data or reversing a database migration is a separate, planned operation; changing the declaration does not perform either one.
The alternative, and the cost
Manual deployment scripts are adequate for a small number of environments with one owner, and many products start there. Fully centralized release control, where the vendor deploys everywhere, is simpler still when customers accept it.
The GitOps pattern gives a common, reviewable operating model across ownership boundaries, which is what the two-customer problem requires. It brings its own responsibilities: controllers to operate and upgrade, configuration to keep coherent across repositories, access to the repositories to manage carefully, and migrations to plan outside the reconciliation loop.
What the series has established
In the architecture described here, people define work as a workflow, with people and design agents proposing changes into one authoritative version. The system executes that work under explicit constraints: durable execution across parallel tasks and long waits, authorization that follows each action, and inference that respects the data boundary. Execution produces evidence. Reviewed learnings pass the promotion gate into scoped knowledge; task-specific model decisions follow their own evaluation and approval process. Components carry those improvements as versioned assets that a workspace chooses to activate, and controlled delivery brings the environment that runs them into line when the environment itself has to change.
None of those parts is unique on its own. The argument of this series is that the interfaces between them are where the value sits, and where the obligations sit too. Each subsystem hands the next one something it must honour: a schema, a permission, a scope, a version. Thank you for following the invoice this far.