ExecutionBackend seam¶
Status: first step landed with this note, 2026-10-30. Part of epic #692; the Substrate backend that will be the seam's first new implementation is #699. The first change moves the existing Job and Celln paths behind the seam with no behaviour change.
Why¶
The AgentRun reconciler runs a run in one of three ways today. Each has its
own branch in Reconcile, reconcilePending, reconcileRunning and
reconcileDelete:
| Path | Selected by | Start | Observe | Cancel / cleanup |
|---|---|---|---|---|
| Kubernetes Job (default) | neither of the other two | buildJob + create, in reconcilePending |
Job and pod status in reconcileRunning |
delete the Job in reconcileDelete |
| Agent Sandbox | spec.agentSandbox.enabled, then status.sandboxName/sandboxClaimName |
reconcilePendingAgentSandbox |
reconcileRunningAgentSandbox |
owner references |
| Celln (scoped platform) | sharedCatalogueSelected, before model normalisation |
reconcilePendingScoped |
reconcileRunningScoped / observeScoped |
cleanupScoped |
Adding Agent Substrate as a fourth branch in each of those functions would make the reconciler harder to reason about for every backend. The seam makes a backend an implementation of one interface instead.
Layering¶
Sympozium stays the coordination layer. These stay above the seam and are never delegated to a backend:
- Agent keys. The Agent is the only identity that can use its model key. Target for every backend (option (a) in #692): the backend's only route to a model is our model gateway, which adds the key. Nothing in the seam carries a key or a Secret reference. Today Celln meets this; the Job backend does not (see Known exceptions).
- Policy and tool gating.
SympoziumPolicyadmission and tool allow/deny run beforeStartand in our tool server. - Model key authorisation (
authorizeRunKeys). Target: above the seam for every backend. Today it is above the seam for Job only; for Celln it still runs insidereconcilePendingScoped(celln_scoped.go), that is below the seam, and moves up with theAdmitsplit. - Spend. See Backend-neutral spend ceiling.
- Ensembles, delegation, channels, schedules, memory, gates, postRun hooks. The reconciler drives these from the run's phase and result, whatever backend produced them.
- Attribution. Traces, token usage, cost and device energy are keyed to the run, using the identity the backend reports.
The interface¶
The target contract:
type ExecutionBackend interface {
Name() BackendName
Capabilities(ctx) (Capabilities, error) // ErrBackendTemporarilyUnavailable => requeue
Admit(ctx, run) (Admission, *Refusal, error) // refuse before anything starts
Start(ctx, Admission, idempotencyKey) (StartResult, error)
Phase(ctx, run) (Phase, error) // includes Uncertain and ContextLost
Result(ctx, run) (*Result, error) // terminal only
Logs(ctx, run) (string, error) // may be ErrLogsUnsupported
Cancel(ctx, RunIdentity) error // stop the work
Release(ctx, RunIdentity) (confirmed bool, err error) // idempotent cleanup, separate from Cancel
Identity(run) RunIdentity
}
Step 1 (this change): wrap, don't split¶
internal/controller/execution_backend.go defines the types below and an
ExecutionBackend interface whose Start, Observe and Release are still
reconcile-shaped, because they wrap today's functions unchanged:
| Verb | Job | Celln |
|---|---|---|
Owns |
not Celln, not Agent Sandbox | sharedCatalogueSelected |
Start |
startJob (the Job tail of reconcilePending, moved verbatim) |
reconcilePendingScoped |
Observe |
observeJob (the Job part of reconcileRunning, moved verbatim) |
reconcileRunningScoped |
Release |
delete the Job (from reconcileDelete) |
cleanupScoped |
Logs |
readAgentLogs (terminal) |
unsupported |
Phase, Result, Identity, Capabilities |
read-only views of status | read-only views of status |
Reconcile, reconcilePending, reconcileRunning and reconcileDelete
call the backends in today's order: a persisted or selected Celln scoped
binding first (before model connection normalisation), then Agent Sandbox
(still inline), then Job. For Job, shared admission (runtime resolution,
policy, gate validation, token budget, key authorisation) stays in
reconcilePending above the seam. Celln is selected before that code runs,
so its admission, including authorizeRunKeys, happens inside
reconcilePendingScoped, below the seam, until Admit is split out.
What step 1 implements versus what is a target:
| Item | Status |
|---|---|
Phase: Refused, Uncertain, ContextLost for Celln |
done. Refused = a shared admission refusal (ModelKeyAuthorized=False, typed MODEL_KEY_* code, Job and Celln alike) or status.cellnScoped.nativePhase == "Refused" (also after StartAttempted); ContextLost = a receiver-reported AUTH_CONTEXT_LOST (CellnContextLost condition); Uncertain = CellnScopedExecution Unknown on a live run, including the NativeOwnerUncertain grace period, which can still end in a lease-elapsed success |
Result with an opaque Receipt |
done, read-only from status |
Admit, the idempotency key on Start, Cancel, and the Unknown phase |
target; not implemented |
Closed enums for RefusalCode / PublicReason |
target. Today RefusalCode is a string; Celln's code is the AUTH_* prefix scopedFailureSummary writes, empty when it mapped none |
A bound on Result.Output size |
target |
"Celln success requires a receipt" enforced in Result |
target; Result reports whatever status holds |
ErrBackendTemporarilyUnavailable |
declared, not returned yet: backend holds (e.g. ScopedDispatchDisabled) still show as Pending |
Capabilities |
Job reports ready; Celln is hard-coded to unknown |
Typed outcome markers (ModelKeyAuthorized, CellnContextLost) |
done. failRun writes them in the same status update as the Failed phase, stamped with LastTransitionTime == completedAt; the seam reads a marker only when the two match, so a crash cannot leave one without the transition and a stale one never relabels a later failure |
| One release path | not yet: reconcileCompleted still calls cleanupScoped directly, a second Celln release path beside Release in reconcileDelete, to fold in later |
Next steps¶
- Split
StartintoAdmitandStart(admission, idempotencyKey), with the caller persisting "attempted" first, ascelln_scoped.goalready does. - Split
ObserveintoPhaseplusResult, sosucceedRun,failRun, memory and postRun hooks are driven by the controller from the typed result instead of from inside each backend. - Separate
CancelfromRelease. - Lift the gateway reserve → start → release → close sequence out of the
Celln dispatcher (
internal/cellnscoped) into the controller, above the seam. That is the hook for #688. - Wrap Agent Sandbox; add Substrate (#699).
Types¶
- Admission / Refusal.
Admitreturns anAdmissionthatStartconsumes, or aRefusal{Code, Message}. Refusal is a distinct result from failing after start: the backend did no work, a spend ledger must not charge it, and retry policy differs. Target: codes are a closed enum; Celln'sAUTH_*codes (internal/cellnplatformresolver,celln_scoped.goadmission) and the sharedMODEL_KEY_*codes map onto it. - Start is idempotent (target). The caller persists "attempted" with an
idempotency key before calling
Start, ascelln_scoped.goalready does before dispatch, so a replayed reconcile never creates a second execution. - Phase is
Pending | Running | Succeeded | Failed | Refused | Uncertain | ContextLost | Unknown.Uncertain(the backend cannot confirm the outcome) andContextLost(the owner reports the context gone) are distinct fromFailed: nothing may be replayed or replaced, and an operator has to look. Celln:scopedUncertain(including theNativeOwnerUncertaingrace period) isUncertain; a receiver-reportedAUTH_CONTEXT_LOSTisContextLost. - Result is
{Outcome, PublicReason, Output, Receipt}.PublicReasonis to be a closed enum (target) andOutputbounded in size (target); free-text native diagnostics never reach tenant-visible status (asscopedFailureSummarydoes today).Receiptis opaque and write-once: Sympozium stores it (Celln:status.cellnScoped.receiptDigest; request ID, resolved hashes, verdict) but never interprets it, and rejects a change to it. Target: a backend may require a receipt on success (Celln would), andResultrejects a success without one; not enforced yet. - Release is idempotent cleanup that returns whether cleanup is
confirmed, separate from
Cancel(stop the work). Celln'scleanupScoped/CleanupConfirmedalready work this way. - Capabilities are
{State, Lifecycles, Logs: none|terminal|stream, Turns}whereStateisready | not_installed | transport_invalid | unknown(Agent Sandbox without its CRDs isnot_installed; a Celln receiver with invalid configuration istransport_invalid). A temporarily-unavailable error tells the caller to requeue, not to fail. Today Celln'sCapabilitiesis hard-coded tounknown, andErrBackendTemporarilyUnavailableis declared but never returned. - Logs may be unsupported or terminal-only, and anything returned must be tenant-safe by contract.
- No secrets in the input (target). No key material crosses the seam,
but
executionStartInputstill carries the Agent and the run, which name Secrets (spec.model.authSecretRef), because the Job backend renders its pod from them (the Job exception below). Celln ignores them; its platform decision already strips Secret names.
Known exceptions and gaps (not fixed by the first change)¶
- Job backend holds the key.
buildContainersinjects the Agent's provider key fromspec.model.authSecretRefinto the agent container's env (oneSecretKeyRefper allowed key) and the provider headers asMODEL_PROVIDER_HEADERS. So "no key inside the backend" is true for Celln and is a known exception for Job (and Agent Sandbox, which shares the pod template). Planned follow-up: route Job model traffic through a gateway (internal/modelgateway; draft #700 does this for harness runs with a per-run token), which is also what the backend-neutral #688 ceiling needs. - Gateway sequence lives inside the Celln dispatcher. Reserve at the
gateway, start, release, close happens inside
internal/cellnscoped's dispatcher today. The next step is to lift that sequence above the seam into the generic controller, where it becomes the hook for #688. - Second Celln release path.
reconcileCompletedstill callscleanupScopeddirectly, besideReleaseinreconcileDelete. Fold it intoReleasewith theCancel/Releasesplit. - Backend immutability. The "backend cannot change after the first
reconcile" guard (
status.cellnOnly) covers Celln only; it should become a per-backend record set once and checked for every backend. - Response gate parity. The response gate (
agentrun_gate_verdict.go) is skipped on the Celln scoped path. - One-shot budget mapping bug. On the one-shot Celln path a gateway 429
with
ReasonExhaustedis reported asGatewayRegistrationUnconfirmed(an uncertain outcome) instead ofAUTH_BUDGET_EXHAUSTED. Turns map it correctly. Effect: a run that hits a ceiling is leftUncertaininstead of refused. Separate fix.
Deferred¶
- Turns and enduring runs.
AgentRunTurnReconcilerstays wired directly tocellnscopedand refuses non-Celln backends; Celln's enduring path (leased parents) stays where it is. Substrate's suspend/resume would be a different shape and is phase 1 of #692 at the earliest. - Server mode (
reconcilePendingServer, web-endpoint) stays a Deployment owned by the reconciler. - Agent Sandbox gets wrapped in the same way in a follow-up; its paths
(
reconcilePendingAgentSandbox,reconcileRunningAgentSandbox) are already separate functions.
Substrate note (#699): its actors resume per turn and suspend between, so turns map onto Substrate naturally. That is a reason to design turns into the seam deliberately later, not to add them now; 0b proves the one-shot shape first.
Per-run attribution on shared workers¶
Job, Agent Sandbox and Celln give each run its own pod or cell, so device
energy and cost can be attributed by pod. Substrate multiplexes idle actors
onto warm workers, and an actor can resume on a different worker (0.42 s in
the phase 0a install record): one pod serves many runs over time. RunIdentity therefore
carries the run (namespace, name, UID) and the backend's workload handle
separately, and attribution must key on the run, using the backend's
per-actor identity and active intervals, not on the pod. This is a proposed exit
criterion for the Substrate spike (Ergoz needs per-actor active windows from
the backend).
Backend-neutral spend ceiling (#688)¶
Plan only; not implemented in this change.
Today budgets live in two places: membrane token budgets checked in
reconcilePending (checkTokenBudget), and Celln's per-run and per-turn
budgets reserved at the model gateway. #688 asks for a per-namespace ceiling
on CellnExecutionPolicy, which would cap only Celln runs.
Proposal:
- Where it lives. A namespace-scoped ceiling on
SympoziumPolicy(or a smallSpendPolicy), not onCellnExecutionPolicy: amount (tokens and, where priced, currency), period (rolling day, calendar month), and behaviour at the limit (refuse new runs, or also end running ones). - Ledger. A per-namespace, per-period ledger the controller owns,
fed from what every backend already reports:
TokenUsageon the terminal run, and the gateway's reservations for runs whose model traffic goes through it. Keyed by run UID so a retried reconcile never double-counts. - Enforcement points. (Prerequisites: Job model traffic routed through
internal/modelgateway, and the gateway reserve → start → release → close sequence lifted out of the Celln dispatcher into the controller, above the seam.) BeforeStartfor every backend (refuse withOutcomeRefused/SPEND_CEILING, no backend work done); and at the model gateway's reservation for traffic that passes through it, which covers Celln today and every backend once Job and Agent Sandbox route through the gateway. - Surfacing. Ceiling, spent and remaining on the policy status; a Kubernetes Event on the refused run (this also closes the second half of #688 for all backends, not only Celln).
- Celln compatibility. Celln's own per-run and per-turn budgets stay as they are; the namespace ceiling sits above them.
Open question: runs on a backend that cannot report usage (some harness adapters). They count as unmetered and follow the existing unmetered-run policy from #349.