Agent-bound model access¶
A Celln Agent's model requests leave its cell for the model gateway,
which adds that Agent's own provider key and forwards the request to the
endpoint the Agent's ModelConnection names. Celln nodes, guests, run specs
and the controller never see the key. There is no operator route list and no
install-time provider: the Agent's configuration is the binding between a
key and the running agent. Any OpenAI-compatible or Anthropic-compatible
endpoint works.
This is how every Celln run gets its model; it is not optional. The fleet guide covers installing the fleet and the gateway. This guide covers connections, keys, what protects them, and the operator's controls.
What protects the key¶
| Property | Where it is enforced |
|---|---|
| One key, one Agent. A key Secret is owned by exactly one Agent (or its Ensemble; sub-agents share their parent's). Another Agent is refused | sympozium.ai/model-key-owner annotation; the controller's claim before any work, the admission webhook, the API (409), and the gateway on every call |
| Pinned to one origin. A key records the endpoint origin it is for and is never sent anywhere else. Moving a connection to another origin is refused; a new origin needs a new key entered by a person | sympozium.ai/model-key-origin annotation; written when the key is created through the API or first claimed, checked by the API, the controller and the gateway |
| Never on a node. The gateway adds the key per request; no node, guest, run spec or controller holds it. Agent pods have no RBAC on Secrets or ModelConnections | gateway; chart RBAC |
| A run cannot carry or redirect a key. No inline credential, header, placement or protocol override with a connection; the connection revision is pinned | modelconnection.Resolve |
| Session binding. One signed permit per run, bound to the cluster, namespace UID, run UID, spec digest, route and budget; turns cannot widen it | controller (issuer), node receiver, gateway |
| Live re-check on every call. Namespace UID, connection UID and spec digest, endpoint origin, Secret UID and key, Agent UID, owner and pin. Revocation bites mid-run | gateway liveRoute |
| Network safety. HTTPS to public addresses by default; private, loopback, link-local, multicast and 100.64/10 refused unless the exact origin is approved; a key never crosses plain HTTP; no redirects; DNS answers validated before connecting; no streaming; provider error bodies hidden; a response that echoes the key is refused | gateway transport and response validation |
| Budgets per run and per turn, reserved before each request | gateway + PostgreSQL |
Multi-team isolation is the namespace: a team's keys are Secrets in its namespace, its Agents are there, and namespace RBAC decides who may write them. An operator may narrow where any key may go (Narrowing origins).
Connections¶
A ModelConnection is a namespaced route: provider, protocol, endpoint,
models and, optionally, the key Secret.
apiVersion: sympozium.ai/v1alpha1
kind: ModelConnection
metadata:
name: cloud
namespace: team-a
spec:
provider: openrouter # a free label, 1-64 letters, digits, _ or -
protocol: openai-chat # or anthropic-messages
endpoint: https://openrouter.ai/api/v1/chat/completions # full HTTPS request URL; a port is fine
secretRef: cloud-openrouter-key # the key Secret in this namespace; omit for a keyless endpoint
# secretKey: API_KEY # key inside the Secret; default OPENAI_API_KEY (openai-chat) or ANTHROPIC_API_KEY (anthropic-messages)
models: [qwen/qwen3-235b-a22b] # exact model identifiers runs may select (1-128)
maxOutputTokens: 2048 # optional, 256-4096 per request; 512 when omitted
# parameters: {chat_template_kwargs: {enable_thinking: false}} # merged into every request by the gateway
# allowInsecure: true # plain HTTP or a private CA, private addresses only, keyless only
provideris descriptive. Nothing matches it against a list.protocolselects the adapter:openai-chatsendsAuthorization: Bearer <key>;anthropic-messagessendsx-api-keyandanthropic-version: 2023-06-01. Any server speaking either works: OpenAI, Anthropic, DeepSeek (https://api.deepseek.com/chat/completions), OpenRouter, Groq, Mistral, Together, xAI, Gemini's OpenAI-compatible endpoint, or a local llama-server, Ollama, LM Studio or vLLM.endpointis the complete request URL, HTTPS, with no credentials, query or fragment. Its origin (scheme://host[:port]) is what the key is pinned to.modelsare exact names. A key may be used for any model its provider serves; the signed decision still binds each run to one concrete model.parametersis pinned on the host side: the guest never sees it, a guest request that sets one of its keys is refused, and the fields the gateway owns (model,messages,max_tokens,stream,tools, …) cannot be set. At most 16 top-level keys matching^[a-z][a-z0-9_]{0,63}$, 3 levels deep, 2048 bytes. Use it for provider switches such as turning a reasoning phase off.maxOutputTokensbounds what one request may ask for; the gateway refuses a request above it. A turn reserves 6 × this value from the run's budget, so a connection created through the API is clamped to what the namespace's policy ceilings can pay for.
Keys¶
The key is a Secret in the Agent's namespace, under the connection's
secretKey (default by protocol: OPENAI_API_KEY or ANTHROPIC_API_KEY;
API_KEY also works when named). Only the gateway reads it.
apiVersion: v1
kind: Secret
metadata:
name: cloud-openrouter-key
namespace: team-a
stringData:
OPENAI_API_KEY: "<your key>"
Two annotations on the Secret carry the binding:
| Annotation | Value | Written |
|---|---|---|
sympozium.ai/model-key-owner |
Agent/<name> or Ensemble/<name> |
by the controller the first time a run of an Agent that grants the Secret uses it (the claim) |
sympozium.ai/model-key-origin |
the endpoint origin, e.g. https://openrouter.ai |
by the API when it creates or re-saves a connection naming the Secret, or by the controller at the claim (a Secret made by hand is pinned on first use) |
Rules:
- Grant. An Agent grants a connection's Secret by selecting the
connection in
spec.execution.modelConnectionRef, or by listing the Secret inspec.authRefs(with the connection'sprovider, or an empty provider for any). A run that names a Secret the Agent neither grants nor owns is refused:model key Secret "x" is not declared in Agent "a" spec.authRefs. - One owner. An Agent whose connection names a Secret another live Agent
or Ensemble owns is refused:
model key Secret "x" belongs to Agent/other; every Agent needs its own key (sub-agents share their parent's). The console does not offer such a Secret, andGET /api/v1/celln-platform/key-secrets?namespace=<ns>&key=OPENAI_API_KEYlists only unowned Secrets that hold that key name (never a value). When the owner is deleted its claim goes stale and the next Agent that grants the Secret takes it over. - One origin. A Secret pinned to one origin and named by a connection to
another is refused whoever owns it:
model key Secret "x" is pinned to https://openrouter.ai and cannot be sent to https://attacker.example; a connection to another provider needs its own key. The API answers409 Conflictwhen a connection is saved that way; the controller fails the run before anything starts; the gateway refuses the call (MODEL_CREDENTIAL_SOURCE_CHANGED) if the pin and the live connection disagree. The pin outlives the owner; to re-use a Secret for another provider on purpose, a person with write access to the Secret removes the annotation (or, better, creates a new Secret). - HTTPS only. A connection with a
secretRefmust have anhttps://endpoint: the resolver refusescredential-bearing model endpoint must use HTTPSand the gateway refusesMODEL_DESTINATION_FORBIDDEN. Plain HTTP is for keyless local endpoints only.
Create an Agent through the API¶
Everything below ran against a real cluster
(test/integration keeps the journey; the shapes are the API server's). The
namespace must be one the fleet admits (any ordinary namespace by default).
kubectl -n sympozium-system port-forward svc/sympozium-apiserver 18080:8080 &
API=http://localhost:18080
TOKEN="$(sympozium token)"
H=(-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json")
NS=team-a
The short way: one call. modelConnection on the Agent create carries the
connection spec and, when the provider wants one, the key. The server writes
the Secret (pinned to the endpoint's origin and owned by this Agent from the
moment it exists), the ModelConnection (named after the Agent unless
modelConnection.name says otherwise), picks the namespace's fleet profile,
creates its toolbox runtime wrapper and sizes the conversation limits from
the policy's session defaults. Anything you set explicitly (runtimeRef,
execution.cellnSelection, execution.enduring, model) is kept.
curl -sf "${H[@]}" -X POST "$API/api/v1/agents?namespace=$NS" -d '{
"name": "researcher",
"execution": {"backend": "celln"},
"modelConnection": {
"apiKey": "sk-or-...",
"spec": {"provider": "openrouter", "protocol": "openai-chat",
"endpoint": "https://openrouter.ai/api/v1/chat/completions",
"models": ["qwen/qwen3-235b-a22b"]}}}'
Then talk to it. POST /api/v1/agents/{name}/messages continues the Agent's
newest live conversation (or the run named by conversation) as a turn, and
starts one when there is none; it answers 201 with the new run or 202
with the turn (200 for a repeated requestId). The answer lands on that
object's status (.status.result on the run,
.status.execution.result.answer on the turn), which you poll or watch as
for any run.
curl -sf "${H[@]}" -X POST "$API/api/v1/agents/researcher/messages?namespace=$NS" \
-d '{"message": "Reply with the single word pong."}'
# {"run": {...}, "started": true}
curl -sf "${H[@]}" -X POST "$API/api/v1/agents/researcher/messages?namespace=$NS" \
-d '{"message": "And now in French.", "requestId": "fr-1"}'
# {"run": {...}, "turn": {...}, "started": false}
A namespace no fleet serves gets 409 no Celln fleet profile is offered …;
a key another Agent already owns gets 409; a key over plain HTTP, 400.
The persona is the Agent's. execution.systemPrompt on the Agent is the
system prompt every run it starts carries into its cell; a run may name its
own instead, and an Agent that names none gets the fleet's default persona
(the one the starter catalogue ships, which tells the model to fetch a live,
dated source before answering anything time-sensitive). It is agent
configuration, editable on the Agent (console: Execution defaults → Persona,
or PATCH /api/v1/agents/{name} with execution), not fleet material. A cell
accepts at most 2048 bytes of system prompt; a longer one is refused with
that bound named.
curl -sf "${H[@]}" -X POST "$API/api/v1/agents?namespace=$NS" -d '{
"name": "analyst",
"execution": {"backend": "celln",
"systemPrompt": "You are a market analyst. Fetch https://news.google.com/rss/search?q=<topic> before answering about recent events and cite dates."},
"modelConnection": {"spec": {"provider": "openrouter", "protocol": "openai-chat",
"endpoint": "https://openrouter.ai/api/v1/chat/completions",
"models": ["qwen/qwen3-235b-a22b"]}, "apiKey": "sk-or-..."}}'
Any OpenAI client. The same Agent is an OpenAI-compatible endpoint at
/api/v1/agents/{name}/v1. POST …/chat/completions sends the newest user
message to the Agent (as above: a turn of its live conversation, or a new
one) and waits for the answer, returned as an ordinary completion, or as SSE
chunks with stream: true; GET …/models lists the Agent as its one model.
The Sympozium token is the API key, and the X-Sympozium-Namespace header
names the namespace, since SDKs cannot add a query parameter (the header
works on every API route; ?namespace= wins when both are given). Earlier
messages in the request are not replayed: the Agent holds its own
conversation. A conversation that is still starting, or answering an
earlier message, takes the message when it is ready. ?timeout= (default
3m, at most 10m) bounds the whole wait; past it the reply is 504 naming
the run and turn to read, and the conversation continues. X-Sympozium-Conversation: <run> continues a named
run instead of the newest. The completion's sympozium field and the
X-Sympozium-Run / X-Sympozium-Turn headers name the objects behind it.
from openai import OpenAI
client = OpenAI(base_url=f"{API}/api/v1/agents/researcher/v1", api_key=TOKEN,
default_headers={"X-Sympozium-Namespace": NS})
reply = client.chat.completions.create(model="researcher",
messages=[{"role": "user", "content": "Reply with the single word pong."}])
print(reply.choices[0].message.content) # pong
A Kubernetes Agent on a session runtime (Pi, Hermes) has the same endpoint; its request is handed to the running session, history and streaming included. The console shows the base URL and a snippet on each Agent.
The long way is what that call does, step by step.
1. The connection, with its key. apiKey makes the server write the
Secret (<connection>-<provider>-key, labelled
sympozium.ai/model-connection: <connection>) and pin it to the endpoint's
origin; or pass spec.secretRef to an existing Secret, which is pinned the
same way. Re-posting the same name updates the connection.
curl -sf "${H[@]}" -X POST "$API/api/v1/model-connections?namespace=$NS" -d '{
"name": "cloud",
"apiKey": "sk-or-...",
"spec": {"provider": "openrouter", "protocol": "openai-chat",
"endpoint": "https://openrouter.ai/api/v1/chat/completions",
"models": ["qwen/qwen3-235b-a22b"]}}'
kubectl -n $NS get secret cloud-openrouter-key -o jsonpath='{.metadata.annotations}'
# {"sympozium.ai/model-key-origin":"https://openrouter.ai"}
Moving it to another origin is refused:
curl -s -o /dev/null -w '%{http_code}\n' "${H[@]}" -X POST "$API/api/v1/model-connections?namespace=$NS" -d '{
"name": "cloud",
"spec": {"provider": "openrouter", "protocol": "openai-chat",
"endpoint": "https://attacker.example/v1/chat/completions",
"models": ["qwen/qwen3-235b-a22b"], "secretRef": "cloud-openrouter-key"}}'
# 409
2. The runtime. The fleet offers one profile per scope; ask the namespace
what it may use and have the runtime wrapper created on first use
(runtimeOnly: true; existing objects are never modified). The .toolbox
profile lends the starter toolbox; the plain profile is chat only.
curl -sf "${H[@]}" "$API/api/v1/celln-platform/profiles?namespace=$NS"
# [{"name":"celln-native-starter","wrapper":"celln-native","mediationOnly":true,
# "tools":[...],"ceilings":{...},"sessionDefaults":{...},
# "toolboxProfile":"celln-native-starter.toolbox","toolboxWrapper":"celln-native.toolbox","toolboxTools":[...]}]
curl -sf "${H[@]}" -X POST "$API/api/v1/celln-platform/wrappers?namespace=$NS" \
-d '{"profile": "celln-native-starter.toolbox", "runtimeOnly": true}'
# {"backend":"native","runtime":"celln-native.toolbox","agent":"","connection":"","created":["celln-native.toolbox"]}
3. The Agent. execution.modelConnectionRef is the grant: the server
resolves the connection, adds its Secret to the Agent's authRefs, and
refuses apiKey, secretName or baseURL alongside it. cellnSelection
selects the wrapper and exactly the toolbox's tools in its order (copy
toolboxTools from the profile; omit them, on celln-native, for a chat-only
Agent). enduring is the budget a conversation asks for, within the
profile's ceilings; sessionDefaults is a sensible one.
TOOLS="$(curl -sf "${H[@]}" "$API/api/v1/celln-platform/profiles?namespace=$NS" \
| python3 -c 'import sys,json; print(json.dumps(json.load(sys.stdin)[0]["toolboxTools"]))')"
curl -sf "${H[@]}" -X POST "$API/api/v1/agents?namespace=$NS" -d "{
\"name\": \"researcher\", \"provider\": \"openrouter\", \"model\": \"qwen/qwen3-235b-a22b\",
\"runtimeRef\": \"celln-native.toolbox\",
\"execution\": {\"backend\": \"celln\", \"executionLifecycle\": \"enduring\",
\"modelConnectionRef\": \"cloud\", \"model\": \"qwen/qwen3-235b-a22b\",
\"cellnSelection\": {\"runtimeRef\": \"celln-native.toolbox\", \"toolRefs\": [], \"clusterToolRefs\": $TOOLS},
\"enduring\": {\"leaseSeconds\": 3600, \"maxTurns\": 8, \"maxModelRequests\": 48, \"maxOutputTokens\": 196608}}}"
4. A run, and its turns.
RUN="$(curl -sf "${H[@]}" -X POST "$API/api/v1/runs?namespace=$NS" -d "{
\"agentRef\": \"researcher\", \"task\": \"Reply with the single word pong.\",
\"backend\": \"celln\", \"executionLifecycle\": \"enduring\",
\"modelConnectionRef\": \"cloud\", \"model\": \"qwen/qwen3-235b-a22b\",
\"cellnSelection\": {\"runtimeRef\": \"celln-native.toolbox\", \"toolRefs\": [], \"clusterToolRefs\": $TOOLS},
\"enduring\": {\"leaseSeconds\": 3600, \"maxTurns\": 8, \"maxModelRequests\": 48, \"maxOutputTokens\": 196608}}" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["metadata"]["name"])')"
kubectl -n $NS get agentrun "$RUN" -o jsonpath='{.status.phase} {.status.result}{"\n"}'
UID="$(kubectl -n $NS get agentrun "$RUN" -o jsonpath='{.metadata.uid}')"
curl -sf "${H[@]}" -X POST "$API/api/v1/runs/$RUN/turns?namespace=$NS" \
-d "{\"message\": \"And now in French.\", \"requestId\": \"fr-1\", \"runUID\": \"$UID\"}"
curl -sf "${H[@]}" "$API/api/v1/runs/$RUN/turns?namespace=$NS"
executionLifecycle: one-shot answers the task once and stops the parent;
the console's Answer once does the same. kubectl apply of the same
objects works too: the Secret and ModelConnection above, the AgentRuntime
the wrapper call created (copy it with kubectl -n $NS get agentruntime
celln-native.toolbox -o yaml), and an Agent with spec.runtimeRef,
spec.authRefs: [{provider: openrouter, secret: cloud-openrouter-key}] and
the same spec.execution. The Secret is then pinned and claimed by the
controller on the first run.
Keyless local models¶
A local OpenAI-compatible server needs no key. Its address is private, so
the operator approves the exact origin once
(Private and local endpoints); the
connection then needs no secretRef and, for plain HTTP, allowInsecure:
curl -sf "${H[@]}" -X POST "$API/api/v1/model-connections?namespace=$NS" -d '{
"name": "local",
"spec": {"provider": "llama-server", "protocol": "openai-chat",
"endpoint": "http://192.168.1.237:8080/v1/chat/completions",
"models": ["Qwen3.8-27B-UD-Q4_K_XL.gguf"], "allowInsecure": true,
"parameters": {"chat_template_kwargs": {"enable_thinking": false}}}}'
The gateway still binds the session, enforces budgets and parameters and
validates the response; it just sends no credential. Reasoning models served
by llama-server can spend a whole request thinking and return an empty
answer; the parameters above turn that off, or raise maxOutputTokens.
Narrowing origins (operator)¶
By default an Agent's own key may go to any public HTTPS origin the gateway
reaches. The scope's CellnExecutionPolicy (celln-fleet-<scope>) may
narrow that with host patterns:
spec:
allowedOrigins: ["api.openai.com", "*.anthropic.com", "llm.internal:8443"] # empty: any origin
deniedOrigins: ["*.example.net"] # refused even if allowed
A pattern is an exact lowercase host with an optional port, or *. over
subdomains (*.example.com matches api.example.com and example.com,
never notexample.com); at most 64 each. The resolver checks the
connection's origin against every policy that selects the namespace when a
run is admitted and again before each turn, and refuses with
AUTH_ROUTE_MISMATCH: policy "celln-fleet-starter" denies model requests
to https://x (pattern) or policy "…" allows model requests only to a, b,
not https://x.
sympozium doctor (Mediated model access) summarises the guardrail, and
the console reads it through the mediation endpoint:
curl -sf "${H[@]}" "$API/api/v1/celln-platform/mediation?namespace=$NS"
# {"enabled":true,"allowedOrigins":[],"deniedOrigins":[],"policies":["celln-fleet-starter"],"routes":[],"pending":[]}
routes and pending are always empty; they remain for 0.11 clients.
Private and local endpoints¶
The gateway sees only an IP, so it cannot tell a LAN model server from a pod or Service address. The rules:
- Public HTTPS (any address that is not private, loopback, link-local, multicast or 100.64/10): reachable by default. Every DNS answer is checked before connecting; mixed public/private answers fail closed; redirects are refused.
- A private destination (RFC 1918, IPv6 ULA, 100.64/10, loopback) needs
its exact origin in
modelGateway.privateOrigins, once for the cluster:
Until then a run to it fails at the gateway with
MODEL_DESTINATION_FORBIDDEN. A listed HTTPS origin is reachable without
the connection setting allowInsecure.
- Plain HTTP to a listed origin additionally needs allowInsecure: true
on the connection, and works only for keyless connections (a Secret never
crosses HTTP). Its DNS answers must all be loopback, private or 100.64/10
addresses.
- A private CA. For an in-house HTTPS server, put the CA in a ConfigMap
in the gateway's namespace and name it in modelGateway.providerCA
(configMap, key: ca.crt); it is trusted only for listed origins, and
public providers never trust it. If you restrict modelGateway.egress,
include the server's address.
Celln's own tools (https-fetch, https-post-json) never reach a private
address, whatever the model connection allows.
Refusals¶
Every refusal names a stable reason. Where a run is refused before it starts,
the reason is on the AgentRun's conditions and the run fails; where a model
call is refused, the gateway logs one line (no credential, route or request
content) and the turn fails:
model-gateway: refused status=403 reason=MODEL_CREDENTIAL_SOURCE_CHANGED op=invoke claimed_run=team-a/researcher-x7k2p claimed_agent=researcher claimed_turn=researcher-x7k2p-recall
The controller refuses a run's model key the same way on every backend (Job,
Agent Sandbox, Celln), before any work: the run fails with the message as
status.error and the ModelKeyAuthorized condition set to False with one
of the MODEL_KEY_* codes below as its reason. The API and webhook refuse
the same cases with 409 and the same message.
| Reason | Meaning |
|---|---|
MODEL_KEY_NOT_GRANTED |
The run names a key (or a harness model credential) its Agent does not declare in spec.authRefs: model key Secret … is not declared in Agent … spec.authRefs |
MODEL_KEY_CONFLICT |
The key belongs to another Agent: model key Secret … belongs to … |
MODEL_KEY_ORIGIN_MISMATCH |
The key is pinned to another endpoint origin: model key Secret … is pinned to … and cannot be sent to … |
AUTH_ROUTE_MISMATCH |
The connection cannot be the route: a policy's allowedOrigins/deniedOrigins refuses its origin, a keyed endpoint is not HTTPS, the connection is disabled or malformed, or the run's model is not among its models |
AUTH_LIMIT_OUT_OF_RANGE |
spec.enduring asks for more than the policy's ceilings, or cannot pay for one turn at the connection's maxOutputTokens |
AUTH_TOOL_ORDER_MISMATCH, AUTH_TOOL_UNKNOWN |
clusterToolRefs is not exactly the toolbox's tools in its order |
AUTH_POLICY_WITHDRAWN |
The namespace is not admitted by any policy |
AUTH_PROTOCOL_UNSUPPORTED |
The node runs a Celln release without a contract the run needs |
AUTH_CONTEXT_LOST |
The node holding the run's native state is gone; the run ends and nothing is re-placed |
MODEL_AUTH_UNAUTHORIZED, MODEL_AUTH_FORBIDDEN |
The permit does not verify, or the request widens it (a pinned parameter, an unregistered run, output tokens outside the connection's bound or the turn's cap) |
MODEL_ROUTE_CHANGED |
The live connection no longer matches the decision: UID, digest, endpoint origin, provider, protocol, model list, or the namespace changed or is being deleted |
MODEL_CREDENTIAL_SOURCE_CHANGED |
The Secret, its key, its UID, its owner or its pin no longer match; or the Agent is gone or has another UID |
MODEL_DESTINATION_FORBIDDEN |
Private, loopback, link-local or unlisted private address; plain HTTP with a key; a redirect |
MODEL_STREAMING_UNSUPPORTED, MODEL_PROTOCOL_UNSUPPORTED, MODEL_REQUEST_MALFORMED |
Request shape the gateway does not forward |
MODEL_REQUEST_CONFLICT, MODEL_RESPONSE_TOO_LARGE, MODEL_PROVIDER_UNAVAILABLE, MODEL_GATEWAY_UNAVAILABLE |
Duplicate request id, oversized or invalid provider response, provider unreachable, gateway or its database not ready |
AUTH_BUDGET_EXHAUSTED, AUTH_BUDGET_CLOSED, AUTH_WORK_DEADLINE_EXPIRED, AUTH_BUDGET_NOT_REGISTERED, AUTH_PROVIDER_USAGE_EXCEEDS_RESERVATION |
Budget accounting: the run or turn has no reservation left, is closed, past its deadline, unregistered, or the provider reported more usage than was reserved |
A run whose node receiver rejects the controller's token reports
ReceiverUnavailable (RECEIVER_AUTH_REJECTED); one that cannot reach it
over trusted TLS reports RECEIVER_UNREACHABLE. Both wait rather than fail,
because nothing was submitted.
Budgets¶
A run's spec.enduring (leaseSeconds, maxTurns, maxModelRequests,
maxOutputTokens) is admitted within the scope's policy ceilings. Every turn
reserves 6 model requests and 6 × the connection's maxOutputTokens output
tokens from the run's totals before its first request, and the gateway
reserves each request's full output bound before forwarding it; unused
reservation is reconciled from the provider's reported usage. A run whose
output tokens run out on a healthy node reports Uncertain and does not end
by itself; size spec.enduring for the turns you expect
(sessionDefaults in the profiles API is computed from the ceilings). The
ceilings themselves are set at install
(Leases and budgets).
Tools on this path¶
The toolbox does not depend on the model route. An Agent on the .toolbox
profile gets the workspace tools (read, write, list, append, search, delete;
files live with the parent and end with it), https-fetch and
https-post-json to any public HTTPS host (or the scope's
--celln-fleet-https-host list), and the borrowed commands, each with the
limits the signed decision carries. See
The toolbox.
How it is wired¶
sympozium install sets celln.mediation.enabled, clusterId (the
kube-system namespace UID), issuer.keyId and modelGateway.image on
every run, and bootstraps the trust right after the chart creates its
namespaces. The trust is never rendered from Helm values:
| Object | Namespace | Keys | Read by |
|---|---|---|---|
Secret celln-mediation-controller |
control plane | issuer.key (PKCS#8 Ed25519 PEM), receiver-token, gateway-token |
controller |
Secret celln-mediation-gateway |
control plane | tls.crt, tls.key, registration-token (= gateway-token) |
gateway |
Secret celln-mediation-node |
celln-system |
operator-token (= receiver-token), tls.crt, tls.key |
dispatcher, receiver sidecar |
ConfigMap celln-mediation-trust |
both | jwks.json, ca.crt |
all three |
Against the same issuer (sympozium-control-plane), keyset, CA and tokens:
the controller signs one permit per run and talks to the
celln-scoped-receiver Service (the router's TLS sidecar, which forwards
each /v1/scoped/* operation to the node that prepared it); the celln-node
dispatchers verify the permit and reach the gateway on 8443; the gateway
verifies it again, registers the run and serves its model calls. A rerun of
the installer reads the existing key id back, verifies the objects and
changes nothing. The chart installs the record ConfigMap
celln-system/celln-mediation, which doctor and the API read.
For a Helm-only install the same objects come from
sympozium celln-mediation bootstrap --cluster-id <id> --values-out
mediation-values.yaml (flags: --system-namespace, --release-fullname,
--gateway-host, --receiver-host, --validity), whose output carries the
credential-free values to pass to helm upgrade.
Rotation¶
Certificates last ten years (--validity) and the CA key is discarded after
signing, so nothing renews them in place: rotation is the renewal.
sympozium doctor warns in the last 60 days and fails once they have
expired. To rotate, delete the five objects (the four above; the trust
ConfigMap exists in both namespaces) deliberately, run sympozium install
(or celln-mediation bootstrap), and restart the controller, the model
gateway and the celln-node pods. The controller and gateway pods carry a
checksum/mediation-trust annotation, so the next helm upgrade or
sympozium upgrade rolls them onto the new trust; trust recreated outside
Helm needs kubectl -n sympozium-system rollout restart
deploy/sympozium-controller-manager deploy/sympozium-model-gateway. Until
then runs report RECEIVER_AUTH_REJECTED. Live conversations end with the
celln-node restart.
PostgreSQL¶
The gateway keeps budgets and registrations in PostgreSQL and refuses to
start without it. With modelGateway.database.secretName empty the chart
runs its own single-pod PostgreSQL (<release>-model-gateway-db, a
StatefulSet on a 1 Gi PVC, digest-pinned postgres:17-alpine; password
generated once and kept across upgrades; a NetworkPolicy admits only the
gateway). It holds accounting, never provider keys. For production, apply
migrations/002_celln_model_budget.sql and
migrations/003_celln_model_gateway.sql to your own database and set
modelGateway.database.secretName to a Secret holding the URL under
database-url; the chart then renders no database. If you restrict
modelGateway.egress, include the database's address.
Gateway RBAC and egress¶
By default the gateway keeps a cluster-wide get on Secrets,
ModelConnections and Namespaces. modelGateway.namespaces replaces that
with one Role/RoleBinding per listed namespace (and sets the gateway's
readiness probe to exactly that access); add a namespace before its Agents
run. modelGateway.egress is the gateway's NetworkPolicy egress: empty
allows any destination (an Agent may use any provider); a list restricts it.
Current limits¶
- Node loss ends a run (
AUTH_CONTEXT_LOST); nothing is re-placed on another node. Workspace files live with the parent and end with it. - A dispatcher restart loses live parents, including the roll caused by a fleet package move or a trust rotation.
- No streaming. The gateway forwards complete request and response envelopes.
- Secret volumes cannot be owned by a non-root user, so a non-root init step
in the controller and gateway pods copies the operator's files into an
in-memory volume with mode
0400; a changed Secret takes effect on the next pod restart. - NetworkPolicies need an enforcing CNI. The gateway admits the controller
and the
celln-nodepods on 8443;celln-nodeadmits the controller on the receiver port only, and its plaintext 8787 stays router-only.