Getting Started with Sympozium¶

This guide walks you through installing Sympozium, activating your first Ensemble, and setting up practical agent patterns for SRE, security, and DevOps workflows.

Prerequisites¶

A running Kubernetes cluster (Kind, minikube, EKS, GKE, AKS, etc.)
kubectl configured and pointing at the cluster
An LLM provider — either:
An API key (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock), or
A cluster-local model — no API key needed

Install¶

Homebrew (macOS / Linux)¶

brew install sympozium-ai/sympozium/sympozium

Shell installer¶

curl -fsSL https://deploy.sympozium.ai/install.sh | sh

From source¶

go install github.com/sympozium-ai/sympozium/cmd/sympozium@latest

Verify the install:

sympozium version

Deploy the control plane¶

Sympozium needs its CRDs, controller, API server, NATS event bus, and webhook installed in your cluster:

sympozium install

This creates the sympozium-system namespace and deploys the control plane, the built-in Ensembles, SkillPacks and policies, the Celln hardware-isolated execution plane, and ergoz accelerator power telemetry (--no-celln / --no-ergoz skip them). It is idempotent — safe to run again to upgrade.

If DEEPSEEK_API_KEY, OPENAI_API_KEY or ANTHROPIC_API_KEY is set (or you answer the prompt in a terminal), the same command installs the Celln fleet: every node with /dev/kvm and a kernel under /boot runs long-running, hardware-isolated agents for every namespace, one model backend per key. On Kind, copy the host kernel into each node first — see Celln Fleet Installation.

Onboard your agents¶

Sympozium offers two onboarding paths:

Ensembles (recommended) — activate a pre-built bundle of agents via the TUI wizard. One action creates multiple purpose-built agents with skills, schedules, memory, and tool policies.
Manual onboard — create a single Agent with sympozium onboard. Best for custom setups or headless environments.

You can also create Agents, runs and Celln conversations from the web dashboard.

Ensemble activation (recommended)¶

Launch the TUI:

sympozium

The TUI opens on the Ensembles tab, listing the built-in Ensembles:

Pack	Agents	Focus
`platform-team`	security-guardian, sre-watchdog, platform-engineer	Security audit, cluster health, scheduled ops
`devops-pipeline-example`	incident-responder, cost-analyzer	Incident triage, resource optimisation
`developer-team`	7 (tech lead, backend, frontend, QA, reviewer, DevOps, docs)	A software team on one GitHub repository
`research-delegation-example`	lead, researcher, writer, reviewer	Delegation and sequential workflows
`code-analysis-team`, `subagent-analysis-example`	1–2	Dynamic sub-agent spawning
`observability-mcp-example`	2	SRE investigation through MCP servers
`local-inference-example`	2	Runs entirely on a cluster-local `Model`

Press Enter on a pack to start the activation wizard:

Step	What it does
1 — Pick personas	Review the personas in the pack, deselect any you don't need
2 — Provider	Choose your LLM provider (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Ollama, LM Studio, llama-server, Unsloth, or custom endpoint)
3 — API key	Paste your API key (stored as a Kubernetes Secret)
4 — Model	Pick a model (e.g. `gpt-4o`, `claude-sonnet-4-20250514`, `llama3`)
5 — Channels	Optionally bind messaging channels (Telegram, Slack, Discord, WhatsApp)
6 — Confirm	Review and apply — the controller creates all agents automatically

Within seconds you'll have multiple agents running on schedules, each with their own skills, memory, and tool policies. The TUI switches to the Agents tab where you can see them come online.

What gets created:

For each persona in the pack, the Ensemble controller creates:

An Agent — the agent identity with model, skills, and auth
A SympoziumSchedule — the recurring task (heartbeat, sweep, or cron)
Persistent memory — the memory SkillPack (a per-Agent memory server with SQLite on a PVC) plus a ConfigMap seeded with initial context

All resources are owned by the Ensemble — deleting the pack cascades to everything it created.

Manual onboard (single Agent)¶

For a single custom agent, or in headless environments:

sympozium onboard           # interactive text wizard

The wizard (also available in the TUI with O or /onboard) walks you through nine steps:

Step	What it does
1 — Cluster check	Verifies the cluster is reachable and Sympozium is installed. Offers to run `sympozium install` if CRDs are missing.
2 — Namespace	Pick the namespace for the Agent.
3 — Agent	Name your Agent.
4 — Provider	Choose your LLM provider (OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Ollama, LM Studio, llama-server, Unsloth, or any OpenAI-compatible endpoint), enter a base URL if needed, your API key, and a model.
5 — GitHub repository	Optionally connect a repository for GitOps skills.
6 — Instructions	Optional system instructions for the Agent.
7 — Channel	Optionally connect a messaging channel (Telegram, Slack, Discord, WhatsApp).
8 — Policy	Choose a policy preset: Permissive (everything allowed), Default (every tool except `fetch_url`), or Restrictive (very locked-down).
9 — Heartbeat	Pick how often the agent should wake up on its own: every 30 min, hourly (recommended), every 6 hours, daily at 9 AM, or disabled.

The wizard creates:

A Kubernetes Secret with your API key
An Agent custom resource (your agent identity)
A SympoziumPolicy (tool-gating rules)
A SympoziumSchedule heartbeat (unless you chose "disabled")

After onboarding you land in the TUI dashboard — your agent is live.

The web dashboard¶

Sympozium ships with a full web UI embedded inside the API server pod. To access it from your workstation, use the CLI:

sympozium serve

This port-forwards the in-cluster API server to http://127.0.0.1:9090 and prints the authentication token. Log in with the token shown in the terminal.

Options¶

Flag	Default	Description
`--port`	`9090`	Local port to forward to
`--open`	`false`	Automatically open a browser
`--service-namespace`	`sympozium-system`	Namespace of the apiserver service

Manual access¶

If you prefer to port-forward manually:

kubectl port-forward -n sympozium-system svc/sympozium-apiserver 8080:8080

Retrieve the UI token:

sympozium token

What you can do¶

The web dashboard provides a graphical interface for all Sympozium actions:

Dashboard — cluster overview with agent counts, run stats, and recent activity
Agents — list, create, and delete Agents; choose the execution plane (Kubernetes or Celln), harness runtime and, on the Celln fleet, the Model backend
Agents → Chat / Harness — start or resume a persistent conversation: an AgentHarness session (Pi, Hermes) or an enduring Celln conversation (with Answer once for one-shots and Restart elsewhere to continue on another node)
Runs — view all AgentRuns, inspect logs, create new runs
Harnesses — inspect administrator-approved harness runtimes and advanced session state
Policies — browse SympoziumPolicy rules
Skills — explore installed SkillPacks
Schedules — list and manage SympoziumSchedules
Ensembles — browse and activate Ensembles
Models, Topology and Infrastructure — cluster-local models, node fitness and, with an energy collector such as ergoz, live accelerator power
MCP Servers and Gateway — external tool providers and web-endpoint routing

Helm values¶

When installing via Helm, configure the web UI through values.yaml:

apiserver:
  webUI:
    enabled: true       # Serve the embedded web dashboard (default: true)
    token: ""           # Explicit token; leave blank to auto-generate a Secret

If token is left empty, Helm creates a <release>-ui-token Secret with a random 32-character token.

The TUI dashboard¶

Once onboarded, launch the terminal dashboard:

sympozium

From the dashboard you can:

Send tasks to your agent by typing a message and pressing Enter.
View runs — see live status of current and past AgentRuns.
Edit an Agent — open the edit modal (press e) to change the heartbeat schedule, review memory, or toggle skills.
Switch Agents — if you have multiple Agents.

See the TUI reference for every view and key.

Running your first task¶

Type a message into the input bar:

List all pods that are not Running across every namespace.

Sympozium creates an AgentRun CR, spins up an ephemeral pod, calls your LLM, and uses the built-in tools to fulfil the task. You will see the result stream back in the TUI.

Built-in tools¶

Every agent pod ships with these eight tools:

Tool	Description
`execute_command`	Run shell commands (kubectl, curl, jq…) in a skill sidecar
`read_file`	Read a file from the pod filesystem
`write_file`	Create or overwrite a file
`edit_file`	Apply one or more exact-string (unique-match) replacements to a file (atomic)
`list_directory`	List directory contents
`send_channel_message`	Send a message to Telegram / Slack / Discord / WhatsApp
`fetch_url`	HTTP GET a URL and return the body
`schedule_task`	Create, update, suspend, or delete SympoziumSchedule CRDs

Tools are governed by the SympoziumPolicy you selected during onboarding. The default policy lets read-only tools run freely and asks for approval before execute_command.

Agent patterns (what Ensembles create)¶

The following patterns show the resources that Ensembles generate automatically. You can also create these manually if you prefer fine-grained control.

Below are three practical agent personas. Each combines a Agent, one or more SkillPacks, and a tailored schedule to create a purpose-built agent.

Tip: The platform-team Ensemble creates the SRE and Security agents below automatically. The devops-pipeline-example pack creates the Incident Responder. You only need to write YAML manually for custom personas.

1. SRE On-Call Agent¶

An always-on agent that monitors cluster health, triages incidents, and can perform rollbacks.

Skills: k8s-ops, incident-response

apiVersion: sympozium.ai/v1alpha1
kind: Agent
metadata:
  name: sre-oncall
spec:
  agents:
    default:
      model: gpt-4o
  skills:
    - skillPackRef: k8s-ops
    - skillPackRef: incident-response
  policyRef: default-policy

Heartbeat — every 30 minutes:

apiVersion: sympozium.ai/v1alpha1
kind: SympoziumSchedule
metadata:
  name: sre-oncall-heartbeat
spec:
  agentRef: sre-oncall
  schedule: "*/30 * * * *"
  type: heartbeat
  includeMemory: true
  concurrencyPolicy: Forbid
  task: |
    Quick cluster health check:
    1. Are all nodes Ready?
    2. Any pods not Running?
    3. Any Warning events in the last 30 minutes?
    Summarise findings. If something looks wrong, triage it.

Example tasks to try:

Why is the checkout-service pod crash-looping in the production namespace?

Roll back the payments-api deployment to the previous version.

Show me the top 5 resource-hungry pods across the cluster.

The k8s-ops skill gives the agent kubectl access through a sidecar container with scoped RBAC. The sre-observability skill adds Prometheus/Loki/Kubernetes metrics and log triage workflows with read-only observability RBAC. incident-response provides structured triage, log analysis, and rollback runbooks so the agent follows a consistent process.

2. Security Auditor Agent¶

A periodic agent that reviews cluster configuration and scans code for anti-patterns. Runs on a daily schedule.

Skills: code-review (includes security-patterns)

apiVersion: sympozium.ai/v1alpha1
kind: Agent
metadata:
  name: security-auditor
spec:
  agents:
    default:
      model: gpt-4o
  skills:
    - skillPackRef: code-review
    - skillPackRef: k8s-ops
  policyRef: restrictive

Heartbeat — daily at 9 AM:

apiVersion: sympozium.ai/v1alpha1
kind: SympoziumSchedule
metadata:
  name: security-daily-scan
spec:
  agentRef: security-auditor
  schedule: "0 9 * * *"
  type: scheduled
  includeMemory: true
  concurrencyPolicy: Forbid
  task: |
    Daily security audit:
    1. Check for pods running as root (runAsNonRoot not set).
    2. Check for containers with privileged: true or ALL capabilities.
    3. Look for Secrets mounted as environment variables instead of volumes.
    4. Check that NetworkPolicies exist in all non-system namespaces.
    5. Report findings with severity (Critical / High / Medium / Low).

Example tasks to try:

Audit RBAC — which ServiceAccounts have cluster-admin?

Check if any deployments are using the :latest image tag.

Review the Helm values in the staging namespace for hardcoded secrets.

The restrictive policy ensures the agent cannot run arbitrary commands without approval — the right guardrail for a security-focused agent.

3. DevOps / Platform Engineer Agent¶

A general-purpose agent for day-to-day cluster operations, deploys, and troubleshooting. Runs with a permissive policy on development clusters.

Skills: k8s-ops, code-review

apiVersion: sympozium.ai/v1alpha1
kind: Agent
metadata:
  name: devops
spec:
  agents:
    default:
      model: gpt-4o
  skills:
    - skillPackRef: k8s-ops
    - skillPackRef: code-review
  policyRef: permissive

Heartbeat — every hour:

apiVersion: sympozium.ai/v1alpha1
kind: SympoziumSchedule
metadata:
  name: devops-heartbeat
spec:
  agentRef: devops
  schedule: "0 * * * *"
  type: heartbeat
  includeMemory: true
  concurrencyPolicy: Forbid
  task: |
    Check in: review any pending tasks in memory.
    Quick scan — any pods restarting or events firing?

Example tasks to try:

Scale the frontend deployment to 5 replicas in the staging namespace.

Create a new namespace called "feature-xyz" with a LimitRange and ResourceQuota.

Show me the rollout history for the api-gateway deployment and explain what
changed between revision 3 and 4.

Drain node worker-3 for maintenance, making sure no PDBs are violated.

With the permissive policy, this agent has free rein on a dev cluster — fast iteration without approval gates.

Built-in SkillPacks¶

Sympozium installs ten built-in SkillPacks. Enable them on any Agent:

SkillPack	Category	What it includes
k8s-ops	Kubernetes	Cluster overview, pod troubleshooting, resource management. Comes with a sidecar that has kubectl and cluster-scoped RBAC.
sre-observability	SRE	Observability triage with Prometheus queries, Loki/kubectl log analysis, and event correlation. Comes with a sidecar and read-only observability RBAC.
incident-response	SRE	Structured incident triage, log analysis, rollback procedures.
code-review	Development	Code review checklist, security anti-patterns, Go-specific review patterns.
software-dev	Development	Software development workflows used by the `developer-team` Ensemble.
github-gitops	GitOps	GitHub repository, issue and pull-request workflows. See GitHub GitOps.
memory	Core	Persistent memory with full-text search across runs. See Persistent Memory.
subagents	Orchestration	Dynamically spawn sub-agents to parallelize or pipeline work.
llmfit	SRE	Node-level model placement analysis. Runs llmfit probes per node and ranks best nodes for requested models. Comes with a sidecar containing `llmfit`, `kubectl`, and `jq`.
web-endpoint	Connectivity	Expose agents as HTTP APIs — OpenAI-compatible chat completions and MCP protocol. Deploys a long-lived web-proxy sidecar with bearer-token auth and rate limiting. See Web Endpoint Skill.

The chart installs them automatically. To apply them by hand from a checkout, use the config/skills/ directory:

kubectl apply -f config/skills/

Or enable them through the TUI edit modal (press e on your Agent, go to the Skills tab).

Channels¶

Connect your agent to a messaging platform so you can interact over chat:

Channel	How to connect
Telegram	Create a bot with @BotFather, get the token, pass it during onboarding or set it in the Agent channel config.
Slack	Create a Slack app with Socket Mode enabled, add the bot/app token during onboarding.
Discord	Create a Discord bot, grab the token, and connect it during onboarding.
WhatsApp	Use the WhatsApp Business API — Sympozium displays a QR code in the TUI for pairing.

Channels are optional. You can always interact through the TUI or by creating AgentRun CRs directly with kubectl.

Policies at a glance¶

Policy	Who it is for	Key rules
Permissive	Dev clusters, demos	All tools allowed, generous resource limits
Default	General use	All tools allowed except `fetch_url`
Restrictive	Production, security	All tools denied by default, must be explicitly allowed, sandbox required

Heartbeat schedules¶

The heartbeat wakes your agent up periodically to check in — review memory, scan the cluster, or run a standing task.

Preset	Cron	Good for
Every 30 min	`/30 * * *`	Active incident monitoring, SRE on-call
Every hour	`0 * * * *`	General ops, default for most users
Every 6 hours	`0 /6 * *`	Light-touch monitoring, cost-sensitive setups
Daily at 9 AM	`0 9 * * *`	Daily audits, reports, security scans
Disabled	—	On-demand only, no background activity

You can change the heartbeat at any time through the TUI edit modal or by editing the SympoziumSchedule CR directly:

kubectl edit sympoziumschedule <agent>-heartbeat

Creating AgentRuns with kubectl¶

You do not need the TUI to run tasks. Create an AgentRun CR directly:

apiVersion: sympozium.ai/v1alpha1
kind: AgentRun
metadata:
  name: quick-check
spec:
  agentRef: devops
  agentId: default
  sessionKey: "quick-check-001"
  task: "How many nodes are in the cluster and what are their roles?"
  model:
    provider: openai
    model: gpt-4o
    authSecretRef: my-openai-key
  skills:
    - skillPackRef: k8s-ops
  timeout: "5m"

kubectl apply -f quick-check.yaml
kubectl get agentrun quick-check -w   # watch status.phase

The phase transitions: Pending → Running → Succeeded (or Failed).

To run the same task in a hardware-isolated Celln cell instead of a pod, set backend: celln and select a Celln backend's wrapper Agent and runtime; see Celln Backend.

Creating custom Ensembles¶

You can create your own Ensemble to bundle a set of agents tailored to your team. Save this as a YAML file and apply it:

apiVersion: sympozium.ai/v1alpha1
kind: Ensemble
metadata:
  name: my-team
spec:
  description: "Custom agents for my team"
  category: custom
  version: "1.0.0"
  agentConfigs:
    - name: log-analyzer
      displayName: "Log Analyzer"
      systemPrompt: |
        You are a log analysis specialist. You parse structured and
        unstructured logs to identify errors, anomalies, and trends.
      skills:
        - k8s-ops
      schedule:
        type: sweep
        interval: "1h"
        task: "Scan pod logs across all namespaces for ERROR and FATAL entries from the last hour."
      memory:
        enabled: true
        seeds:
          - "Focus on patterns that repeat across multiple pods"
    - name: doc-writer
      displayName: "Documentation Writer"
      systemPrompt: |
        You are a technical writer. You review cluster configuration,
        CRDs, and RBAC policies, then produce clear documentation.
      skills:
        - k8s-ops
        - code-review
      schedule:
        type: scheduled
        cron: "0 8 * * 1"
        task: "Audit all namespaces and produce a weekly cluster inventory report."
      memory:
        enabled: true

kubectl apply -f my-team-ensemble.yaml

The pack appears in the TUI Ensembles tab in Pending phase. Press Enter to activate it with your API key — the controller does the rest.

Troubleshooting¶

NetworkPolicy blocks API server on non-standard ports (k3s)¶

Sympozium's default network policies allow egress on ports 443 and 6443 for Kubernetes API server access. On clusters where the API server listens on a non-standard port (e.g. k3s with https-listen-port: 6444), the kubernetes ClusterIP service maps 443 → 6444. Some CNI implementations (notably kube-router in k3s) evaluate egress rules after DNAT, so the actual destination port seen by the policy is 6444 — which is not in the default allow list.

Symptoms: kubectl commands from skill-k8s-ops sidecars fail with The connection to the server <ip>:443 was refused.

Fix: Add the non-standard port to networkPolicies.extraEgressPorts in your Helm values:

networkPolicies:
  enabled: true
  extraEgressPorts: [6444]

Then upgrade:

helm repo add sympozium https://deploy.sympozium.ai/charts && helm repo update
helm upgrade sympozium-crds sympozium/sympozium-crds -n sympozium-system
helm upgrade sympozium      sympozium/sympozium \
  -n sympozium-system --skip-crds --set createNamespace=false -f values.yaml

What's next¶

Expose agents as HTTP APIs — see Web Endpoint Skill
Write a custom SkillPack — see Writing Skills
Add a new tool — see Writing Tools
Write integration tests — see Writing Integration Tests
Run agents in hardware-isolated cells — see Celln Backend
Read the architecture — see Architecture