Observability API (/api/v1)¶

leo_surface_file, the snapshot's surfaced_files and the file_surfaced event were removed (2026-10-12); show a file with leo_chat_post instead — see Leo Chat.

Read-only HTTP API exposing live agent and task state from the Leo daemon. Consumers: The Den (pixel-office visualizer, separate repo), leoterm, the macOS app, and anything else that wants to watch the fleet. See 2026-07-31-den-pixel-office.md for the concept that motivated it — but nothing in this API is Den-specific.

Two endpoints on the daemon's existing TCP (web) listener:

• GET /api/v1/state — full snapshot of the world right now.
• GET /api/v1/events — SSE stream of changes to that world.

The contract is snapshot-then-stream: a consumer fetches the snapshot, then applies events. Every event carries enough state to be applied without a refetch.

Design rules¶

• Read-only. No mutations here; control stays on the existing endpoints.
• Consumer-agnostic. No rooms, sprites, weather, or camera concepts. Those belong to the consumer.
• Additive-only evolution. Consumers must ignore unknown fields and unknown event types. New fields and event types are not breaking changes; removals and renames are, and require /api/v2.
• Cheap. The snapshot must be servable from in-memory state plus the agentstore; no shelling out to tmux per request.

GET /api/v1/state¶

Wrapped in the existing apiResponse envelope ({ok, data, error}) used by every other /api route:

{
  "ok": true,
  "data": {
    "version": 1,
    "server_time": "2026-07-31T18:40:06-04:00",
    "leo_version": "0.10.3",
    "agents": [ /* Agent */ ],
    "tasks":  [ /* Task */ ],
    "recent_runs": [ /* TaskRun */ ],
    "recent_messages": [ /* AgentMessage */ ]
  }
}

GET /api/v1/events is the one exception: SSE frames are written directly, with no envelope.

Agent¶

{
  "name": "den",
  "display_name": "den",
  "template": "fable",
  "repo": "blackpaw-studio/leo-den",
  "workspace": "/Users/alice/.leo/agents/leo-den",
  "branch": "main",
  "status": "running",
  "activity": "working",
  "model": "fable",
  "harness": "claude",
  "restarts": 0,
  "wake_on_message": false,
  "started_at": "2026-07-31T18:42:33-04:00",
  "last_activity_at": "2026-07-31T18:44:01-04:00",
  "current_action": { "kind": "pane", "detail": "Running go test ./..." }
}

• display_name — name without the internal leo- prefix (leo-home → home), for UIs to show. Additive: name stays the raw stored/tmux name and is the key for routes. Present on every agent object (/state, /events agent_spawned).
• status — lifecycle, from the agent record: starting | running | stopped | restoring. restoring is reported only while the daemon is still restoring agents at boot (before /health is ready) for a stored agent that is neither live nor deliberately stopped; it is never wake_on_message, and after the daemon is ready it never appears. Daemons that advertise the agent_status_restoring capability (/events hello, /api/v1/version) emit it; older daemons report such an agent as stopped, so a client must treat any status it does not recognize as stopped. An internal-only crash-loop-backoff state (restarting) folds into starting, the closest lifecycle equivalent a consumer can act on. Leo has only one dormant state; a suspended-then-resumable agent and a deliberately-stopped one both report stopped — see wake_on_message to tell them apart.
• wake_on_message — always present, meaningful only when status is stopped: true means the agent was parked by the idle sweep and an inbound message auto-starts it again; false means it is either not dormant at all, or was stopped deliberately (by an operator, or by the daemon after a failed restore) and stays down until explicitly started. status and wake_on_message are always set together and can never disagree — wake_on_message is never true for a non-stopped agent.
• activity — live work state, from the activity tracker: working | idle | unknown. Orthogonal to status: a running agent may be idle. Non-running agents report unknown.
• last_activity_at — when the agent's tmux session last produced output or received input.
• current_action — best-effort, human-readable hint at what the agent is doing, or null. Only sampled for agents that are currently working.
• kind describes where detail came from. pane is the only kind Leo emits today: the last non-empty line of the agent's tmux pane, with ANSI and control characters stripped, truncated to 120 characters. The field exists so a future structured source can be added without reshaping the object, so consumers must render an unknown kind by falling back to displaying detail as plain text rather than dropping the action.
• detail is whatever the harness happens to be rendering: untrusted display text. Escape it before rendering, never parse it or branch on its contents, and handle it being absent or garbage.
• model / harness — resolved values after the defaults→template→agent cascade.
• environments, environments_source, environment_error — advertised by the agent_environments feature; see Named environments below.
• appearance — {emoji?, color?, emoji_source?, color_source?}, absent when neither is set; advertised by the agent_appearance feature. See Agent appearance below.
• moved_to, moved_at — set on the dormant record an agent leaves behind when it is moved to another host: moved_to is the client.hosts key it went to, moved_at the RFC 3339 time it was marked. Both are absent for an agent that lives here, and status stays stopped. Advertised by the agent_move feature; see Moving and copying agents.

agent_spawned's embedded Agent (see the event table below) is populated the same way, sourced from the agentstore record and the same model cascade — template, repo, and branch come from the agentstore record for that agent name (persisted before spawn, so it's reliably present), and model from resolving that record's template through config. A record or config that can't be loaded at spawn time (e.g. no config file present) degrades those fields to empty/zero rather than guessing — a consumer that needs them reliably should treat agent_spawned as a hint and fall back to GET /api/v1/state if any of template/repo/branch/model come back empty for an agent it cares about.

Agent retention¶

agents lists every agent Leo knows about, including stopped ones. There is no time-based aging out: an agent stays in the snapshot until it is explicitly removed from the agent store (leo agent reset/remove), so a long-stopped agent keeps appearing indefinitely with status: "stopped" and activity: "unknown". Consumers that only care about live agents must filter by status themselves.

Consequently, an agent disappearing from the snapshot means it was deleted, not that it merely stopped — those are different situations and consumers should treat them differently.

How activity is derived¶

tmux.ListSessionActivity already reports session_activity (an epoch that advances on pane output, injected input, and interactive typing) plus attached-client count for every session in one list-sessions call. The tracker sweeps on a ticker (~2s):

• session_activity advanced since the previous sweep → working, and last_activity_at moves.
• no advance for longer than the idle threshold (15s) → idle.
• no tmux session, or agent not running → unknown.

One tmux call per sweep covers the whole fleet; a capture-pane is issued only for agents that just advanced, so idle agents cost nothing. Deriving activity from tmux rather than from Claude's session JSONL keeps this harness-agnostic — it works identically for claude, codex, and opencode — and avoids coupling Leo to another tool's private on-disk format.

Task¶

Config-level description of a scheduled task: name, schedule, timezone, enabled, runtime (oneshot | persistent), template (persistent only), workspace, model, harness, last_run_at, next_run_at.

TaskRun¶

A single firing: id, task, status (running | succeeded | failed), started_at, ended_at (null while running), duration_ms, error (null unless failed). recent_runs is capped (default 50, newest first).

A run also carries workspace, model, and harness — the values resolved for that firing. They are deliberately denormalized rather than left as a join through tasks[], because the run producer already holds them and the join is not always available: a task can be renamed, disabled, or deleted while one of its runs is still in flight, and a task_run_* event can reach a consumer before it has ever fetched a snapshot. Carrying them makes a run self-describing.

Both oneshot and persistent tasks publish the full task_run_started / _succeeded / _failed sequence with identical fields — a persistent task's firing is just as visible as a fresh claude -p invocation. The one semantic difference: a persistent task's error on a failed run is a free-form diagnostic string (e.g. "enqueue: ...", "rejected: ...", "await: ...", "task: ...") describing which stage of session-router dispatch failed, rather than the oneshot path's history reason vocabulary (timeout, channel-init, etc.) — both are just display text for error, never something a consumer should parse or branch on.

recent_runs is assembled by merging two sources, newest first, deduplicated by id, capped at MaxRecentRuns:

• The run log — a bounded, in-memory record of runs as they pass through the event publisher. It is the only source that knows about a currently-running firing, and it carries the honest wall-clock timing the producer itself measured.
• Task history (durable, on disk) — tops up older completed runs the run log has already evicted or never saw (e.g. right after a daemon restart, when the run log starts empty). History entries recorded before a run's start time and duration were tracked have neither: started_at falls back to the entry's completion timestamp (the only one those old entries have) as a best-effort value, since the field is mandatory, but ended_at and duration_ms are omitted rather than fabricated — a completed run must never report started_at == ended_at as a stand-in for "we don't actually know".

AgentMessage¶

{ "from": "chronicle", "to": "plex", "at": "2026-08-03T19:47:10-04:00" }

One agent-to-agent message: who, to whom, when. Never the message content — this API reports that two agents are talking, never what about. recent_messages is capped (default 50, oldest first) and drops entries older than 10 minutes, so a consumer connecting mid-conversation can seed the pairs it missed instead of waiting for the next message. Consumers wanting a live signal use the agent_message event; recent_messages exists for load and reconnect.

from is omitted when the sender is not an agent — a human messaging from the web UI has no agent identity, and leo does not invent one. A consumer that wants agent-to-agent activity specifically should require both from and to.

from is self-asserted by the sending agent: it is the sender's own process name, passed through as a field, not an authenticated identity. It is fine for display and animation; it must not be used for authorization or attribution.

Task-run ingress (internal, not part of this API)¶

recent_runs/task_run_* events reach the daemon two ways: in-process (the supervisor/session router publish directly), and over IPC from a leo run subprocess via POST /observe/task-run on the daemon's existing 0600 Unix socket — a different listener from /api/v1, with no bearer-token auth (the socket's filesystem permissions are the access control) and not reachable from the TCP web listener. It exists solely so a subprocess with no in-process handle on the daemon's RunLog can still announce a run.

The daemon does not trust this input verbatim:

• Run.Status is never taken from the request — it is derived server-side from Type (a 1:1 mapping), so a run's status can never disagree with the event type a consumer receives it under.
• Run.ID and Run.Task are required; either being empty is rejected with 400 rather than forwarded (an empty ID would otherwise collapse every such run into a single RunLog slot, since records are matched by ID).
• The request body is capped at 64 KiB (http.MaxBytesReader), and Run.ID/Run.Task/Run.Model/Run.Harness are truncated to 200 characters, Run.Workspace to 1024, and Run.Error to 4096 — the same untrusted-display-text discipline current_action.detail already gets, since this input is retained (up to MaxRecentRuns entries) and rebroadcast to every /api/v1/events subscriber, not discarded after one use.
• A rejected or truncated event is invisible to the task that produced it: PublishTaskRun treats every response as best-effort, so observability being unavailable, rejecting, or truncating never fails or slows a leo run invocation.

GET /api/v1/events¶

text/event-stream. Each message is a named SSE event with a JSON payload:

event: agent_activity
data: {"seq":184,"at":"2026-07-31T18:44:01-04:00","agent":"den","activity":"working",
       "current_action":{"kind":"pane","detail":"Running go test ./..."}}

Every payload carries a monotonic seq and an at timestamp. On connect the server sends a hello event whose seq is the sequence number of the last event already published — i.e. the first event a consumer receives afterwards is guaranteed to be hello.seq + 1, with no gap — and whose at is a real timestamp, never the zero value. If no event bus is wired at all, hello.seq is 0 (there is nothing to report a sequence for), but at is still a real timestamp. A consumer whose stream drops should reconnect and refetch the snapshot rather than trying to replay gaps (no event history is retained). A comment heartbeat is sent every 20s so idle proxies don't close the stream.

Event types:

A resume is represented as agent_state_changed with the new status; consumers key off status, not a distinct event name. agent_stopped's wake_on_message carries the same meaning as the snapshot Agent's field above: true only for an idle-sweep park, false for every other departure — a manual stop with no wake requested, a deleted agent, a rename's old name disappearing, or a transient kill ahead of an immediate respawn (leo agent reset, leo agent restart, and leo agent set-template all issue StopAgent(name, false) before respawning under the same name).

At the instant agent_stopped lands, no field distinguishes "about to respawn" from "parked indefinitely" — both look identical on the wire (wake_on_message:false). A consumer that needs to tell them apart must wait for the follow-up event: a genuine departure has none, while reset/restart/set-template are followed shortly by agent_spawned (or agent_state_changed with status:"starting") for the same agent name. Treating a bare agent_stopped as a permanent departure without waiting for that follow-up will misreport routine restarts as agents leaving.

agent_message announces that one agent messaged another, and carries the pair only — there is no field for the message body and none may be added. See AgentMessage below for the from and trust semantics, which apply identically to the event.

Renaming a running agent (leo agent rename) is represented as agent_stopped for the old name followed by agent_spawned for the new one, in that order, rather than a dedicated rename event — a consumer that only tracks agents by name doesn't need a new event type to stay correct, it just sees the old name leave and the new one appear. The agent_spawned payload's template/repo/branch/model are empty at this point (the agentstore record is re-keyed to the new name only after the rename completes), degrading the same way a spawn does when those sources aren't available yet — see the note above. workspace and harness are also empty here — the supervisor's in-memory process state carries neither, and reloading the (still-under-the-old-name) agentstore record to fill them in isn't worth the extra I/O on this already-degraded, best-effort payload. A consumer that needs any of template/repo/branch/model/workspace/harness reliably should treat every field on a rename's agent_spawned as a hint and fall back to GET /api/v1/state — the same fallback the spawn note above already establishes.

A live rename's agent_spawned carries no appearance either; an agent_appearance_changed for the new name follows it once the record has moved.

Renaming a stopped agent publishes the same agent_stopped/agent_spawned pair, but sourced from the agentstore record directly (rather than the supervisor's in-memory state), so template/repo/branch/workspace/model/harness are populated exactly as they would be for a live agent in GET /api/v1/state — the record hasn't been re-keyed away from the source data yet at publish time the way the live-rename path's process state has.

Slow consumers are dropped rather than buffered without bound: each subscriber gets a bounded channel, and a subscriber that fills it is disconnected (it will reconnect and resnapshot).

Named environments¶

Advertised on the SSE hello as the agent_environments feature. Names only: an environment's variables can hold credentials and never cross this API.

• Agent fields (every Agent, in /state and agent_spawned):
• environments — the effective ordered names: the agent's own override, else its template's (or task's) environments, else defaults.environments. Always an array.
• environments_source — "override" (the agent was given its own list at spawn or by set-environment) or "default".
• environment_error — a string or null. Non-null when a name in the list is no longer defined in config, so a restart would be refused; computed for live and stopped agents alike, with the check restart, start and the stale-agent report share.
• agent_state_changed carries environments / environments_source when set-environment restarted the agent, so a subscriber updates the row without refetching. That event always carries all three of environments ([] when the effective list is empty), environments_source and environment_error (an explicit null when it resolves). Ordinary lifecycle state changes carry none of them: absent means "leave the row alone".
• GET /api/v1/environments → {"ok":true,"data":[{"name":"base"},{"name":"work"}]}. Alphabetical: the config map loses YAML order at parse and every web-UI save rewrites it sorted, so only a sorted list is stable. No harness field: an environment is a bare env map any template can run under.
• GET /api/v1/templates → data: [{"name","harness","model","environments":[...]}], sorted by name; environments is the template's own default list ([] when it sets none, in which case defaults.environments applies). A template's env is never included.
• POST /api/v1/agents/spawn — the /api/agent/spawn body (template, repo, name, branch, base, prompt, env) plus environments: ["a","b"], overriding the template's list for this agent.
• POST /api/v1/agents/{name}/environments {"environments":["a","b"]} — switch a live or stopped agent: its env is rebuilt, it restarts and its conversation is resumed where the new account can read it. An empty list clears the override. See Environments.

Both writes require the operator token, like the other /api/v1/agents/... writes (the agent token is refused with 403). Failures are {"ok":false,"error":"...","code":"..."}:

config_unavailable (500) means the config would not load; the cause is in the daemon log only. Other failures (unknown agent 404, malformed body 400) carry no code.

Agent appearance¶

Advertised on the SSE hello as the agent_appearance feature. An agent's optional emoji and palette color, set by hand or by ordered appearance_rules, are resolved by the daemon; see Agent Appearance for the rules, the resolution order and the routes.

• Agent field (every Agent, in /state and agent_spawned): appearance — {emoji?, color?, emoji_source?, color_source?}, a source being "manual" or "rule:<i>". Unset values are omitted, and the key is absent when nothing is set.
• agent_appearance_changed — agent, appearance (an explicit null when nothing is set). Published after a manual set, for every agent after each config reload (no diffing: apply it as-is), after set-environment and set-template, and after a live rename.
• GET/PUT /api/v1/appearance/rules (operator or paired device token) read and replace the rules; PUT /api/v1/agents/{name}/appearance {"emoji"?, "color"?} sets an agent's manual values (null clears, absent is unchanged).

Access¶

Both routes register on the existing apiMux and inherit its conventions unchanged: bearer-token auth (Authorization: Bearer <token>) plus the global host/origin gating. No new auth mechanism, no new middleware.

Consumers are server-side, not browsers. Leo adds no CORS headers and no exemption to hostOriginMiddleware. A browser-direct consumer would force two bad outcomes — the bearer token shipped into page source where any viewer can read it, and EventSource cannot set an Authorization header at all. So a web consumer like The Den ships a thin server that holds the token, consumes this API server-side, and re-exposes what its own browser client needs on its own origin. That keeps the token server-side and leaves Leo's security posture untouched.

Testing¶

• Activity tracker: unit tests over the working/idle state machine and the pane-line sanitizer, driving the existing activityExecCommand seam with fixture output — no live tmux.
• Event bus: subscribe/publish fan-out, unsubscribe, and slow-consumer drop.
• Endpoints: httptest snapshot shape, SSE framing, heartbeat, and auth rejection.