Production Checklist

Production checklist

Use this before exposing Kazma beyond loopback. Aligns with AUDIT_DEEP_2026-09-01_EXEC.md (waves 0–8 shipped) and the historical REMEDIATION_PLAN_2026-07-21.md (Phases 0–4). Invariants: AGENTS.md.

Feature smoke (research / KB / proxy / memory explain): see Smoke test matrix — run after related deploys.

Is the install actually alive? Full-system battery — chat tools (A), HTTP/ops (B), one HITL write (C). A green Part A is not a green Kazma.

P0 — Secrets & bind

KAZMA_HOST is intentional (127.0.0.1 or proxy-only 0.0.0.0)
KAZMA_SECRET is strong, unique, not the historical kazma-local-dev-secret
Non-loopback bind fails closed without secret (CLI / serve)
KAZMA_PRODUCTION=1
KAZMA_VAULT_KEY set
Healthcheck uses /health/live (process) and /health/ready (critical deps). /health/details is auth (leaks model/MCP). Poll /health/deep as the silent-no-op canary (open, TTL 30s)

P0 — Lifecycle & fail-closed

Graceful shutdown drains swarm/cron (no orphan tasks)
NullBus denies danger tools (no headless auto-approve)
YOLO disabled unless KAZMA_ALLOW_YOLO=1 (avoid in real prod)
Circuit breaker half-open probe semantics intact (_probe_in_flight)
Deploy via kazma_guard.py --reload — never hand-kill python/uvicorn (the guard respawns the old port holder)
Behind nginx/Caddy/Docker: KAZMA_TRUSTED_PROXIES set; peer 127.0.0.1 is not a credential
CSRF on: mismatched Origin/Referer host → 403 on non-GET /api/*

P1 — Security depth

Persisted workspace and execution policies

An administrator can use the authenticated PUT /api/settings atomic batch instead of editing an installation’s environment file:

[
  {"key": "security.workspace_roots", "value": ["/srv/kazma-projects"]},
  {"key": "security.execution.container_required", "value": true}
]

Choose existing absolute directories on the server. Multiple explicit roots can retain an existing workspace and confine new repositories separately. Resolved symlinks must stay inside a configured root. If KAZMA_WORKSPACE_ROOT is present, a path must also stay inside that environment root; a Settings save cannot expand it. Production without either policy refuses workspace creation and switching. Missing or invalid policy storage fails closed.

The strict container profile requires Docker for python_exec, skips E2B, disables shell_exec, and forbids local fallback even when legacy environment flags allow it. It also refuses the local process profile for document parsing, OCR, conversion and PDF mutation (worker_isolation_required): choose the container profile in Settings → Documents (on a native install the guard runs the document worker broker; see Document processing, “Isolated document workers”). Native file and Git tools retain their approval gates. Check the actual Docker daemon and run an approved disposable snippet before qualifying execution; daemon readiness alone does not verify the image or mount permissions. The strict profile makes code_execution a required readiness capability automatically; an unavailable Docker daemon refuses traffic. These install policies are protected from agent config tools and excluded from portable settings backups with the other security.* values.

Runtime approval notes distinguish an authorization from a successful tool result. Workspace-bound MCP servers use separate scoped instances when a task targets a different root; the global binding does not describe that instance. This supplies verified facts to the model, without guaranteeing every freeform answer it generates.

Multi-user / active-passive recovery (if applicable)

KAZMA_DATABASE_URL set; migrate script run
Opaque sessions / RBAC admin user created
OIDC vars if SSO
KAZMA_PUBLIC_URL correct for redirects
HA compose / LB retains one active owner; Postgres and the complete state volume are backed up and restored together
Memory: tenant scoping is automatic with KAZMA_PRODUCTION=1; set KAZMA_MEMORY_ENFORCE_TENANT=1 for a multi-tenant install that is not flagged production
Before KAZMA_MEMORY_STATE_ROLE=primary: python scripts/reconcile_memory_mirror.py --dry-run is clean (no dead-in-mirror rows)
Document catalog, job and intake tables are on KAZMA_PG_TABLES when the document store is on Postgres (H-13)

Document Intelligence (if enabled)

Guide: Document Intelligence · Ops: Document processing.

Upgrades (git install)

Operators use kazma update (not ad-hoc git pull) — see Kazma Update
After update: kazma --version / HEAD matches origin/main; import kazma_cli works; kazma serve starts
Optional extras still present (rag / document-platform); use kazma update --reinstall -y if wiped

Smoke

& .venv\Scripts\python.exe scripts\smoke_production.py --base http://127.0.0.1:9090 --secret $env:KAZMA_SECRET
# Document platform (optional but recommended when documents.enabled):
& .venv\Scripts\python.exe scripts\certify_documents.py
# Optional industry smoke (HITL card, health/deep):
# scripts\industry_smoke.ps1

Confirm GET /health/deep is 200 after reload. HITL: one Approve card, second click 409.

Also run document rows in the Smoke matrix.

Commitment Layer

Kill-switch verified: KAZMA_COMMITMENT_ENABLED=0 disables the gate (fail-open)
GC cadence running: _start_commitment_gc_scheduler every 15 min (check logs for commitment GC: summary)
Flags: swarm_scope_enforce defaults ON (workers capped at HIGH since 2026-08-15); enforce_unknown_mutators defaults ON; soul_requires_confirm defaults OFF on a lab and auto-ON in production / multi-user — operators toggle via ConfigStore / env
Metrics endpoint: GET /metrics shows kazma_commitment_decisions_total{decision=...} + kazma_commitment_pending
Soul confirm queue: GET /api/commitment/soul/pending lists held deltas; POST /api/commitment/soul/{cid}/confirm approves
Semantic-tool exceptions fail closed (kill-switch still fail-open)

HITL Gate Registry & Turn Delivery

kazma-data/hitl_gates.db is writing rows (pending → claimed → settled)
Web paints from _serverGates — no ghost Approved stamp; second claim is 409
Swarm FanOut (2+ platforms) is tri-state (Deny is a vote, not an instant settle)
KAZMA_GATE_REGISTRY=0 is not set in prod (thin execution fallback only)
close_turn keeps the turn open while a pending row exists

Backup & ops alerting

Universal + native_pg_backup cadence is 6h (not 24h); local PG staging retention default 3 (restic keeps history)
With Postgres, the weekly deep drill’s restore rehearsal is on (default since 2026-09-27) and the database role has CREATEDB; the drill log names postgres:restore
Chat step history retention is what you want (Settings → System; default 30 days, 0 keeps every checkpoint). An install upgrading with years of history: set it before the upgrade if you want to keep them
Universal backup checks PG dump freshness — a missing/stale dump is not "ok": true
ops_alerts reach Telegram (and FanOut if Discord/Slack configured): Settings → Adapters & Routes → Send a test alert names each route that took it; kill-switch KAZMA_OPS_ALERTS=0 only during a noisy incident
Guard Telegram is separate (child-down). 503 cause-quality (database: … vs Service Unavailable) is deferred: GUARD_OPS_ALERTING_CAUSE_QUALITY.md