Production Checklist
Production checklist
Section titled “Production checklist”Use this before exposing Kazma beyond loopback. Aligns with docs/audits/REMEDIATION_PLAN_2026-07-21.md (Phases 0–4 shipped in code).
Feature smoke (research / KB / proxy / memory explain): see Smoke test matrix — run after related deploys.
P0 — Secrets & bind
Section titled “P0 — Secrets & bind”-
KAZMA_HOSTis intentional (127.0.0.1or proxy-only0.0.0.0) -
KAZMA_SECRETis strong, unique, not the historicalkazma-local-dev-secret - Non-loopback bind fails closed without secret (CLI / serve)
-
KAZMA_PRODUCTION=1 -
KAZMA_VAULT_KEYset - Healthcheck uses
/health(or live/ready as documented)
P0 — Lifecycle & fail-closed
Section titled “P0 — Lifecycle & fail-closed”- Graceful shutdown drains swarm/cron (no orphan tasks)
- NullBus denies danger tools (no headless auto-approve)
- YOLO disabled unless
KAZMA_ALLOW_YOLO=1(avoid in real prod) - Circuit breaker half-open probe semantics intact
P1 — Security depth
Section titled “P1 — Security depth”- Auth default-deny on
/api/* - Discovery SSRF protections enabled
-
code_execDocker force where required (KAZMA_CODE_EXEC_DOCKER=force; host-local disabled in prod/multi-user) - Untrusted / multi-user code: E2B Firecracker (
E2B_API_KEY+pip install 'kazma[sandbox]';KAZMA_E2B=0to disable) - Multi-hour swarm: Temporal (
KAZMA_TEMPORAL_HOST+pip install 'kazma[durable]') - Multi-replica: unique
KAZMA_REPLICA_ID+ LB sticky onkazma-replicacookie - Shell allowlist + env scrub active
- Workspace root confinement in production
- Cron concurrency / stop / stale RUNNING handled
- Multi-operator: platform allowlists set +
KAZMA_GATEWAY_STRICT_ALLOWLIST=1(2026-08-19; adapters otherwise run allow-all for backward compat) -
KAZMA_HITL_CANONICAL_FLOOR=1on strict deployments (danger list cannot narrow below canonical; 2026-08-19) - Offsite/cloud-sync backups verify TLS (WebDAV default ON since 2026-08-19;
backups.offsite.webdav.tls_verify=falseonly for self-signed labs)
Multi-user / multi-replica (if applicable)
Section titled “Multi-user / multi-replica (if applicable)”-
KAZMA_DATABASE_URLset; migrate script run - Opaque sessions / RBAC admin user created
- OIDC vars if SSO
-
KAZMA_PUBLIC_URLcorrect for redirects - HA compose / LB only if Postgres shared state verified
- Memory:
KAZMA_MEMORY_ENFORCE_TENANT=1when more than one tenant exists - Before
KAZMA_MEMORY_STATE_ROLE=primary:python scripts/reconcile_memory_mirror.py --dry-runis clean (no dead-in-mirror rows)
Document Intelligence (if enabled)
Section titled “Document Intelligence (if enabled)”-
documents.enabledintentional (default compatibility: enabled, not default-authoritative) - Do not run multi-replica against a shared document store until metadata is Postgres — check
GET /api/documents/ops/readiness -
documents.capacity.storage_free_floor_bytesset conservatively (default 512 MiB) - Nightly document backup path known (
kazma-data/backups/document-store-*or equivalent) - Migration plan includes
documents.db+ content tree (Migration) - Optional engines understood (fitz / WeasyPrint / LibreOffice may be CONDITIONAL)
- PDF Arabic / electronic text:
pip install -e ".[document-platform]"(PyMuPDF + pypdfium2); parser readiness should be ready, not text-only degraded - Scanned Arabic PDFs: system Tesseract +
ara(andeng) traineddata on PATH - Malware: install ClamAV (
clamscan/clamdscanon PATH); considerdocuments.security.malware_scan=on+ fail-closed; check readinessmalware.available - Cert smoke:
python scripts/certify_documents.pyexits non-FAIL; record report if promoting canary
Guide: Document Intelligence · Ops: Document processing.
Upgrades (git install)
Section titled “Upgrades (git install)”- Operators use
kazma update(not ad-hocgit pull) — see Kazma Update - After update:
kazma --version/ HEAD matchesorigin/main;import kazma_cliworks;kazma servestarts - Optional extras still present (rag / document-platform); use
kazma update --reinstall -yif wiped
& .venv\Scripts\python.exe scripts\smoke_production.py --base http://127.0.0.1:9090 --secret $env:KAZMA_SECRET# Document platform (optional but recommended when documents.enabled):& .venv\Scripts\python.exe scripts\certify_documents.pyAlso run document rows in the Smoke matrix.
Related ops
Section titled “Related ops”- Postgres & SaaS
- Multi-replica & SaaS residual
- Disaster recovery
- Multi-region
- OIDC
- Document processing
- Environment variables
Commitment Layer
Section titled “Commitment Layer”- Kill-switch verified:
KAZMA_COMMITMENT_ENABLED=0disables the gate (fail-open) - GC cadence running:
_start_commitment_gc_schedulerevery 15 min (check logs forcommitment GC:summary) - Flags:
swarm_scope_enforcedefaults ON (workers capped at HIGH since 2026-08-15);soul_requires_confirm,enforce_unknown_mutatorsdefault OFF — operators toggle via ConfigStore / env - Metrics endpoint:
GET /metricsshowskazma_commitment_decisions_total{decision=...}+kazma_commitment_pending - Soul confirm queue:
GET /api/commitment/soul/pendinglists held deltas;POST /api/commitment/soul/{cid}/confirmapproves