Recent features guide

Recent features guide

New in 2026-10-16 through 2026-10-21 — approvals, forgetting, documents and tools that answer in one call. An X card offers one approval and Deny, names what it does (“Post now”), and a draft the agent saves is posted only after you have seen it, in a later message. A shell command or browser call the host would refuse gets no card, and a refusal before anything ran is an ordinary error the agent corrects. An unanswered card expires as “expired”, not “denied”; in Telegram, Discord and Slack the chat is told and gets the rest of the turn. /stop is /abort, and an aborted request is answered “cancelled”. Asking in chat to forget a fact forgets the turn it came from, and “forget everything that mentions X” reaches every memory, fact and lesson holding it. Generated documents and HTML converted to PDF share one house style, each table reads in its own language, and Kazma reads its own bilingual PDFs back as they are. Two read-only git tools, git_log and git_diff, answer “what changed” without the shell; check_swarm_task waits for the task (60 seconds by default, at most 90) and answers the moment it finishes; a Gmail listing shows each mail’s sender, date and subject. A crawl or pipeline the agent runs in its own turn no longer raises a research alert. The Knowledge Library’s vector store replays a library’s write log on one thread, after a race in Chroma’s parallel replay crashed the process (see Knowledge Library). Dependency minimums were raised on 2026-10-17; an install updated with git pull reports the packages it is behind at boot until python -m kazma_cli update --reinstall -y runs (see Updating Kazma).

New in 2026-10-13 — documents: refusal, recovery and measured limits. PDFs with scripts, outside actions, attachments or links to files and network shares are refused on upload, on every edit and on every PDF Kazma renders, always inside the document worker. A Web upload is answered before its body is read when a limit would refuse it; an upload interrupted by a crash is rolled back, and one whose owner stops renewing its lease is taken over and fenced. Every job read checks access, and an outage (scanner, disk, database) is retried, never recorded as bad content. A Postgres document store is backed up with its files as one point in time and restored only into an empty database; searches are re-published at the next start. Arabic PDFs read numbers, ligatures and table cells in order, and ruled tables on scans are read cell by cell. An optional container worker profile runs each job in its own container with no network or mounts. Limits: the default process profile does not confine files or network and is not qualified for hostile documents; Arabic OCR with Tesseract is not qualified (it does not read Arabic-Indic digits). Since then, the neural OCR engine (2026-10-15) meets the release targets on every OCR case of the corpus; an independent review closed on 2026-10-15 with one finding fixed; and a 48-hour soak passed on 2026-10-19 on the process profile with SQLite stores. See Document Intelligence and the hardening tracker.

New in 2026-10-10 through 2026-10-12 — local-first research with visible outcomes. Page acquisition defaults to local HTTP/extraction and optional local Chromium. Hosted extractors and proxies require an explicit mode and provider allowlist; saved API keys alone do not enable them. Research, crawl and Knowledge jobs retain bounded durable journals, preserve uncertain effects and require explicit recovery after interruption. Automated crawls enforce robots checks by default and share origin limits and cooldowns.

Completion now creates a persistent notification linking to the saved run and results. Partial, failed, cancelled and interrupted runs have distinct alerts; no observed progress produces a warning, and resumed progress clears it. Settings → System → Web acquisition controls monitoring and completion alerts, the no-progress threshold (default 180 seconds) and fresh low-credit threshold (default 100). Cached errors, usage, request reservations and balances are visible without provider calls. Unsupported balances and charges remain unknown; credit refresh is an explicit permitted action. External notification receipts distinguish platform acceptance from a requested or uncertain send.

Local-only acquisition does not make AI planning or synthesis offline: those steps use the configured model. Strict Chromium isolation is qualified on Linux with the required host controls; Windows strict rendering refuses launch, and Windows compatibility endurance remains unqualified. Local HTTP remains available. See Web research, Configuration and Known gaps.

New in 2026-10-10 — final answers stay final across chat apps. Telegram, Discord and Slack deliver the shared graph’s answer without adding a greeting or sign-off afterward, and save the completed reply once. Native live checks covered one Arabic sentence, a bare JSON object and two English bullets.

New in 2026-10-09 — answer-quality safeguards and review. Ambiguous bare tool-argument text receives at most one model recheck; text is never executed as a guessed tool call. Full attachment paths remain distinct. Explicit English/Arabic one-paragraph prose requests and verbatim supplied JSON copies receive conservative terminal formatting checks. Code delimiters render correctly in Web and Telegram, and saved history keeps the authoritative final reply after a recheck or disconnect. The administrator dashboard reports bounded structural quality signals without storing prompt or answer text; these are not accuracy grades. The owner completed review of 60 development answers and 24 fresh answers: current approved sets are 60/60 and 24/24 Pass, with original 59/60 and 23/24 results retained. This is bounded reviewed evidence, not a production-wide accuracy estimate or automatic-publishing qualification.

New in 2026-10-06 through 2026-10-08 — commands and recovery. Native Discord/Slack /kazma commands and Telegram menu diagnostics use existing authorization and approval gates. Private command replies keep their follow-ups private. IDE retries use durable action receipts; uncertain effects remain held, and a swarm worker cannot automatically replay after a recorded mutation. The IDE protects unsent drafts, workspace changes and Git operations. Docker execution under the Windows service uses the configured workspace correctly. See Native chat commands, IDE and Known gaps for setup and limits.

New in 2026-10-05 — X subject and reply-language controls. Start with a target and Support/Against; Advanced preserves scope, matching and permissions. Choose a tone independently of stance, and set language, dialect, slang and length. Allow uncensored language is off by default and permits selected ordinary profanity while retaining factual checks, hard lines and approvals. Compare saved/proposed previews and test the X model without posting. Conversations show the applied position and tone. These controls change the evaluation fingerprint; real reviewed cases still qualify unattended replies.

New in 2026-10-04 — durable settings and real-model evaluation. If the settings database cannot initialize, Kazma stops before workers or automation start. A failed save returns HTTP 503 instead of reporting success for values held only in RAM. Restore storage, then reload and review the current settings before retrying: a lost acknowledgement does not prove a write rolled back. Configuration and Troubleshooting.

The live evaluation CLI runs an explicitly selected provider/model through the supervisor graph with isolated stores and fixture tools. It records tool attempts, usage and mechanical checks separately from human-reviewed Arabic and English holdout accuracy. Synthetic examples do not establish accuracy or grant automatic-publishing qualification. The default agent prompt now separates source facts from inferences; missing readiness or approval details remain unknown. This improves grounding, but does not certify accuracy. See the operator evaluation workflow and X Studio dataset collection.

New in 2026-09-30 — a database outage is not a restart. When the database went away for two minutes (a Docker Desktop update), Kazma froze waiting on it and the guard restarted it. Now nothing waits on the database on the server’s event loop, the health check answers in seconds whatever its checks do, the guard rides out “Kazma is up, its database is not” and pages you instead, and the connection pools are back seconds after the database is. When the server does exit, its last words are in the page. The weekly resilience report names every mechanism, leads with what needs a look — restarts with their reasons, a scheduled job below its schedule, and what blocked the event loop — and counts alerts sent, not repeats held back. Backup snapshots and repository maintenance no longer collide, and a missed snapshot is reported. Settings → Restart server goes through the guard. Kazma also runs at normal priority: a Windows Scheduled Task registered without one starts it below every normal program on the machine, and it froze whenever something heavy ran beside it. The restart card counts only real downtime, and a Slack connection says why it reconnects and warns when another program shares the app’s token. Deployment §6, Diagnosis map §1.

New in 2026-09-28/29 — more accounts, and chat apps that account for every message. Settings → Email → Other accounts adds more mailboxes — a Google or Microsoft sign-in under a short name you choose, or an app password over IMAP — each with its own calendar; in chat, name the account or its address (Email). Each chat app’s Test (Settings → Providers & Connectors → Platform Connectors) diagnoses the connection: the token, the chat or channel Kazma delivers to, direct messages, allowed users, and what became of the last message a person sent. Every message a chat app receives is either answered or logged with the reason it was not (Gateways). Discord keeps its session through the reconnects Discord asks for. Tables in an answer read on a phone on Telegram, Discord and Slack. A restart sends one status card — how long Kazma was down and whether each chat app connected — instead of three messages (Settings → Adapters & Routes → Server status messages; Deployment). Every page reads in Arabic, the login page included.

New in 2026-09-26/27 — memory you can see and steer. A turn that used memory shows a Memory used row in its activity: what the model was given, in order, stored with the turn (below). Settings → Memory → About me is a short text you write about yourself, read at the start of every reply. Deleting a chat asks whether to also forget what Kazma learned from it; the chat menu’s Memory… keeps a chat out of memory altogether. Recall injects only memories that clear an evidence threshold — nothing when nothing matches — reads weekly topic summaries, and archiving never erases a memory (Memory). Chat step history (the checkpoints undo and resume use) is bounded on Postgres too: every chat keeps its newest 200 steps and a chat idle for 30 days its newest 10 (Settings → System → Chat step history; 0 keeps everything). With Postgres, the weekly backup check restores the newest dump into a scratch database by default (Disaster recovery). Pages stay quiet while the server restarts and keep rendering when an API fails.

New in 2026-09-22/23 — reliability, not features. Chat memories are no longer archived while you still recall them, and an archived one keeps a short stub instead of nothing (Memory — episode lifecycle). The server’s event loop no longer freezes on ~30 background paths (config and session reads, X polling, backups, TLS setup) — the guard’s one forced restart that week was such a freeze. The Telegram weekly resilience report now covers the whole week, counts only real restarts, and includes event-loop stalls. Backups are pruned per kind with 30 daily snapshots (Disaster recovery). Discord and Slack take an Allowed user IDs list (Settings → Connectors); without one Discord accepts everyone and Slack no one. New symptoms and where to look: Diagnosis map §1.

New in 2026-09-19: X auto-reply — mention the connected account and Kazma drafts a reply. Settings cards lock against / support; emoji is tone on a match and side when nothing matches (شرايك or 👍 and 👎 together = opinion on this post). Quotes and status links are the text it reads, even under Kazma’s own tweet. A trusted follow-up under Kazma walks up to the original post; strangers need #Open (#Close stops them). X Studio → Conversations is the log (Refresh polls X). Kill-switch KAZMA_X_REPLY=0. Typed web chat does not auto-play TTS — click 🔊 (Voice).

This page is the operator-facing tour of the features landed in the research → KB → memory polish tranche (including the /memory admin graph/rename/hub work). Use it to turn features on, try them once, and find the deep docs when you need detail.

New in 2026-09-14: The providers page is a control plane — master-detail with state pills, declared capabilities, and a Test that sends a real completion (reachable ≠ chat_ok); /api/providers/* is the one provider surface, the duplicate /api/settings/providers/* is deleted. Prompt injection is measured, not claimed: 996 runs × 4 conditions on AgentDojo — the fence scores 10.4% attack success vs 18.1% undefended, and statistically ties the literature’s 4-character delimiter; a fencing failure now fails closed, and the "Error:" prefix bypass is gone (Security, the full numbers). Backups prove themselves: the restore drill runs daily (5 min after boot, every 24 h from last completed run) and a weekly deep tier reads the bytes back — Postgres data sections streamed through pg_restore, restic check --read-data-subset, and a HEAD read-back of the offsite object. kazma mcp hands the whole tool registry (155 tools) to any MCP client with the real HITL gate — ungated, danger tools are withheld, not refused (CLI). Releases are verifiable: Sigstore signatures, SLSA provenance, CycloneDX SBOM (gh attestation verify). OTel GenAI spans for every LLM call and tool execution (ops guide). Config safety: an empty write can no longer erase a stored secret anywhere, and an approval given in one surface is honoured in all of them.

New in 2026-09-10 (Hands 0.11): Chat is home (/ → /chat; first-run asks for one provider key + model). file_apply_patch_set is one HITL card (optional verify=true runs nearby pytest). Web /ide is CodeMirror 5 fromTextArea (nord) — Monaco is gone. Read-only git is not a danger tool. kazma acp sends structured diffs and honours session/cancel. Docker force blocks host shell_exec unless KAZMA_HOST_SHELL=1.

New in 2026-09-01/02: Turn Delivery V2 (journal + close_turn; the chat bubble projects). HITL Gate Registry (hitl_gates.db — one card, one row, 409 on a second Approve). Industrial audit waves 0–8 shipped (exec). SSRF pin-IP on direct scraping. Context integrity (scratchpad merge, stored proposals, trim summary net, shift_explicit vs inferred). FanOut HITL is tri-state. Reload: kazma_guard.py --reload. Guard 503 cause-quality is deferred.

New in 2026-08-31: X Studio (/x) is the compose and plan surface (Post now, Schedule, reschedule, threads, delete, saved drafts). IBM Plex Sans / IBM Plex Sans Arabic is the shared face across the web UI, Docusaurus, and generated documents; the letterhead K is the logo and favicon. kazma_guard.py --reload is the deploy path — it skips the crash backoff ladder so a fifth reload in one day is not a 300s wait (restart KazmaAgent once if an old guard is still climbing). Do not hand-kill uvicorn. Filesystem tools no longer stall /health/ready. Plan hops stay in the Plan widget. Windows Postgres checkpoints need the Selector event loop (kazma serve / the guard, not python -m uvicorn).

New in 2026-08-27: Task Ledger (durable intent resolution + git-write blast radius), transcript recall fallback (Memory & RAG), the X audit log (X publisher), and the SearXNG truth-notes fix (Web research).

Smoke checklist (when you test later): Smoke matrix.
Architecture context: Web research · Knowledge Library · Document Intelligence · Document phases · Memory best path.

---

What’s new — reliability & ecosystem (August–September 2026)

Area	What you get	Where
Providers control plane (2026-09-13)	/settings providers page is master-detail: state pills, declared capability badges, wire facts, and two real checks (GET /models + a real completion — reachable ≠ chat_ok, latency + served model). One /api/providers/* surface; the duplicate /api/settings/providers/* is deleted. Key resolution goes through ModelRegistry.resolve_provider_credentials — the same path a real message uses.	Web UI; API & extension points
Prompt fence, measured (2026-09-11/13)	AgentDojo, 996 runs × 4: fence 10.4% attack success vs 18.1% undefended (p < 0.001); ties the 4-character spotlighting delimiter (p = 0.39). "Error:" prefix bypass removed — trust comes from the caller’s is_error flag, never the content; fencing failure returns a placeholder, fail-closed. MCP tool names still classify, and a safe-looking name does not skip approval — allowlist the tool or approve the call. KAZMA_PRODUCTION=1 does not allow the name.	Security §10; INJECTION.md; THREAT_MODEL.md
Self-proving backups (2026-09-13/14)	Restore drill daily (24 h from last completed run) + weekly deep tier: Postgres data streamed through pg_restore --file=-, restic check --read-data-subset=5%, offsite HEAD read-back (a short upload and a complete one look identical from the sender). Manifest marks proven_in_production=False until a live drill result exists. The busiest database no longer misses the backup; a backup that cannot write offsite refuses to report success.	Disaster recovery; KNOWN_GAPS.md
kazma mcp (2026-09-11)	Whole tool registry as an MCP server for any client; tools/call rides LocalToolRegistry.execute() — commitment auth, PreToolUse hooks, HITL bus. No approval path → danger tools withheld (fail-closed). KAZMA_MCP_TOOLS narrows; tiers map to destructiveHint/readOnlyHint.	CLI §2.8; docs/MCP_SERVER.md
Verifiable releases (2026-09-11)	Wheel + sdist with Sigstore keyless signatures, SLSA build provenance (gh attestation verify), CycloneDX 1.6 SBOM from --frozen uv.lock, signed SHA256SUMS.	SUPPLY_CHAIN.md
OTel GenAI spans (2026-09-12)	Standard GenAI semantic-convention spans for every LLM call and tool execution — appears in an existing OTel collector with no Kazma exporter. Opt-in via OTEL_EXPORTER_OTLP_ENDPOINT; installs its exporter; never replaces a provider you configured.	OpenTelemetry (ops)
Config integrity fixes (2026-09-14)	An empty write can no longer erase a stored secret (the guard’s veto is honoured at every layer, the sqlite branch rolls back first). atomic_update holds an RLock — a swarm approval can no longer deadlock the whole process. Config writes with a dead SQLite settings DB warn about the live Postgres shadow. Vault detects divergent duplicate keys.	CHANGELOG 2026-09-12/14
Hands 0.11 (2026-09-10)	Chat is home. file_apply_patch_set (one HITL card, -/+ hunks, optional pytest verify). Supervisor re-hops on TESTS FAILED (cap 3). ACP diffs + session/cancel. Docker jail force.	IDE; CLI; examples/hands-demo/
Web /ide CodeMirror (2026-09-10)	File pane is CodeMirror 5 fromTextArea (nord, --bg-deep). Textarea-first so a blocked CDN still shows the file. Read-only git is a subprocess, not HITL.	/ide; IDE
Turn Delivery + HITL registry (2026-09)	Chat journal is SoT; close_turn is the only closer; client projects. One HITL row in hitl_gates.db (pending = live buttons; second claim 409). FanOut is tri-state.	Security; Diagnosis map; AGENTS.md §30–§31
SSRF pin-IP (Wave 8)	Direct scrape connects to the validated public IP; private peer abort. Skip pin when a proxy is set.	Security; acquisition defaults to local-only; remote mode and provider allowlist required
Context integrity (2026-08-30)	Scratchpad merges; drafts live in agent_artifacts.db; trim always summarizes dropped turns; only an explicit topic pivot disarms recall.	Memory; AGENTS.md §29
X Studio (2026-08-31)	First-class /x composer + X-only planner. Post now / Schedule / reschedule / thread hops / delete. Saved drafts stamp proposal_id (stored text wins). Chat x_post stays always-HITL. All clocks → /scheduled (mixed cron + X).	X publisher; sidebar → X Studio
Brand type + mark (2026-08-31)	IBM Plex Sans / IBM Plex Sans Arabic across UI, docs, and generated documents (Amiri naskh fallback). Letterhead K is logo, favicon, avatar. PPTX/XLSX no longer hard-code Calibri.	Arabic & cultural; Document rendering
Guard reload + stall fixes (2026-08-31)	--reload stops the recorded child and port holder, skips crash backoff, waits the boot budget, kicks KazmaAgent only if the watcher is dead. file_search off the event loop. File-tool results capped at 32k. Plan-only hops stay out of the chat transcript.	python scripts/service/kazma_guard.py --reload; Deployment
Chat as the product (2026-08-26)	/ is immersive (no Chat/Home/Chat header). Composer: attach + input + send; Long/YOLO/cost live under ⋯. Sidebar stays the grouped Work / Activity / Settings list (More was reverted — it never collapsed).	GOAL; restart after pull
Sampling HITL + native CUA + CI smoke (2026-08-25)	MCP sampling is a real Once card (KAZMA_MCP_SAMPLING=1). computer_use calls Anthropic CUA / Gemini function when that model is active. CI Playwright job: /health/live + #chat-input.	MCP; KAZMA_CUA_PLANNER=0
Post-industry leftovers (2026-08-25)	MCP resources/prompts (fenced; sampling HITL). CUA planner adapters on computer_use. LiveKit TTS published into the room. 429 backoff on Anthropic. Router word-boundaries + models.defaults. Eval tool-trace. Bright Data/Oxylabs stubs.	GOAL; MCP
LiveKit duplex (web) (2026-08-25)	Live button: WebRTC AEC + barge-in (interrupt TTS). Brain is still LangGraph. Needs LIVEKIT_URL + API key/secret. Telegram stays voice notes.	Voice; KAZMA_VOICE_DUPLEX=0
LiteLLM optional gateway (2026-08-25)	KAZMA_LITELLM_URL (or llm.gateway.url) sends OpenAI-compatible calls through a LiteLLM proxy. Native Anthropic/Azure/Bedrock/Gemini stay direct. Locals stay direct. Not exclusive.	FAQ; KAZMA_LITELLM=0
Computer use + leftover polish (2026-08-25)	computer_use screenshot→action loop (HITL). Langfuse auto-on with keys. Hosted embed fleet (KAZMA_EMBED_FLEET=1). Docling/LlamaParse salvage for hard PDFs (remote LlamaParse/Reducto upload only with documents.security.remote_parse, off by default). Voice is turn-based, not Realtime.	Tools; KAZMA_COMPUTER_USE=0; Voice
kazma ask + ACP (2026-08-25)	kazma ask "…" runs the graph without the web server. Tokens stream; TTY HITL (y/N). kazma acp is ACP stdio with session/update + session/request_permission. --yolo is headless.	Quickstart; kazma ask --help
IDE LSP backend (2026-08-25)	/api/ide/lsp hover/complete/definition/diagnostics still exist. The Web editor is CodeMirror syntax-only; the industrial editor loop is kazma acp.	/ide; KAZMA_IDE_LSP=0; IDE
Plan mode (2026-08-25)	/plan on inspects (write/exec blocked). /plan go or Proceed executes. HITL still on.	Plan pill; Slash commands; KAZMA_PLAN_MODE=0
Pre/Post tool hooks (2026-08-25)	Claude Code–style PreToolUse / PostToolUse (deny, rewrite, observe). Not a permission system — HITL still gates danger tools.	agent.hooks.*; KAZMA_TOOL_HOOKS=0; Architecture · Security
Strict tool schemas (2026-08-25)	Tool JSON Schema is closed (additionalProperties: false). OpenAI strict tools + response_format are opt-in.	KAZMA_STRICT_TOOLS=1; Tools catalog; Architecture
Codebase index (2026-08-25)	codebase_search finds functions/classes (tree-sitter or regex) plus live ripgrep. Index refreshes on write/patch.	extra kazma[index]; KAZMA_CODE_INDEX=0; IDE
E2B + Temporal (2026-08-25)	Opt-in Firecracker python_exec (E2B_API_KEY) and Temporal-wrapped swarm dispatch (KAZMA_TEMPORAL_HOST). Defaults unchanged.	extras kazma[sandbox] / kazma[durable]; env vars
apply-patch (2026-08-25; Hands 0.11 set)	Agent edits use file_apply_patch / file_apply_patch_set (HITL) instead of rewriting whole files. Web /ide is CodeMirror 5, not Monaco.	/ide; IDE; Tools catalog
pgvector memory search (2026-08-25)	When Postgres is on and the server has the pgvector extension, dense recall uses it (auto); otherwise sqlite-vec, exact. Postgres-primary recall ranks like local recall (by evidence, since 2026-09-26). KAZMA_PGVECTOR=0 keeps sqlite-vec.	Settings → Memory; Memory & RAG; Postgres & SaaS
Memory system audit (2026-08-24)	Ego-graph hub anchors (no more floating concept nodes), PG-mirror tombstones + scripts/reconcile_memory_mirror.py, tenant-scoped graph-clear (no all-tenants wipe), FTS drift rebuild on the 6h sweep, merge-ledger archive, Ungroup, honest truncation banner	/memory; restart after git pull
Universal backup	One unified backup of ALL data: every SQLite DB (WAL-safe), all assets (document-store, workspace, attachments, vectors). Auto 6h + manual; checks PG dump freshness (does not dump twice). Progress bar, delete/archive/download	Settings → Backup tab; POST /api/backup/now
Postgres backup	Automatic pg_dump of KAZMA_PG_TABLES (atomic, validated; local staging retention 3, restic keeps history) + boot-time schema verification + one-command restore	kazma-data/backups/pg/; python scripts/pg_backup.py backup|restore --latest|list
Agent Skills marketplace	Browse/install the open agentskills.io ecosystem (GitHub topic:agent-skills) from the UI; the agent can search_agent_skills + install_agent_skill	/skills → Marketplace tab
Bundled starter skills	3 Kazma-native skills ship in-tree (release-notes, conventional-commits, ui-conventions), checksum-verified	/skills (scope: bundled)
Windows tool fixes	Browser tools, shell_exec, telemetry, Ollama pull, runtime installs now actually work on Windows (selector-loop subprocess trap fixed)	transparent — tools that silently did nothing now run
Memory recall fix	Search no longer returns [] for queries containing punctuation (==, //)	transparent
429 failover restored	A rate-limited primary model now fails over to the backup chain instead of hard-failing the turn	transparent
Boot fixes	Paused-task restore no longer hangs (deferred timeout arming); PG schema guard actually verifies; research export fixed	transparent

Operator actions after git pull: kazma ask / kazma acp work without a restart (CLI; tokens stream, TTY HITL). Restart the server to activate X Studio (/x), the IBM Plex / letterhead-K brand, the --reload backoff skip (restart KazmaAgent once so the guard is on the new code), computer_use, Langfuse auto-on, Docling/LlamaParse salvage, Hands 0.11 (file_apply_patch_set, CodeMirror /ide, ACP diffs), plan mode (/plan on · Plan pill), tool hooks (agent.hooks.*; KAZMA_TOOL_HOOKS=0 disables), the closed tool schemas (additionalProperties: false; optional KAZMA_STRICT_TOOLS=1 for OpenAI strict), the memory audit fixes (hub anchors, FTS sweep, graph-clear bind), pgvector dense recall (when KAZMA_DATABASE_URL is set — run CREATE EXTENSION vector first), the template fixes (MCP/skills buttons), the backup system (first universal backup ~2 min after boot), and the Windows tool fixes. Full engineering detail in CHANGELOG.md and AGENTS.md §21–§23. Audit: AUDIT_MEMORY_SYSTEM_2026-08-24.md.

---

At a glance

Area	What you get	Where
X Studio	Compose / schedule / reschedule / thread / delete on X (official API)	/x; X publisher
Deep research	Multi-source pipeline, live sessions, routing, rubric	/research, chat, /research deep
Document Intelligence	Quarantined ingest with a PDF active-content policy, OCR, index, generate/convert/redact, ops (capacity/GC/audit), cert; optional per-job container worker	/documents, /api/documents/*, /documents slash, document_* tools, TUI Documents
Proxy Provider	Residential proxy for scrape/crawl/Playwright/SERP	Settings → System
Knowledge Library	Smart re-index, gone-URL prune, hybrid inject; document_index bridge	/knowledge, Settings → Memory, Documents → Index
Memory admin	Graph dedupe, rename, list↔graph, belief edit, hub brand, Group/Ungroup, truncation honesty	/memory
Memory explain	The turn’s Memory used row (with channels) + Dashboard probe	Settings → Memory → Explain recall
Golden eval	Offline recall regression	Dashboard → Run golden eval
Topic-shift focus	Explicit pivot disarms recall; inferred drift only re-ranks. Interrogative check-ins never count as drift	Settings → agent.topic_drift.*; AGENTS.md §29
Non-Stop & Self-Healing	Supervisor watchdog, model failover chain, call ledger, orphan recovery, HITL timeout	Settings → Agent → Non-Stop Execution
Scraper Hardening	Size caps (5MB default), 5xx retry backoff, robots.txt compliance	read_url, crawl_site, KAZMA_FETCH_MAX_BYTES
Truncation Auto-Retry	Double max_tokens on length truncation + file_append chunk tool	llm_provider, LocalToolRegistry
Turn Delivery / HITL registry	Journal + close_turn; one gate row; 409 on second Approve	Security · Diagnosis
Commitment Layer	Resolve-before-act gate; semantic clarify/confirm cards; exec denylist; modes + kill-switches	Commitment Layer
Steer / Abort	/steer (soft), /steer! (pause+inject), /abort for a running task	Slash commands
Path grants	Outside-workspace access by permission (session grant or durable extra_roots)	IDE → Path grants
UI theme overhaul	ABYSS design tokens, server-authoritative theme, dark-slate palette, mobile chrome	Web UI → Theming

---

0. Latest shipping (2026-09, then 2026-08)

September 2026 (industrial audit waves 0–8):

• Turn Delivery V2 — event-sourced chat: journal + close_turn. Token deltas append; only turn_complete replaces. Do not restore a second painter in chat.js.
• HITL Gate Registry — hitl_gates.db CAS (pending → claimed → settled). Ghost / pre-stamped Approved cards were the incident class. Kill-switch KAZMA_GATE_REGISTRY=0 is a thin execution fallback, not a second author. Security · AGENTS.md §30.
• FanOut tri-state — with 2+ chat platforms, a Deny is a vote until expected_voters or the deadline. Web first-claim stays 200/409.
• SSRF pin-IP — direct scraping pins the public IP; abort if the peer is private. Do not pin through proxy=.
• Context integrity — scratchpad merge reducer; durable proposals; summary net on every trim; shift_explicit vs shift_inferred.
• Docs pass — AGENTS.md, architecture, swarm, system map, security, memory, commitment, diagnosis, this page, production checklist.
• Guard 503 still says unreachable: Service Unavailable (Docker/Postgres). Deferred: Guard/ops alerting.

The August tranche — beyond the items in the table above:

• X Studio (/x) — compose, schedule, reschedule, thread hops, delete a live tweet, load a saved draft. Chat x_post stays always-HITL; the Web click is the approval. All clocks opens /scheduled (cron + X). X publisher.
• Brand type + mark — IBM Plex Sans / IBM Plex Sans Arabic (UI, docs, generated documents). Letterhead K is logo/favicon. Amiri remains the naskh fallback. Arabic & cultural.
• Guard --reload skips crash backoff — a deploy kill is not a crash, so the fifth reload of the day is not a 300s wait. Restart KazmaAgent once if an old guard is still climbing. Deployment.
• Steer & Abort (/steer, /steer!, /abort, /stop) — out-of-band signals to a running task. Soft steer folds text into the next step; hard steer pauses, injects a requirement, and resumes (demoting to soft if the task is finalizing); /abort cancels and abandons. See Running task commands.
• Commitment Layer (resolve-before-act) — a policy gate between the LLM and durable mutations. All phases (0–8) shipped: authorize_effect, semantic clarify/confirm interrupt cards with per-option buttons, exec denylist, config protected keys, outbound allowlist, swarm scope tokens, soul-confirm gate, modes (strict/balanced/autonomous/yolo), and kill-switches. Dedicated guide: Commitment Layer.
• HITL semantic-resume parity (PR1–PR5) — a single enforced resume chokepoint, WS/SSE parity for semantic interrupts, the terminal-clarify invariant that kills the clarify loop, args-first memory-checked remind gate, and the semantic clarify option-cards + /metrics loop counter.
• Path grants — outside-workspace access is deny-by-default but openable by permission: smooth session grants via the request_path_access HITL card, or durable workspace.extra_roots. IDE → Path grants.
• Primary kazma update — a fail-closed operator upgrade path: preflight, named stash, hard reset to origin/main, extras-preserving reinstall, and postflight verification. Kazma Update.
• Arabic PDF ingestion — PyMuPDF primary with a multi-engine bake-off (pypdfium2/pdfplumber/pypdf), layout-aware reading order, a fuzzy RTL round-trip check of generated PDFs, and ara+eng OCR routing. Numbers, ligatures and table cells in Arabic PDF lines were still reversed until the 2026-10-13 glyph re-read, and Arabic OCR remains unqualified. Document Intelligence.
• Document rendering richness — code/syntax highlighting (PDF+HTML), XLSX charts (multi-series + axis titles), approved-asset image embedding, and a real PDF TOC + clickable HTML TOC. Document rendering.
• UI theme overhaul (P1–P4) — unified ABYSS design tokens, server- authoritative theme (persists across devices), dark-slate palette with real layer depth, mobile chrome fixes, and forced HTML revalidation. Web UI → Theming.

---

1a. Document Intelligence (product path)

What it does

End-to-end durable document platform: streamed intake → quarantine CAS → policy sniff → isolated parse/OCR → durable jobs → optional Knowledge index, generate/convert/redact, ops (capacity, GC, audit, cert). The default process worker bounds memory and CPU but does not confine files or network; the container profile (KAZMA_DOCUMENT_WORKER_ISOLATION=container) runs each job in its own container with no network and no mounts, through a document broker.

How to run

Entry	How
Web	/documents upload + ops; Settings → Documents
Agent	document_import / document_read / document_index / …
Gateway	/documents list|read|status|… (alias /docs)
TUI	Documents tab
Cert	python scripts/certify_documents.py

Install engines (optional)

pip install -e ".[document-platform]"
# System: Tesseract (OCR), ClamAV (malware), LibreOffice (some convert)

Deep docs: Document Intelligence · Ops · Security · Phases 0–10.

---

1. Deep research (product path)

What it does

run_research_pipeline plans queries, searches via web_acquire, ranks URLs, acquires full pages, digests, synthesizes, optional gap-fill, and writes a report under research/reports/ with a structural rubric.

Industry stages (R0–R4): ranking + claims + fail-closed deep → adaptive plan + gap loop → durable sessions + SSE → soft route to pipeline + eval API.

How to run

Entry	How
Web	Open /research → topic, depth (Deep/Brief), max sources → Start
Chat	“Deep research on …” (supervisor prefers run_research_pipeline)
Gateway	/research deep <topic> (progress pings while running)

Live sessions (R3)

• Session rows: kazma-data/research_sessions.db
• Start: POST /api/research/sessions
• Live progress: GET /api/research/sessions/{id}/stream (SSE)
• Cancel button while status is running / pending
• List shows [Deep] sessions + pipeline papers + swarm research tasks

Routing (R4)

Deep-worded asks get a system hint to call the pipeline once instead of a long manual web_search chain. Manual multi-hop on a deep request gets one “prefer pipeline” nudge. Disable: KAZMA_RESEARCH_ROUTE=0.

Quality

• Report + rubric.json under the paper folder
• Session stores rubric_score / rubric_ok
• Score a report: GET /api/research/eval?path=research/reports/… or ?session_id=… (only a file inside a research/reports folder is read)

Deep dive: Web research.

---

2. Shared web acquisition + Proxy Provider

Shared stack

Search / scrape / crawl I/O live in kazma_core.web_acquire (used by research and KB page extract). LLM APIs never go through this stack.

Proxy coverage (Settings → System → Proxy Provider)

When a provider is configured (e.g. anyIP), it applies to:

Path	Proxied?
Page fetch (httpx ladder) research + KB	Yes
Playwright recovery (read_url + KB)	Yes
crawl_site link spider	Yes
KB sitemap / robots discover	Yes
Bing / Wikipedia SERP	Yes
DuckDuckGo (ddgs)	Yes (when Proxy Provider configured)
Remote SearXNG	Yes
Local / Docker SearXNG	No (loopback — no hairpin)
Jina / Firecrawl API calls	No (they fetch the target server-side)
LLM provider APIs	Never

Config is live (no restart). Password vault-encrypts.

Deep dive: Web research → Bulletproof scraping.

---

3. Knowledge Library hardening

Re-index

Situation	Behavior
Page unchanged (same chunks in the same order)	Skipped: nothing written
Page changed or shrank	Replaced in one transaction: a search sees the old page or the new one, never a mix
URL missing from a later discovery	Kept: a sparse or failed discovery never deletes indexed pages
Page fails	Counted as failed; the crawl ends Finished with failures, never “Done”

Job UI / toasts show skipped · unchanged · failed. Since 2026-10-14 (Knowledge audit) the library has its own knowledge.db, one injection per turn, a crawl journal with one row per job, admin-only /kb changes and a meaning-search outage that warns and alerts: see Knowledge Library.

Recall + inject (one hybrid stack)

Stores stay separate — no one-table merge with V2 beliefs.

Chat turn
  ├─ V2 memory recall (beliefs + episodes)
  ├─ KB inject (RRF: Chroma semantic + FTS5)  ← same as knowledge_search
  └─ All inject fenced as untrusted docs

Mode	Libraries
Inject	auto_inject=1 libs (+ smart search expansion)
Federated / tools	All active non-archived libs with chunks

Settings → Memory

Toggle	Key	Effect
Inject Knowledge into chat	memory.v2.merge_knowledge_into_chat	Per-turn KB inject
Smart Knowledge search	knowledge.smart_search	On technical Qs, inject from all active libs with chunks
Explain recall	memory.v2.explain_recall	Channels on the turn’s Memory used row (below)

Kill switch: KAZMA_KB_AUTO_INJECT=0.

Library text reaches the turn only: it is never saved into the chat’s memory. Until 2026-10-13 a “Promote top KB hits” switch (on by default) copied it in as if you had said it; the switch is gone, and Kazma clears the copies it made.

Deep dive: Knowledge Library · Memory best path.

---

4. Memory explain + golden eval

The turn’s “Memory used” row

A turn that used memory shows a Memory used row in its activity (inside the turn’s thoughts fold): how many facts, memories of past conversations, weekly summaries and library passages the model was shown, and one line per item, in the order the model saw them. The row is stored with the turn, so a reload shows it again, and every tab watching the chat shows it (SSE and WebSocket). With Explain recall on (Settings → Memory, the default), each line also names the retrieval channels that found it:

Channel	Meaning
fts5 / belief_fts	Lexical
dense	Embedding similarity
belief_ppr	Multi-hop on the belief graph (default 3-hop)
session_boost	Same-thread episodes
kb_rrf	Knowledge hybrid RRF

With Explain recall on and nothing found, the row says “nothing matched”. (The earlier “Memory context” panel stopped drawing on 2026-09-20; this row replaced it on 2026-09-27.)

Dashboard

• Memory probe — same channel chips
• Federated — memory + KB labeled
• Run golden eval — offline fixture pass rate (POST /api/memory/v2/eval/golden)

Belief multi-hop (PPR)

Belief-graph Personalized PageRank uses confidence-weighted edges and a configurable hop radius (memory.v2.ppr_hop_radius, default 3) so chains like user → works_at → Acme → located_in → Paris can surface.

Deep dive: Memory & RAG.

---

4b. Memory admin UI — graph, rename, list bridge (2026-08)

Operator page /memory: graph on top, entities/beliefs/merge/hygiene below.

Capability	Behavior
No duplicate graph ids	Object text that equals an entity id (e.g. shipx) is one real node, not entity + virtual fact
Display rename	Id stable; name + aliases change (ShipX, hub Mubder)
List ↔ graph	Click row ⇄ click node; rename/merge/invalidate refresh canvas
Edit belief	Beliefs → Edit → PATCH triple (object/predicate/subject)
Hub identity	ent_* person User shells map to hub user; rename syncs hub label
Ego-graph anchors	Payload-object facts (subject → pred → literal) also get user → related_to → subject so they are not a disconnected component
Group / Ungroup	View-only clustering from inspect; 30s poll uses groups on the graph payload (no extra GET)
Truncation honesty	Banner reports nodes and connections hidden by the top-N cap

Deep dive: Memory & RAG — admin UI · Memory best path.

---

4c. Memory page overhaul + cron reminders (2026-08)

A full pass on the operator /memory page — usefulness, performance, correctness, accessibility — plus a long-standing cron-reminder crash fix.

Memory page (/memory):

Capability	What changed
Pagination + counts	Lists show “Showing 1–150 of 3,412” + Load more; graph shows a truncation banner. No more silent cap masquerading as empty.
Real search	Diacritic-insensitive (francais→Français), alias-aware FTS5 search across beliefs + entities.
“Why recalled”	Click a belief → recall history (count, last time, origin episode) + Probe from this belief.
Undo	Invalidate / link / edit / delete show an [Undo] toast for 60s. Merge shows a rewired-count receipt.
~10× faster page-open	Materialized entity counts replace per-row correlated subqueries; self-heals if a write site is missed.
Multi-tenant	KAZMA_MEMORY_ENFORCE_TENANT=1 isolates memory per tenant (off by default).
Accessibility	aria-live status, table captions, dynamic canvas descriptions, single belief-edit modal.

Cron reminders now actually deliver:

• The scheduler was constructed without a graph_builder=, so every scheduled reminder crashed with “No graph builder configured” on fire. Now wired.
• Reminders also never reached Telegram because the delivery target wasn’t captured; now delivery_target (the originating telegram:<chat_id>) is captured at schedule time and used at fire time. (SessionStore lookup is not a viable fallback — sessions TTL-evict after 5 min.)

Deep dive: Memory & RAG — operator capabilities · env vars.

---

4d. Non-Stop Execution & Self-Healing Engine (2026-08)

An enterprise-grade self-healing execution layer designed for long-horizon autonomous tasks.

Capability	Behavior
Supervisor Watchdog	supervised_invoke() wraps graph execution with node heartbeats, stall detection (default 60s), and incident classification (stalled, transient_llm, context_overflow, panic). Auto-rolls back to last durable checkpoint, injects reflection, and resumes up to N attempts.
Model Failover Chain	Exhausted primary models fail over down agent.nonstop.failover.chain with per-model cooldowns (default 300s) without mutating active settings profiles.
LLM Execution Ledger	Durable SQLite WAL (kazma-data/llm_calls.db) recording thread, iteration, model, token usage, cost, latency, status, and failover origin for every LLM call.
Startup Orphan Recovery	Swarm tasks stranded in status='running' by process crashes or restarts are requeued to pending on startup up to 3 attempts.
HITL Approval Timeout	Background watchdog scans pending HITL approval interrupts every 15s and expires unanswered approvals after safety.hitl.approval_timeout_seconds (default 300 s); the model is told the card expired, not that you refused it.
Resilient Chat	Non-graph LLM calls (swarm workers, research planner, research synthesizer) use resilient_chat with transient retries, failover chain, and tool-execution timeouts (agent.tool_timeout_seconds, 120s).
Settings UI Card	Agent Settings tab includes a Non-Stop & Self-Healing section with live-re-read toggles, thresholds, and failover chain controls (EN/AR i18n supported).

---

4e. Scraper Hardening & Truncation Recovery (2026-08)

Industry-grade web scraping resilience and model output truncation recovery.

Feature	Details
Response Size Caps	read_url streams body reads and enforces KAZMA_FETCH_MAX_BYTES (default 5 MB) to prevent memory exhaustion and gzip-bomb exploits.
Content-Type Gate	Non-textual binary payloads (PDFs, images, archives) fail fast with actionable guidance instead of polluting the text extractor.
5xx Retry Loop	HTTP scraper retry loop covers 500, 502, 503, and 504 status codes with backoff and jitter (3 attempts).
robots.txt policy	Automated crawl/KB/research saves enforce bounded KazmaBot checks by default; failed checks refuse crawling and disallowed URLs report robots_disallowed. Explicit opt-out is visible. See web research for limits.
Auto-Retry Truncation	When LLM completion stops with finish_reason='length', llm_provider.chat transparently retries ONCE with doubled max_tokens (capped at 4x configured / 32k) instead of returning broken tool JSON.
file_append Tool	Built-in tool in LocalToolRegistry allowing agents to write large files in chunks (file_write to initialize, file_append for subsequent sections).

---

5. Quick operator recipes

A. First deep research paper

1. Ensure web search works (SearXNG recommended).
2. /research → topic → Start.
3. Watch stages; open MD when done; note rubric score on the card.

B. Cheap KB refresh

1. Crawl a library once.
2. Refresh immediately.
3. Confirm unchanged / skipped in the job bar (little re-embed work).

C. “Agent just knows my docs”

1. Ingest library; set auto_inject on that library.
2. Keep Inject Knowledge into chat on.
3. Optional: Smart Knowledge search for technical questions across all active libs.
4. Ask a docs question; expect citations / knowledge footer.

D. Debug what memory used

1. Explain recall on.
2. Ask a personal-fact question after “Remember …”.
3. Open the turn’s Memory used row (its channels) and/or the Dashboard probe.

E. Harden scraping

1. Settings → System → Proxy Provider → anyIP (or none).
2. Test Connection.
3. Retry a blocked read_url / KB crawl.

F. Brand the memory hub (You → Mubder)

1. Open /memory.
2. Entities tab → find person User or ent_… (or hub).
3. Rename → Mubder (or Kazma).
4. Hard-refresh; canvas center label should match. Click the row → hub zooms.

G. Fix a bad belief without SQL

1. /memory → Beliefs → search.
2. Edit → correct object (then predicate/subject if needed).
3. Graph refresh should show the new edge text.

---

6. API cheat sheet

Method	Path	Role
GET	/api/research/ready	Preflight (optional ?live=1)
POST	/api/research/sessions	Start deep research
GET	/api/research/sessions	List sessions
GET	/api/research/sessions/{id}/stream	SSE progress
POST	/api/research/sessions/{id}/cancel	Cancel run
GET	/api/research/eval	Rubric score
GET/PUT	/api/settings/memory/merge-kb	Inject, promote, smart search, explain
GET/PUT	/api/settings/proxy	Proxy provider
GET/PUT	/api/settings/agent/nonstop	Non-Stop & Self-Healing settings
POST	/api/memory/v2/probe	Recall dry-run (explain on)
POST	/api/memory/v2/federated-search	Memory + KB labeled
POST	/api/memory/v2/eval/golden	Golden recall suite
GET	/api/memory/v2/graph	Belief canvas payload (unique ids, hub label, embedded groups)
DELETE	/api/memory/v2/graph/groups/{id}	Ungroup (view-only)
POST	/api/memory/graph/clear	Tenant-scoped bi-temporal invalidate + Neo4j cleanup
GET	/api/memory/v2/entities	Entity list (is_self, graph_id)
POST	/api/memory/v2/entities/{id}/rename	Display rename (+ hub sync for self)
PATCH	/api/memory/v2/beliefs/{id}	Operator edit triple
GET	/memory	Memory admin HTML page

Full tables: API routes.

---

7. What we deliberately did not do

Idea	Status
Merge KB + beliefs into one SQLite table	Won’t fix — federated / inject only
Require Neo4j for memory	No — SQLite SoT; Neo4j dual-write optional
Proxy LLM traffic	No — scraping only
Postgres-primary / multi-region as default	Later (scale issues #76–#78)

---

8. Verify after upgrade

Operator reload (do not kill python/uvicorn):

& '.venv\Scripts\python.exe' scripts\service\kazma_guard.py --reload
& '.venv\Scripts\python.exe' scripts\service\kazma_guard.py --status

Then: GET /health/deep 200; HITL one-card Approve + 409 on the second click; Smoke matrix. Production go-live: Production checklist.

---

Related docs

Topic	Link
Full research pipeline & proxy	Web research
KB ingest & smart re-index	Knowledge Library
V2 + inject operator path	Memory best path
Web surfaces	Web UI
Multi-path debugging	Diagnosis map
Build contract	AGENTS.md (§30–§33)
Industrial audit	AUDIT_DEEP_2026-09-01_EXEC.md