Failure modes

What AgentWatch does when one of its own dependencies fails, at every decision point. Each "Tested" row names the fault-injection test in backend/tests/test_failure_modes.py that proves it; each test is mutation- checked by scripts/guard_sweep.py (break the behaviour, the test goes red). Rows marked Gap are behaviour we have not built or not proved; they are listed so nobody has to discover them.

Three rules

1. Telemetry failure never blocks the agent. A dead collector, Redis or Socket.IO does not fail POST /steps.
2. A policy check that fails follows the passport's enforce_mode. fail_closed (the default) on an enforcing passport denies. fail_open, or a passport in dry_run, allows, and the allow is recorded as a policy_check_failed_open security event (Gate also writes a gateway_verdicts row with reason failopen_internal_error). An allow caused by a failure is never silent.
3. One bad detector degrades to "no finding", never a 500.

denied_tools is evaluated before any of this and is enforced by design, in every mode.

Matrix

Decision point	Dependency failure	Behaviour	Proof
POST /steps/preflight	passport tool check raises	fail_closed+enforce: deny, reason internal_error:passport_check. fail_open: allow, reason failopen_internal_error:passport_check, security event	test_preflight_check_error_fail_closed_denies, test_preflight_check_error_fail_open_allows_and_records
POST /steps/preflight	egress check raises	same, stage egress_check	same tests, egress_check case
POST /steps/preflight	any check raises, passport in dry_run	allow + security event (dry_run never denies)	test_preflight_check_error_under_dry_run_allows_and_records
Gate (/gw/*)	egress check raises	fail_closed: JSON-RPC error egress_check_error. fail_open: call forwarded, security event (surface gate)	test_gate_egress_error_fail_closed_denies, test_gate_egress_error_fail_open_allows_and_records
Gate (/gw/*)	any preflight check raises	fail_closed: deny, internal_error. fail_open: allow, failopen_internal_error. Both persisted to gateway_verdicts	test_gate_internal_error_follows_enforce_mode_and_is_persisted
Gate (/gw/*)	upstream MCP server down / 5xx	JSON-RPC error to the agent; the API does not crash	test_gate_upstream_down_is_an_error_not_a_crash
POST /steps	a detector raises	that detector yields nothing; the others still run (an injection in the same step is still flagged)	test_ingest_survives_a_raising_detector
POST /steps	security.scan_step raises	step stored, received: true	test_ingest_survives_a_security_scan_failure
POST /steps	a guardrail rule raises	other rules still evaluated. Errored enforce rule on a fail_closed agent: run paused for a human. Otherwise the step continues and a security event (stage guardrails) is recorded. Errored dry_run rules never pause	test_guardrail_error_on_fail_closed_agent_pauses_the_run, test_guardrail_error_on_fail_open_agent_continues_and_records, test_guardrail_error_on_a_dry_run_rule_never_pauses
POST /steps	Redis down (job queue)	step stored, request succeeds; the async task for that step (semantic detectors, injection classifier, confidence score) is dropped and counted in agentwatch_task_enqueue_failures_total{task=…}	test_ingest_survives_the_job_queue_being_down
POST /steps	Socket.IO / Redis pub-sub down	live-view broadcast is skipped; ingest unaffected. At boot, an unreachable Redis falls back to the in-process manager	test_realtime_emit_failure_is_swallowed
OTel export	collector unreachable	BatchSpanProcessor (off the request path); export_run returns in under 0.5 s with the collector dead	test_otel_export_is_batched_and_does_not_block_on_a_dead_collector
Semantic detectors (worker)	embedder times out	task finishes without a finding; no crash	test_semantic_task_survives_an_embedder_timeout
SDK telemetry	AgentWatch API unreachable	agent keeps running; steps are dropped	test_sdk_telemetry_fails_open_when_the_api_is_unreachable
SDK preflight, fail_closed=True	AgentWatch API unreachable (at run start or at a check)	AgentWatchBlocked, reason preflight_unavailable. Before G3 an unreachable API at run start made the run unsampled and every later check returned allow	test_sdk_fail_closed_preflight_blocks_when_the_api_is_unreachable
SDK preflight, default	AgentWatch API unreachable	allow, reason preflight_unavailable (fail-open is the SDK default; preflight is cooperative)	SDK default, sdk/agentwatch/run.py
API worker (re)start	traffic already flowing	Boot migrations skip already-applied DDL and serialise on an advisory lock, so a restarting worker takes no table lock that live requests wait on	test_a_worker_booting_under_load_takes_no_table_lock (test_migration_cost.py)
POST /steps (many concurrent writers, one tenant)	chained audit rows	The audit-chain head is locked only at commit, so a tenant's requests no longer queue behind each other's whole transaction	test_head_lock_is_not_held_between_flush_and_commit (test_audit_chain.py)
Any endpoint	DB connection pool saturated	Requests queue for a connection (up to 30 s) instead of the worker stalling. The pool (10 + 20 per worker) outsizes the 20-thread request pool, the browse-only check no longer queries on the event loop, and Gate no longer holds a connection across its upstream call	test_pool_outsizes_the_threadpool, test_browse_only_check_does_not_block_the_event_loop, test_gate_releases_its_db_connection_during_the_upstream_call
Any endpoint	Postgres down	request fails (5xx). SDK telemetry fails open; SDK fail_closed preflight denies (rows above)	Gap: no server-side test with Postgres stopped
Any endpoint	Postgres slow	no server-side statement_timeout on the ingest or preflight path; a request waits until the SDK's 10 s client timeout, then the SDK row above applies. Only /nl/query has a 5 s timeout	Gap
Any endpoint	disk full on the DB host	Postgres PANICs on WAL fsync and restarts; writes fail until space is freed	Gap: observed in development during G2, not handled

What changed in G3

Building this matrix, and the G4 load test after it, found these places where a failure was a silent allow or an outage:

• /steps/preflight swallowed exceptions from the passport and egress checks, so the check passed silently. Now it follows enforce_mode and records it.
• Gate's fail_open egress-check path allowed without a record. It now writes a security event.
• An unreachable Redis at enqueue time turned POST /steps into a 500 after the step was committed. The SDK then retried and double-logged it, or dropped it. Now the task is dropped and counted.
• One guardrail rule raising failed the whole ingest. The SDK then treats that as "unreachable" and carries on, so a broken rule was a silent bypass. Now rules fail independently, under the agent's enforce_mode.
• The SDK's fail_closed didn't cover run start (see the SDK row).
• Above about 15 concurrent callers per worker, every request stalled for 30 s. A query on the event loop waited for a pooled connection held by request threads, which needed the loop to give it back. Gate also held a connection across each upstream LLM/MCP call. Measured: preflight at 50 concurrent callers fell to 2.9 requests/s, all timing out.
• A worker restarting under load deadlocked live ingest (500s) with its boot migrations. A tenant's concurrent ingests also serialised on the audit-chain head lock, held for the whole request. Both are fixed; see the two rows above.