Security coverage
What AgentWatch computes, what it cannot see, and what its brakes actually do. The checks follow section 5 of the AI Agent Security Field Guide.
What the kill switch actually does
A brake has to live outside the agent. A stop that depends on the agent's cooperation is not a brake, so AgentWatch says plainly which kind each agent has:
- Enforceable — the agent's model and MCP calls go through Gate. After a kill or a revoked key, Gate refuses the next call. The agent cannot get past it on that path.
- Cooperative — the agent reports through the SDK. After a kill, every further step gets HTTP
409and the SDK raisesAgentWatchKilled. An agent that ignores that, or never asks, keeps running. AgentWatch cannot stop it.
Isolate stops an agent, not just a run. It revokes every gateway key on the agent's passport, suspends the passport so a key minted later is refused too, sets an overlay that denies every tool, kills the agent's open runs and refuses new ones. On the Gate path the next call fails. On the SDK path the next call raises; an agent that ignores that keeps running. Under a dry-run passport, isolate only records what it would have done. Restoring the passport lifts the suspension and the overlay. Revoked keys stay revoked, so mint new ones.
What AgentWatch can do for every agent is tell you definitively whether it ignored a stop. Every step, preflight or Gate call that arrives after a kill is recorded with how many seconds after the stop it came. Each agent page shows its stop compliance. A cooperative-only agent fails posture check AW-P12, and is not counted as having a kill switch.
The model is not a security boundary
AgentWatch does not stop a model from being fooled. Prompt-injection detectors flag suspicious content and exploitation patterns; they reduce risk, they do not guarantee anything. What AgentWatch does is observe and constrain around the model. It limits what the agent may call (Gate), requires a human before consequential actions (approvals), and flags structural risk before an attack (posture: the lethal trifecta).
What AgentWatch can see
AgentWatch sees what the SDK, Gate, Cursor hooks, platform connectors and MCP registry report, plus what you declare on an agent's passport. It has no endpoint collector, no provider admin APIs and no package-registry feed. Any check that would need one is reported as not_collected, never as passing.
Posture checks
| Check | Name | Class |
|---|---|---|
| AW-P01 | Lethal trifecta | A computed from existing data |
| AW-P02 | Unused privilege | A computed from existing data |
| AW-P03 | Borrowed identity | B computed from a field added for it |
| AW-P04 | Long-lived keys | A computed from existing data |
| AW-P05 | Approvals off | A computed from existing data |
| AW-P06 | Unpinned supply chain | B computed from a field added for it |
| AW-P07 | Unapproved component | B computed from a field added for it |
| AW-P08 | Unsafe model format | C not collected — needs a data source we do not have |
| AW-P09 | No sandbox | B computed from a field added for it |
| AW-P10 | Unscreened model traffic | A computed from existing data |
| AW-P11 | Content logging on | B computed from a field added for it |
| AW-P12 | No kill switch | A computed from existing data |
Runtime detections
| Check | Name | Class |
|---|---|---|
| AW-D01 | Hijack after untrusted read | A computed from existing data |
| AW-D02 | Tool outside allowlist | A computed from existing data |
| AW-D03 | New destination | A computed from existing data |
| AW-D04 | Tool description or version changed | A computed from existing data |
| AW-D05 | Approval-skipping launch | A computed from existing data |
| AW-D06 | Secret hunting | A computed from existing data |
| AW-D07 | Key misuse | B computed from a field added for it |
| AW-D08 | Cost or volume spike | A computed from existing data |
| AW-D09 | Memory write from untrusted source | A computed from existing data |
| AW-D10 | Rubber-stamp approvals | B computed from a field added for it |
| AW-D11 | Ignored stop | A computed from existing data |
| AW-D12 | Package install of unknown name | C not collected — needs a data source we do not have |
| AW-D13 | Autonomy rising | A computed from existing data |
What a typical install does not collect
- Never collected: AW-P08 unsafe model format, AW-D12 unknown-package install.
- Not collected until the passport is filled in: AW-P02, AW-P03, AW-P04, AW-P09.
- Fail by construction: AW-P10 (0% of model calls screened) and AW-P12 (cooperative stop only). That is the honest result for an SDK-only agent.
- AW-D04 tool-change detection works only for MCP servers registered with their tool list or proxied through Gate.
With Gate, AW-P10 and AW-P12 can pass, key-misuse detection gets caller IPs, and stop violations are blocked rather than only recorded.
Thirteen real incidents
We replay each incident from the field guide as telemetry in an automated test suite. The result: 5 caught, 5 partially caught, 2 missed, 1 out of scope. The known blind spots:
- Data leaked through a rendered image or link in the model's own reply, with no tool call (EchoLeak).
- What a tool's own code does with the data, e.g. postmark-mcp's hidden BCC, and compromised Python/npm packages.
- Processes started outside a monitored agent, such as an install script launching an AI CLI.
- Package reputation (slopsquatting).
Full tables: docs/DATA_SOURCES.md and docs/INCIDENT_REPLAY.md in the repository; live per-agent results on the Posture page.