Build Log: Prompt-Injection Hardening for Household AI Agents

The two butler agents on my household platform are interactive LLM agents with real household reach: they read and write calendars, tasks, and spreadsheets, they manage files, and they do web research on request. That capability list doubles as the attack surface. A malicious web page doesn't need to exploit anything — it needs one sentence that makes the model call a tool on its behalf.

Prompt injection isn't hypothetical for an agent that browses; it's the default outcome of letting untrusted text share a context window with tool access. So the agent stack got a dedicated hardening pass: four controls, each closing a specific path from influenced model to changed household state. The platform around them is covered in the FR4M3W0RK architecture overview.

The threat model

Enumerate the paths before designing the controls:

1. Web content → butler context → tools. The butlers browse; a hostile page can plant instructions aimed at the tool layer.
2. Agent → automation. The butlers call n8n workflows over webhooks; a webhook that can be spoofed can be fired by anyone who can reach it.
3. A compromised agent container. Whatever an attacker gains inside the agent's runtime — memory files, session state, network position — should be worth as little as possible.
4. Anything that turns model output into execution. An agent that can run code is an agent whose prompt is a remote code interface.

Each control below maps to one of those paths.

Control 1: Research is delegated, not direct

The butlers lost their direct web tools entirely. All web research is delegated to a separate, sandboxed, read-only research agent that has access to no household tools whatsoever.

This closes path 1 by subtraction: a malicious page can still influence what the research agent reports, but the report is the ceiling. Research can shape what the butler knows; it cannot reach the calendar, the tasks, the spreadsheets, or the filesystem. The untrusted input never shares a tool surface with the trusted capabilities.

Control 2: Approval gates at the gateway

A policy plugin intercepts sensitive tool calls — calendar, task, and spreadsheet writes, and any file write outside memory — and requires explicit human approval in the chat before the call executes.

This closes the gap that remains after control 1: an injected instruction that does reach the tool layer still stops at a human. The design intent isn't to catch every malicious prompt; it's to make every state-mutating action pass an explicit, auditable gate. The agent proposes; a person approves; the decision is visible in the conversation.

Control 3: Signed internal webhooks

Agent-to-automation requests are HMAC-SHA256-signed with a shared secret and verified at the top of every workflow, replacing a static header. And the webhooks are never exposed publicly at all — they're only reachable on the internal Docker network.

This closes path 2 twice over, which is the point: network position and a cryptographic check are independent properties. Either one can fail without opening the other. A spoofed webhook now needs both a route into the internal network and the signing key.

Control 4: No execution, no inbound

The agents have no exec capability and no published ports. At the container layer they run with no-new-privileges, with NET_RAW and NET_ADMIN dropped, reaching their bot identities, their LLM provider, and the automation webhook via outbound connections only.

This closes paths 3 and 4. A compromised agent container has no shell to reach, nothing listening to attack, and no route back in. There's an earlier post in the Agent Notes series about the opposite failure — a sandbox that existed only in the documentation, where the runtime shipped the sandbox off with the Docker socket mounted. The lesson there was that a behavioral boundary isn't an isolation boundary; this pass is the standing answer: isolation enforced in configuration, not asserted in docs.

What the controls buy — and what they don't

The honest summary: injection can still influence what the model says. The distance between "influenced" and "household state changed" now crosses a delegated, toolless research boundary, a human approval gate, and a signature check.

Two limits are worth stating plainly. First, the approval gates are only as good as their taxonomy — the policy plugin's list of sensitive actions is the real security boundary, and it's maintained by hand. A tool category missing from the list inherits no gate. Second, delegated research removes reach, not presence: the research agent's report still enters the butler's context, so injection pressure on conversational output remains. What's gone is the direct line from hostile text to household state.

The stance

Treat the LLM as an untrusted planner, and make every state-mutating action pass an explicit, auditable gate. That's the whole design stance behind all four controls — and it scales past household agents. The same principle, applied to unattended agents where no human is in the chat to approve, shows up in the STR4NG3L00P runner architecture: if no one can click approve, the gates have to be deterministic instead.