AgentOps for a personal project
AgentOps was not a folder called deploy/.
I know this because I had the folder. I had the scripts. I had doctor checks, metrics, memory sanitization, soul generation, backups, and deployment automation. The folder was immaculate. The scripts ran on schedule. The output was well-formatted.
It turned out that what I actually had was a collection of promises of varying seriousness — some were instructions the model might follow, some were configuration the gateway might enforce, and some were scripts that would report on problems after they'd already happened. Discovering which was which became the work.
The control maturity ladder
instruction
↓
configuration
↓
runtime enforcement
↓
external detection
↓
operator response
↓
independent verification
A soul could instruct a butler not to execute commands.
An agent allowlist could deny exec.
A sandbox could prevent filesystem escape.
A metrics script could flag a suspicious entry.
An operator could investigate the alert.
A separate test could verify that the boundary remained true after deployment.
These were different controls with different evidence. I had not previously thought about them this way.
What the repository checked
deploy/doctor.sh checked selected deployment conditions, including configuration and plugin state. deploy/agent-metrics.sh computed post-hoc reports such as lexical repetition and conceit-leak signals. deploy/sanitize-memory.sh scrubbed structural leaks, checked pre-session loop signals, and locked prior memory files read-only. deploy/build-souls.sh composed source souls into generated butler files.
The scripts were valuable because they moved some reasoning outside the model.
They were still only as effective as their scheduling, permissions, coverage, and operator follow-through.
agent output
↓
sanitation / metrics / doctor
↓
report or quarantine
↓
operator reviews evidence
↓
fix is deployed
↓
independent check confirms the fix
The first question was not "does the script exist?"
It was "what runs it, with which permissions, against which files, and what happens when it fails?"
Detection is not prevention
The Round 1 review surfaced a useful example.
The Maker's loop defenses had been mostly instructions: compare recent entries, avoid repetition, permit quiet days, and pivot when stuck. The metrics layer could score repeated n-grams after the fact. Neither prevented semantic repetition that used different words.
Lexical similarity was not semantic similarity.
A structural scrub could remove raw tool-call token leakage without stopping the agent from writing the same narrative shape ten days in a row.
A post-hoc report could identify a dangerous action without preventing the action.
The maturity label mattered:
| Control | Actual class | What it proved |
|---|---|---|
| Soul pet peeve | Instruction | Intended voice constraint |
| Tool deny list | Configuration/runtime candidate | Depends on gateway enforcement and test |
| Memory read-only lock | Structural enforcement | Files locked after the relevant step, if the hook ran |
| N-gram metric | External detection | Repeated surface patterns |
| Embedding gate | Planned external detection | Semantic similarity, if implemented and validated |
| Operator review | Response | Human decision after evidence |
The soul round trip
The butler personas originated in soul/master.soul.md and per-agent sub-souls. deploy/build-souls.sh generated the combined workspace files.
soul/master.soul.md + soul/husband.soul.md
↓
deploy/build-souls.sh
↓
workspaces/husband/SOUL.md
↓
gateway loads generated file
A round-trip check could compare generated output with the source composition. It could catch accidental drift between the files in Git.
It could not prove that the model obeyed every line. There is, it turns out, no script for that.
That required scenario tests: ask for a prohibited execution, a work-domain wife event, or a memory claim and observe the actual response and tool behavior.
Failure-to-control matrix
| Failure mode | Candidate control | Evidence required | Status |
|---|---|---|---|
Butler receives exec | Tool deny policy | Runtime attempt and logs | Partially verified |
| Tool-call syntax leaks into memory | Structural scrub | File inspection after cron | Implemented in repo; deployment evidence required |
| Repeated narrative | N-gram metric | Longitudinal reports | Limited |
| Semantic repetition | External similarity | Scores, thresholds, stop behavior | Planned |
| Soul source drifts from generated file | Round-trip build | Diff-free rebuild | Scripted; test on deployment |
| Backup exists but cannot restore | Restore drill | Successful recovery in a clean target | Incomplete |
| Agent reads outside workspace | Sandbox/isolation test | Non-destructive verification | Blocked in documented pinned state |
The table was the operational heart of AgentOps: a control was not described by its name but by the evidence it could produce.
Memory immutability and the limits of soft controls
The memory sanitizer attempted to lock prior-day files read-only and quarantine structural leaks. This addressed a real class of failure: an agent should not rewrite its old diary entry after the fact.
But the lock did not make a model truthful.
It did not fix a stale MEMORY.md pipeline.
It did not make a Docker container isolated.
It did not replace a review of whether the hook actually ran.
Likewise, documented maker timeouts and exec-call limits had not been described as hard controls unless the runtime enforced them. The review explicitly distinguished instructions and detection from prevention.
The dangerous comfort of tool names
The configuration included memory_search and memory_get in several tool surfaces, but the active configuration globally disabled memory search and did not configure an embedding backend.
I discovered this the way one discovers most documentation drift: not through a systematic audit, but through a failure that shouldn't have been possible given what the files said.
A tool listed in an AGENTS file was not necessarily available to the running agent.
An agent's statement that it searched memory was not evidence that the search ran.
This is the kind of thing that looks obvious in retrospect and was completely invisible before the investigation. The tool was documented. The tool was configured. The tool did not exist at runtime. Three separate files all agreed the system had a capability it did not have. I had, in effect, built an agent that could confidently report on searches it never performed — which, when you say it out loud, is a fairly precise description of a language model's relationship with truth in general.
What AgentOps changed
Before an autonomous session, a pre-flight record was needed:
- gateway and model version;
- enabled cron jobs;
- workspace and sandbox mode;
- tool allow/deny policy;
- expected output files;
- backup state;
- and the check that would run afterward.
After the session, the record included:
- the output file;
- the execution log;
- metrics and sanitation results;
- errors and retries;
- operator disposition;
- and, for important actions, an independent end-to-end test.
That was more paperwork than "the agent ran."
It was also the difference between an experiment and an anecdote.
What remained unsupported
- Longitudinal metrics evidence was limited.
- Doctor coverage was incomplete for backup restoration and every live service condition.
- Lexical scoring did not establish semantic novelty.
- Memory-search documentation and runtime configuration disagreed.
- Sandbox and gateway hardening required independent deployment tests.
- A script's existence did not prove it was invoked by cron or that its failure stopped the session.