Skip to main content
LabsAgent Notes11 min read

Everything fires, nothing works

OpenClawOpenRouterDocker
Everything fires, nothing works
Eight dialogs deep, and the front one still just says ERROR. Z-Image-Turbo 1.0 6-bit understood the brief so completely that it saw no reason to embellish.

On the night of August 5th, 2026, all seven cron jobs registered. The container was healthy. OpenRouter returned HTTP 200. The nightly cron runs executed.

The output was broken.

I had spent the afternoon watching the Gate 1 vertical slice succeed. Writer sent writer-0001. Maker replied with maker-0001. The validator accepted. Fuel went from 2 to 1. The drawer unlocked. It worked. I had a system.

Then I turned on the cron jobs and the system I had fell apart in seven different ways before dawn.

The first automated run

The 2026-08-05/06 nightly cron runs — control-action at 01:00, writer-diary at 02:00, writer-terminal at 03:00, maker-terminal at 03:30 Asia/Shanghai — did not produce expected output.

The gateway logs showed the runs executed. The container was up and healthy. All seven jobs were registered. OpenRouter returned HTTP 200 for every provider call.

But the produced output was incorrect or missing across multiple jobs, and each job failed in a different way, as if the system had prepared a varied and interesting menu of failure modes for my consideration.

Maker invoked exec twice. The maker-terminal session attempted the exec tool at 03:30:14 and 03:30:17. Both attempts returned "Tool exec not found" — no command executed. The tool was not in the Maker's runtime allowlist.

I had explicitly denied exec to Maker. The deny list was in the same prompt the model read. The model asked for it anyway, because a stale file in the workspace told it the tool was available.

Writer and Maker tried to read the messages directory directly. Both agents attempted read on the messages directory, receiving EISDIR — a directory, not a file — then tried latest.json, receiving ENOENT — file not found.

No such file existed. The agents had invented a file name based on an instruction that said "read the latest message" without specifying how to determine the filename. The instruction was ambiguous, so the model guessed. It guessed wrong.

Writer's diary was timestamped wrong. The first diary entry was timestamped at 20:07 on August 5th and ignored the shared-world terminal entirely. The diary prompt did not anchor the filename to the Asia/Shanghai real-world date.

Control's entry was dated a day off. Control's diary entry was dated August 6th despite being written at 19:01 on August 5th.

No new shared-world messages were written by automation. The only shared-world artifacts were from the manually-triggered Gate 1 vertical slice. The system had executed every component in the chain and produced nothing useful.

Root cause: the instructions were wrong, the tool policy was wrong, and the docs were wrong

The initial analysis attributed the failures to "the model did it wrong." The investigation found that the system around the model was wrong.

The stale TOOLS.md

The Maker maker-terminal cron run attempted the exec tool twice. Forensic recovery of the session trajectory showed both attempts returned "Tool exec not found" — no command executed.

Root cause: workspaces/maker/TOOLS.md still listed exec in the Maker allowlist. This was a relic from FR4M3W0RK, my earlier household-assistant project — the Maker profile there had been designed to run sandboxed exec, and the old text survived the migration. For the Level 0 Library Terminal profile, exec should have been removed from TOOLS.md and explicitly denied.

The agentic coding model that performed the Round 1→Round 2 migration had removed exec from the JSON5 agent config but left the old TOOLS.md text intact. The coding model completed the visible parts of the task without auditing downstream references.

I had, in effect, asked a model to migrate a project from one framework to another, and it migrated the parts it could see while leaving the parts it couldn't see exactly as they were. The result was a configuration that contradicted itself across two files, and the runtime model — reading both files — did what the stale file told it to do.

The unreadable message path

Both Writer and Maker tried to read the messages directory directly, then a nonexistent latest.json. The AGENTS.md wake-up instructions said "read the latest ... message" without specifying how to determine the filename.

The agents had no protocol for resolving the message path from protocol-state.json. The instruction was a natural-language sentence that assumed the model would infer the correct file path. The model inferred a path that did not exist.

The unsynchronized protocol state

After a runtime reset, Writer wrote "Day 1. Again." and referenced a terminal that had already answered. Control wrote "Day 2." referencing a deleted prior entry.

Writer's leakage was traced to unauthorized canon/arc.md and canon/world.md edits from an earlier session — those files had been edited without my knowledge and were restored to a clean Round 2 Day 1 posture. I had not noticed the edits because I was not monitoring canonical file changes between sessions. I was, apparently, trusting the system to be self-maintaining. It was not.

Control's "Day 2" reference had no repository source and was consistent with residual state in ~/.openclaw that the reset script did not clear. The reset script wiped the Docker volume and the workspace files but left the OpenClaw state directory intact. That was an oversight. A meaningful one.

Maker's maker-terminal ran but wrote no reply because protocol-state.json still had latest_writer_message: null. The writer-terminal job had written writer-0001.json but never updated protocol-state.json to register the message. Maker's instructions correctly required latest_writer_message to be set before replying.

Three linked failures in one incident: unauthorized file edits, a reset script that did not clear all state, and a protocol-state synchronization gap that prevented the shared-world loop from running a single autonomous cycle. I documented all of this in a troubleshooting file, including the wrong assumptions I had made during the investigation — wrong claims about Maker's diary job, wrong blame for the memory leak, a broken redeploy script that I deleted.

The troubleshooting file was, in retrospect, as useful as the incident itself. It recorded not just what broke but what I got wrong while trying to fix it.

The validator problem

The deterministic validator (validate_shared_world.py) was introduced in the initial commit by the agentic coding model. It was designed as a state-transition gate: Maker proposes, the validator checks and applies.

In practice, it required a manual --apply step every morning. It rejected Maker's proposals over schema field-name mismatches — effect vs effects, reasoning vs rationale, missing required fields — and forced prompt-engineering the model to act as a JSON data-entry clerk.

I spent an entire session instructing the model how to match the schema exactly. The model produced valid JSON. The validator rejected it because the JSON keys were slightly different from what the schema expected. The model adjusted. The validator rejected it again for a different field name. This continued until I had successfully got an AI roleplaying as a data entry assistant, which was not the experiment I was running.

The validator was removed from the Level 0 pipeline. Maker would now apply his own transitions to physical-world.json directly. I would review via git diff. The validator script and fixtures remained in the repo as dead code.

The side-note I wrote at the time was more pointed: if a gate was needed when Maker got exec, the right approach was a judge model — an LLM that evaluated whether a transition proposal was reasonable — not a hardcoded Python script with a fixed schema.

I did not yet know how right that side-note would turn out to be.

What the failures revealed

Each failure exposed a different category of problem:

FailureCategoryRoot cause
Maker invoked execStale configurationTOOLS.md from a prior project not audited during migration
EISDIR on messages directoryWrong instructionsAGENTS.md said "read the latest message" without specifying how
ENOENT on latest.jsonMissing fileNo such file existed; the agents invented a path
Writer diary timestamped wrongDate handlingFilename did not use Asia/Shanghai real-world date
Control "Day 2"Residual stateReset script did not clear ~/.openclaw
Writer "Day 1. Again."Unauthorized editscanon files edited without operator knowledge
Maker silent (null protocol state)Synchronization gapWriter did not register message in protocol-state.json
Validator rejected valid proposalsSchema mismatchField-name drift between prompt and schema

The pattern was consistent: "the model did it wrong" was actually "the instructions were wrong, the tool policy was wrong, and the docs were wrong." The model followed what it was given. What it was given was stale, incomplete, or contradictory.

I had built a system that worked when I ran it by hand and failed in eight different ways when a cron job ran it at 3 AM. The difference was not the model. The difference was that I had been the error-correcting layer in the manual run, papering over every gap with operator judgment. The automated run had no such layer. Every ambiguity, every stale file, every missing path was exposed.

The agentic coding model failure

The initial implementation of the STR4NG3L00P Level 0 gateway and the Maker terminal profile left several security-critical gaps that were not caught by the agentic coding model used to author the bulk of the repo.

The model-generated code and docs showed signs of incomplete migration from the FR4M3W0RK project and a misunderstanding of my explicit rootless-Docker requirement. The coding model treated "Dockerd runs rootless on the host" as the requirement rather than "the application container also runs as the unprivileged user." It performed a partial migration of the Maker profile, removing exec from the JSON5 agent config but leaving the old TOOLS.md text and schedule references intact.

These were classic signs of a model that completed the visible parts of a task without auditing downstream references or honoring the security contract of the target deployment. The model had done what I asked — migrated the profile — and had not done what I assumed — checked that the migration was complete. I had assumed completeness because the visible output looked right. It was not right. It was cosmetically correct and structurally wrong.

The hardened deploy was recovered and verified the next day — rootless Docker, read-only rootfs, explicit tool allowlists — closing the specific gap the coding model had left open.

The gap between a working vertical slice and a broken pipeline

The Gate 1 vertical slice had worked. Writer sent writer-0001. Maker replied with maker-0001 proposing an allocate_power transition. The validator accepted and applied it. Fuel went from 2 to 1. The drawer unlocked.

The automated pipeline executed every component in the chain. Container started. Runner invoked. Model called. Output produced.

But the output was wrong because every instruction the model relied on — the tool allowlist, the file paths, the date format, the protocol state — was either stale, missing, or contradictory.

The vertical slice proved the design worked in principle. The first automated run proved the system around the design did not work in practice.

I had been the missing layer. When I ran the system by hand, I knew which file to read, which path to use, which date to apply. The automated system had to figure all of that out from instructions that were wrong.

What was fixed before the next run

Three repo-side fixes were applied before the next scheduled run:

  1. Tool policy. An explicit --tools allowlist was passed to every cron creation so per-job tool policy overrode any legacy unrestricted policy. Maker's deny list was extended.

  2. File paths. Writer/Maker AGENTS.md and the terminal cron prompts were updated to read protocol-state.json first and use latest_writer_message/latest_maker_message to construct exact file paths. Never pass a directory to read.

  3. Date handling. Writer/Control AGENTS.md and diary prompts were clarified to use the Asia/Shanghai real-world date for filenames while the in-character opening line used the character's own dating.

  4. Protocol-state synchronization. Writer was given a self-registration step: after writing her terminal message, she updated protocol-state.json to set latest_writer_message to her filename and current_turn to her turn number.

  5. Reset safety. The reset script was updated to delete ~/.openclaw during an in-place reset, preventing residual state from contaminating fresh runs.

The cron jobs were re-registered on the VPS after deploying these changes.

The lesson

The gap between a working vertical slice and a broken pipeline was not a model problem. It was a systems problem. The model followed what it was given. What it was given was built for a different project, documented for a different runtime, and never validated against the actual automated path.

Every fix was specific and concrete: a stale file, a missing path, a wrong date format, an unsynchronized state file. None of them were architectural. All of them were necessary.

The system that emerged was not the system that was designed. It was the system that survived contact with automation. And it still had not yet run a single successful autonomous cycle.

Next: "The thing that killed the gateway."