Thirteen nights
On the night of August 12th, all four timers fired on the VPS for the first time. Every container died at entrypoint. No run records were written.
It was thirteen nights into Round 2, and each night I fixed one thing only to discover the next layer was broken. By the end, I understood something I had not understood at the start: the gap between "works on my machine" and "works under systemd at 3 AM server time" is measured not in lines of code but in layers of assumptions.
The validation
On August 8th, the OpenClaw→runner migration validated on a disposable runtime. 111 tests. Schema meta-validation. Systemd unit render and sync. Three reviewed Library Terminal exchanges with judge auditing.
The exchanges proved the protocol worked on the new runner. Writer sent a terminal message. Maker replied proposing an allocate_power transition. The judge accepted. Fuel went from 2 to 1. The drawer unlocked. Writer acknowledged the state change. The judge rejected a bad proposal and left state unchanged. Both decisions were recorded as events.
Gate 2 — a 7–14 day pilot — was approved conditional on VPS install and one clean real-scheduled day. I had the validation record. I had the tests. I had the reviewed exchanges.
I did not yet have a working deployment.
The remediation
An independent implementation review on August 8th found problems I had not noticed. The generated systemd units invoked run --job %i without --profile, while the runner CLI required --profile. Scheduled jobs would fail at argument parsing. Systemd owned retries with Restart=on-failure while the contract required runner-process retries. Missed turns were inferred from absent records rather than written durably. Writer-before-Maker was only timer ordering, not a completion gate.
I spent four days fixing these. Profile authority. Retry ownership. Completion gate. Harness bootstrap. Reset safety. Documentation reconciliation. Migration export. Test count rose from 111 to 138.
The remediation was methodical and necessary. Each fix was small and specific. None of them were architectural. All of them were the kind of work that separates "the design is sound" from "the implementation matches the design."
By August 12th, the remediation was complete. The runner was at 138 tests. The harness passed. The CLI and docs were mutually consistent. I installed the timers on the VPS.
Night one: the code was wrong
On August 12th, all four daily timers fired. No run records were written.
Root cause: the runner image copied the package to /opt/STR4NG3L00P/runner but set no WORKDIR or PYTHONPATH. python -m runner.cli ran from CWD / and hit ModuleNotFoundError: No module named 'runner'. Every container died at entrypoint.
The timer journal proved the scheduler worked end-to-end. Rootless Docker, compose, volume, network — all functioned. The container logs proved the code did not.
I had built a runner that passed 138 tests and could not import itself.
Fix: the Dockerfile gained WORKDIR and PYTHONPATH. The compose service gained working_dir. The runbook gained init-runtime as a required install step.
Night two: the code was right but the system hid the filesystem
On August 13th, all four daily timers fired. Zero provider calls. Zero run records.
Journal showed every service failed before Docker ever spawned: Changing to the requested working directory failed: No such file or directory.
Root cause: the installed unit rendered WorkingDirectory=/home/appuser/STR4NG3L00P and carried ProtectHome=true, which hides /home from the service. Systemd could not chdir into the checkout. Docker never ran.
I ran the same command interactively — bash deploy/STR4NG3L00P run writer-diary --date 2026-08-13 — and it succeeded in 16.4 seconds with a real provider call and a committed run record. The pipeline worked interactively. Only the systemd unit killed it.
The 08-13 terminal slots remained deliberately un-replayed under the no-delayed-replay policy, becoming the first missed turns in the experiment record.
Fix: ProtectHome=true removed from the service template. ProtectSystem=strict retained. Explicit TimeoutStartSec derived from job retry windows — the stock 90s default would have SIGTERM'd real inference runs.
I had been running interactive verification commands that did not exercise the systemd unit. The timer journal had not been inspected after the first night. I had been looking at container logs, not systemd journals. The failure was in a layer I was not watching.
Night three: the provider returned silence
After the ProtectHome fix, the first systemd-path run reached inference, then crashed unhandled.
OpenRouter returned HTTP 200 with message.content == null. extract_json_object(None) raised AttributeError: 'NoneType' object has no attribute 'strip'. The process exited 1 with no run_end record.
A provider returning HTTP 200 with no content is a silent failure mode. The call "succeeded" but produced nothing. The crash-vs-missed distinction was the difference between losing data and recording a durable absence.
Fix: null or blank content now raised the retryable ProviderTransientError, so the engine retried within the slot window and ended with a durable missed record instead of crashing.
The first unattended day
On August 14th, all four daily timers fired on schedule. For the first time, every firing produced a run record through the systemd path.
control-action succeeded. Writer-diary succeeded. Writer-terminal failed output_invalid — the model invented id writer-t1, two characters too short for the schema. Maker-terminal failed gate_blocked — the completion gate correctly refused because Writer had no committed same-slot run.
The runner pipeline was proven end-to-end. Timer, Docker, container, runner, record. But the shared-world loop still had zero Writer messages, because the model invented an ID that was one character too short.
Root cause: the engine validated the model's raw envelope against the contract schemas before overwriting the runner-owned identity fields. A placeholder ID that would always be replaced could still fail the run.
I had built a schema validation layer that rejected its own inputs before the engine could fix them.
Fix: the engine now derives the actor prefix from the contract, allocates the ID to always match the per-contract pattern, and injects it into the model output before schema validation.
The first green day
On August 15th, every daily job succeeded and the Writer→Maker chain completed on the VPS runtime for the first time.
control-action succeeded in 26 seconds. Writer-diary in 31 seconds. Writer-terminal in 15 seconds — writer-0001 committed. Maker-terminal in 17 seconds — maker-0001 committed, gate passed.
The shared-world loop had its first exchange. Writer sent a terminal message to the library. Maker replied. Maker carried no transition proposal — by design. The judge was not called. The state did not change. But the loop was open.
After thirteen nights of fixing one thing and discovering the next layer was broken, every job succeeded for the first time.
What the thirteen nights proved
Each night revealed a different class of problem:
| Night | Date | Failure | Class |
|---|---|---|---|
| One | Aug 12 | ModuleNotFoundError: No module named 'runner' | Build: image import path |
| Two | Aug 13 | CHDIR: No such file or directory | Deployment: systemd security |
| Three | Aug 14 | AttributeError: 'NoneType' object has no attribute 'strip' | External: provider null content |
| — | Aug 14 | output_invalid: writer-t1 (2 chars, needs 3+) | Schema: runner-owned identity |
The escalation was structural. Night one: the code was wrong. Night two: the code was right but the system hid the filesystem. Night three: the system worked but the provider returned silence. Night four: the provider worked but the model invented an ID one character too short.
Each fix revealed the next dependency. The interactive-vs-systemd gap was the deepest lesson: the pipeline worked when I ran it manually but failed when systemd ran it, because systemd's security model hid the checkout directory. I had been the missing layer again — not in the code, but in the verification. I was testing the path I ran, not the path that actually fired.
The ProtectHome=true finding was the pattern that would repeat throughout the project: the system was configured correctly for one deployment context and incorrectly for another, and the mismatch produced no errors until the mismatch was exercised. You do not discover that ProtectHome hides /home by reading the systemd documentation. You discover it by watching every timer fail with the same error and realizing you have been testing the wrong path.