Architecture Notes: A Boring Cron-Driven Rig for Long-Horizon Agent Experiments | Labs

STR4NG3L00P is my long-horizon experiment with generative agent personas: a diary-writing character, a shared-world collaborator, and an A/B control variant used in comparison rounds, run as scheduled, non-interactive writing jobs on a single small VPS. All model inference is delegated to a hosted LLM API over HTTPS. The infrastructure is deliberately boring by design — no chat gateway, no listening services, no agent-facing APIs — because the experiment is about the agents' behavior, not the platform. The interesting engineering is almost entirely about constraining the agents and keeping one operator in sole authority over the host.

This post is the standing reference for the rig as it exists today: what it refuses to have, how it's built, and the rules it enforces. The story of how it got this shape — including the death of the platform it replaced — is told chronologically in the Agent Notes series.

Constraints as features

The fastest way to understand the rig is by what it doesn't have, and what each absence buys:

No chat gateway. The jobs are cron-driven and non-interactive; a gateway would be a scheduler the operator doesn't control wrapped around an attack surface the experiment doesn't need. (How the previous platform died of exactly that is its own story.)
No listening services. No ports are published anywhere — not in the compose files, not in the unit files, not in the image. The only network activity is outbound: the inference API, and the backup target.
No long-running services. Every job is a one-shot container: a timer fires, a container runs, the container exits. There's no restart policy anywhere. A dead job is a durably recorded missed turn, not a zombie process or a 3 AM restart loop.
No orchestrator. One host, declarative YAML, and rendered systemd units a person can read. For one box, that is the platform layer.

Each of these was paid for — the series documents what each constraint cost to learn — but the standing architecture treats them as load-bearing.

The standing system

The scheduler is host systemd. User-level timers — owned by the deploy user, surviving logout thanks to a one-time enable-linger — launch one-shot rootless Docker containers (docker compose run --rm). Enabling linger is the single root action in the entire lifecycle; ordinary deploy, schedule, and pause operations need no sudo, which shrinks the blast radius and the operational friction simultaneously.

The runner is a small declarative Python program. Runner code is baked into the container image at build time, stamped with the git commit; configuration, personas, and prompts are bind-mounted read-only from the checkout. The runtime volume is the single writable mount — generated diaries, reflections, shared-world state, judge records, locks, and reports — and generated output never enters Git.

The job configuration is declarative YAML. jobs/*.yaml is the single source of truth for schedules, ordering, retry policy, declared context reads, and typed output contracts. A renderer generates the systemd units from it; the rendered units are committed as a review snapshot and byte-checked by validation, so drift between the config and the units is a validation failure, not a surprise. Disabling a job renders no timer, and install prunes the orphaned one.

The trust boundary

The rig runs unattended agents — nobody is in a chat to click "approve." So the trust model doesn't ask the model to behave; it removes misbehavior as a capability:

The model's only capability is returning typed JSON that the runner validates against a schema. Text that isn't a valid contract doesn't reach anything.
All filesystem access is policy-enforced against per-job declarations — deny before allow. A job reads only the files its contract declares and writes only its declared outputs.
Locks, state-version conflict checks, and atomic writes are owned by deterministic runner code. The agent proposes; the engine commits.
The characters coordinate exclusively through files in the runtime volume — a protocol-state file plus per-turn message files — with a second, bounded LLM acting as a judge over proposed state transitions. Only deterministic code ever mutates canonical state.

The container layer backs that contract up: read-only root filesystem, every capability dropped, no Docker socket, no exposed ports. Each of those hardening choices has an incident behind it, documented in the series — the deployment gauntlet and the invisible-failure review cover the worst of them.

It's the same stance as my household agents' prompt-injection hardening, translated for the unattended case: there, every state-mutating action passes a human approval gate; here, no human is in the loop, so the gates are deterministic — schemas, declared capabilities, and code that never lets the model near a mutation it doesn't own.

The time/retry contract

Systemd owns time; the runner owns retries. Timers never restart failed jobs. Retry policy — attempts, backoff, slot window — is declared per job in YAML and enforced by the engine; a turn that misses its window is durably recorded as missed rather than replayed hours later. For a behavioral experiment, replaying a "daily diary" turn at the wrong time is worse than skipping it.

Cross-job ordering is enforced twice: in the timer units (ordering directives) and by a deterministic completion gate — the reply job won't run unless a committed same-date writer turn exists. Two layers, because a schedule constraint that only exists in the schedule isn't a constraint on what actually ran.

Image currency as an operational invariant

Because runner code is baked into the image, a bare git pull leaves the timers running stale code — and that genuinely invalidated a prior experiment round. The remedy is process, not tooling: a single update command (pull → rebuild → revalidate) is the only supported code path, and a status check compares the commit stamped at build time against the checkout, warning loudly on drift. "The deploy worked" and "the deployment is running what's in the repo" are different claims, and only one of them is checked automatically.

Three-tier data separation

Git holds the source of truth: config, personas, prompts, seeds.
The VPS runtime volume holds all generated state — nothing else writes there.
Encrypted offsite snapshots (restic, with retention measured in days to years) hold the runtime tier.

Round transitions are clean resets, not migrations: the runtime volume is deleted and re-seeded from source-controlled seed state, with pre-reset reports and a snapshot taken first. Reset semantics are fail-closed — destructive resets require an exact runtime-root confirmation plus a recorded backup ID — so a rollback can never silently overwrite narrative state.

Observability without dashboards

The only operational surfaces are a single operator CLI — one script wrapping validate, build, update, install, status, run, report, export, restore, and backup — and a generated static Markdown report: run status, judge decisions, state changes, rejected and malformed outputs. Backup failures notify through the journal. Boring, reviewable, and impossible to accidentally expose. There is no dashboard to secure and no metrics port to forget about, which is a feature wearing overalls.

Provisioning: IaC-lite, on purpose

No Terraform, no Ansible, no Helm. Everything is declarative config in the repository plus procedural operator steps, on the theory that a one-host research rig should be reviewable, not templated. The documented sequence — host prep, bootstrap, validate, build, seed, install, secrets, update — is the deployment. When the rig outgrows a single host, that decision gets revisited; until then, the template would be the thing to maintain.

The principle

The rig was designed for exactly one deployment context: a single operator on a single VPS running cron-driven one-shot jobs with no external surface. It needs no device pairing because there is one device; no gateway because there is one caller; no scope management because there is one scope. The reduction isn't a compromise — it's the correct architecture for the actual deployment, and everything "missing" from it is missing on purpose.