Three failed agent migrations taught me what LLMOps actually is
I started with a simple household-AI brief: run two personal butlers, a diary writer, and a maker on a small VPS.
The first deployment used OpenClaw with local inference.
Then I moved to Hermes Agent.
Then I built a custom Python runner because the framework and the hardware were fighting each other.
All three migrations were responses to the same constraint: a four-vCPU, 8 GB VPS was being asked to run both an agent framework and local model inference.
Each migration solved one visible problem and exposed another.
By the time I returned to OpenClaw with cloud inference, I had a better definition of LLMOps than any feature list had given me:
LLMOps is the discipline of making the model, framework, tools, deployment, schedules, evidence, and operator decisions work as one traceable system.
Prompt quality matters.
It is not the boundary of the system.
What FR4M3W0RK was meant to be
FR4M3W0RK was a self-hosted multi-agent household system.
The intended household architecture placed two Telegram butlers, a cron-driven writer, and a cron-driven maker under one gateway.
The butlers used Calendar, Tasks, and Sheets through a native JavaScript plugin.
The plugin called authenticated n8n webhooks, and n8n owned the Google OAuth integration.
Inference was provided by OpenRouter rather than by a model running on the VPS.
That was the v3 shape as it stood by July 8, once the rebuild had landed.
Some pieces had been confirmed in practice — a calendar-list workflow had been tested end to end.
Other pieces still needed proving.
Partway through the rebuild, I noticed the deployment logs referenced a newer OpenClaw build than the one I thought was running. It turned out the image had been quietly pulled to v2026.7.1 during a docker compose build --pull on July 13, but the gateway process had never been restarted — the old v2026.6.11 was still handling requests. When the first restart happened on July 14, every cron job broke with a TypeError in the new version's cron module. I pinned the Dockerfile back to openclaw@2026.6.11 and had it working again by July 15. The version that ran in production was the older one, not the one the STATUS.md file at the time listed at the top.
(I would like to state for the record that I did not ask for this upgrade. The image pulled itself during a routine build. I was unaware that my production environment had changed underneath me until the restart made it impossible to ignore.)
The maker sandbox was another example of a gap between what the repository declared and what the runtime actually did. The repository contained a restored sandbox block, but the v2026.6.11 gateway rejected it at startup. The maker had exec access inside the gateway container, but the intended Docker-level isolation was not active.
The migration timeline
Snapshot What failed What it taught us
────────────────────────────────────────────────────────────────────────────────────────────────────
OpenClaw + local inference Version/schema/pairing friction Framework boundaries must be tested
↓
Hermes + local inference Latency, context, tool overhead Performance is a systems property
↓
Custom Python runner Regex-scraped fake tool calls Text that resembles execution is not execution
↓
OpenClaw v3 + OpenRouter Destination still settling Trace the whole transaction
This was a sequence of architecture snapshots, not a ladder where each rung was universally better than the one below it.
Hermes was a real intermediate architecture.
It provided independent profiles and a simpler file-based scheduler, even though the overall performance and operational constraints remained significant.
Attempt 1: OpenClaw with local inference
The original attempt used OpenClaw as the gateway and local inference on the VPS.
The deployment produced a long list of failures.
The post-mortem recorded a nonexistent npm version pin, rejected bind syntax, missing or deprecated configuration fields, changed sandbox formats, include-path problems, and deployment-script errors.
Those were not all the same class of failure.
Some were documentation or version-drift failures.
Some were schema failures.
One was architectural: OpenClaw's device-pairing flow made multi-agent cron administration difficult because the CLI needed a WebSocket connection and an approved device scope before the operator could complete the setup.
The post-mortem described that as a design limitation rather than a missing flag.
The lesson was not "never use OpenClaw."
The lesson was that a framework is part of the runtime contract.
A configuration copied from an older version is not a configuration.
It is a hypothesis.
A feature described in a document is not a capability until the running version accepts it and the resulting behavior is tested.
Failure-mode matrix
| Boundary | Intended behavior | What actually happened | Operational lesson |
|---|---|---|---|
| Package/version | Install the documented OpenClaw release | The documented version pin did not exist in the npm registry | Pin versions that exist, then validate the installed version |
| Configuration schema | Load gateway, agents, and sandbox settings | Multiple fields had moved, disappeared, or changed shape | Treat schema validation as a release test |
| Device administration | Operator can create and manage cron jobs | Pairing and scope approval formed a setup deadlock | Test the administrative path, not only agent responses |
| Inference | VPS answers in acceptable time | Local inference competed with framework overhead | Benchmark the whole host, not the model in isolation |
| Deployment | Scripts reproduce the intended host | Backup, path, auth, and install assumptions were wrong | A deployment script is production code |
Attempt 2: Hermes Agent
The next migration moved to Hermes Agent.
It was not merely a failed detour.
Hermes offered independent profiles and a file-based scheduler, and the post-mortem recorded genuine improvements in profile isolation and scheduler simplicity.
But local inference remained the bottleneck.
The post-mortem recorded 65K minimum context and multi-pass loops producing responses that took 48 minutes or more on the CPU VPS.
It also recorded more than 50 automatically loaded tool schemas adding roughly 8K tokens to every prompt, and a thinking-mode configuration that could consume the output budget without returning useful content.
These were findings about this particular combination of model, context, tools, host, and loop design — not a verdict on Hermes as a framework.
The migration also reproduced a familiar operational pattern: the docs and the installed version disagreed.
The documented hermes cron create --tz option was absent in v0.17.0.
The native gateway install could hang waiting for dbus.
The migration script carried OpenClaw environment-variable names into Hermes, where they did not apply.
The architecture was now simpler in some ways, but simplicity in one subsystem did not erase the cost of local inference in another.
Attempt 3: the custom Python runner
The custom runner was the most instructive failure because it appeared to solve the original problem.
It reduced some performance and framework overhead.
It also moved responsibility for channels, sessions, scheduling, tool dispatch, and error handling into code I controlled.
That was where the boundary changed.
The runner did not have native structured tool calls.
Instead, the model was instructed to emit text conventions such as:
CALENDAR: personal | Dentist | 2026-07-04 10:00 | 2026-07-04 11:00
A parser then searched the response with regular expressions and attempted to convert matching lines into webhook requests.
The model could produce a line that looked like a tool call without the tool ever executing.
A parser could misread a line, fail to match it, or send a malformed request.
A webhook could accept the request while Google rejected the operation.
And a polished assistant response could imply success at every stage.
That was not function calling.
It was prompt engineering with a parser attached.
The system had confused representation with execution.
(I built this. I tested it. It appeared to work. The calendar events did not appear in Google Calendar. I should have checked Google Calendar first.)
Representation versus execution
Custom runner (historical)
model text
│
├── "CALENDAR: ..."
│
▼
regex parser
│
▼
webhook request
│
▼
external service
A textual marker could be present even when any later step failed.
The v3 architecture drew the boundary differently:
v3 intended integration
native model call
│ structured arguments
▼
OpenClaw tool router
│
▼
FR4M3W0RK-tools native JS plugin
│ authenticated webhook
▼
n8n workflow
│ OAuth-owned API call
▼
Google Calendar / Tasks / Sheets
The native tool boundary was a substantial improvement.
But it did not prove downstream success.
Typed arguments did not prove that the webhook path was correct.
A successful webhook response did not prove that Google stored the requested object.
The transaction still needed an end-to-end check.
The model protocol mattered too
The v3 design first tried GPT-OSS-120B and then removed it.
The recorded reason was protocol incompatibility: Harmony response delimiters caused tool-call JSON to appear in the content field rather than the expected tool_calls structure, with malformed names and broken parallel calls.
The project then selected DeepSeek V4 Flash through OpenRouter at a similar price point.
The important point was not that one model was good and another was bad.
The point was that a model's ability to produce fluent text was not the same as compatibility with the framework's tool-calling protocol.
Model choice belonged in the operations design.
What changed in the fourth architecture
The return to OpenClaw was not a return to the first architecture.
It changed the division of responsibility.
Responsibility map — v3 boundary
OpenClaw channels, sessions, cron, agent routing, native tool routing
OpenRouter model inference and provider protocol
FR4M3W0RK-tools typed Google-domain tools and webhook requests
n8n authenticated integration workflows and Google OAuth boundary
Google calendar, task, and sheet systems of record
Docker process and service packaging; security posture depends on mounts and user
Operator version pins, deployment, credentials, verification, rollback, and response
This map described the v3 boundary as it stood after July 8.
It was not a claim that every boundary was independently enforced.
The project's later reviews found material security caveats: the gateway Docker socket, root execution, mounted environment secrets, shared network, and the maker sandbox discrepancy.
Those findings belonged in the architecture's risk description, not in a success-story footnote.
The project's own rule was useful here:
instruction → configuration → runtime enforcement → external detection → operator response → independent verification
An instruction that said "stay in the workspace" was not isolation.
A tool schema was not a successful API transaction.
A deployment file was not proof that the live host matched the repository.
The test that changed how I think about agents
The useful test was not:
Did the assistant say it added the event?
The useful test was:
Did the requested event appear in the correct Google Calendar, with the correct fields, after the complete request path ran?
That test crossed the model, gateway, plugin, webhook, workflow, credentials, and external API.
It also forced an answer to the question that agent demos avoid: which layer failed?
The same principle applied to a scheduled writer.
A reflection that produced a good paragraph was not the same as memory being updated.
A maker that stayed inside its workspace was not evidence that a sandbox existed.
A cron job listed in configuration was not evidence that it ran at the intended local time.
LLMOps began where these distinctions became operational checks rather than editorial caveats.
What I learned
First, framework responsibilities are architecture.
Channels, sessions, cron, tool routing, and device administration were not conveniences around the "real" agent.
They were part of the real agent.
Second, performance is a stack property.
A model benchmark did not tell me whether a multi-pass agent loop with dozens of schemas would be usable on a small VPS.
Third, text is not execution.
A model could emit a convincing marker, a framework could accept a structured call, and an external service could still fail.
The only reliable story was the trace.
Fourth, configuration drift is a production failure mode.
The repository, installed package, running gateway, workflow service, and external API each had a version and a state.
LLMOps had to connect them.
Finally, migrations were not wasted.
Each one exposed a constraint that the previous design had hidden: version and pairing friction, host-level latency, context and tool overhead, protocol incompatibility, and the need for end-to-end evidence.
The migration that survived was not the one with the most elegant prompt.
It was the one whose boundaries could be named, tested, and revisited when the next version changed them.
MBA bridge: the sunk-cost trap
The custom runner was also a build-versus-buy lesson.
Once I had invested in polling, session state, parsing, and webhook dispatch, continuing to repair it felt cheaper than replacing it.
That was escalation of commitment.
The relevant cost was not what had already been spent.
The relevant question was whether the architecture could provide a trustworthy end-to-end transaction at an acceptable operational cost.
A migration was not automatically waste.
Keeping an architecture because it already existed was not automatically thrift.
Where things stood
By July 15, the v3 architecture was running on OpenClaw v2026.6.11 with cloud inference through OpenRouter.
The native JavaScript plugin exposed six tools — calendar list and add, tasks list and add, sheets read and append — through authenticated n8n webhooks.
Writer and maker ran on cron at 02:00 and 04:00, Asia/Shanghai.
The butlers had Telegram channels, though the wife's n8n credentials and Telegram pairing were still pending.
The heartbeat blocks had been removed, and the unwanted 56-plus daily API calls had stopped.
The writer's diary resilience had been fixed after an EISDIR trap showed the model was giving up on read errors instead of writing.
A 47-file repo cleanup had removed legacy paths and dead configuration.
The custom runner, Hermes, and the original OpenClaw v2026.6.10 configs were frozen in the repository as historical evidence.
The maker sandbox was declared in the repository but not enforced by the running gateway — the Docker socket was mounted, but the sandbox block was rejected at startup.
The v2026.7.1 cron regression had been found, reproduced, and pinned back.
None of this was a finished system.
It was the architecture that survived three migrations, each of which had taught something the previous one could not.