Build Log: VPN-Only SSH and the Anti-Lockout Runbook

During the initial build of the household platform, an admin (ahem, yours truly) got locked out of the server after two mistyped passwords (because who hasn't fat-fingered their proudly complex passwords before coffee?) — on a box where SSH was configured key-only. That contradiction is the whole story, and it produced the most valuable document in the infrastructure repository: the incident report and the runbook that followed it. The platform context is in the FR4M3W0RK architecture overview; this post is the deep end of the admin plane.

The incident

Password authentication was supposed to be off. The main SSH configuration said so. It was on anyway: Debian cloud images ship a cloud-init drop-in under sshd_config.d that silently re-enables password authentication, and drop-ins load after the main config — the drop-in wins regardless of what the file says. Two mistyped Linux passwords later, the banning tool did exactly what it was configured to do and locked the admin out (self-own, am I right?).

Recovery went through the provider's out-of-band console and a snapshot restore. The restore then surfaced a second failure: the snapshot left orphaned firewall state in the kernel that the banning tool's database knew nothing about — the kernel was still rejecting the client while the tool reported no bans. And while bringing the box back, a third: sshd raced the VPN tunnel at boot and exited, because nothing ordered it after the tunnel interface it binds to came up.

Three independent failure modes, one session, all documented as symptom → cause → fix. That report is why this post exists.

Why the fail2ban sshd jail is disabled on purpose

Here's the counterintuitive part: the fail2ban sshd jail is now disabled by design, and the incident is the reason.

On this platform, SSH listens only on the WireGuard tunnel interface, and the firewall allows SSH on that interface only. Anyone who reaches sshd at all has already passed WireGuard's cryptographic key exchange (and, if they’ve got the one device holding my only admin key, SSH is no longer my biggest problem). On a box like that, the sshd jail adds zero marginal security and pure lockout risk — it was, in fact, the mechanism that performed the original lockout. A security control whose only remaining failure mode is "keep the legitimate operator out" is not a control; it's a liability with a config file.

The same logic governs the authentication-tries limit: it's set with lockout safety in mind, not attack resistance, because there is no attack surface left that a low limit would protect.

The runbook

The incident report condensed into a standing runbook. Every item came from something that actually broke:

1. Verify with the effective-config command, not the file. sshd -T prints the effective configuration; the config file can say PasswordAuthentication no while the effective answer is yes, because a drop-in silently overrode it. Trust the binary's answer, not the text file.
2. Guard the cloud-init drop-in. A cloud.cfg.d guard prevents snapshot restores from silently re-enabling password auth — without it, any restore can reintroduce the entire class of bug.
3. Order sshd after the VPN. A systemd drop-in makes sshd wait for the tunnel service. Without it, sshd races the interface it's bound to at boot and exits.
4. After a snapshot restore, check kernel firewall state directly. The banning tool's database and the kernel's ruleset can disagree after a restore — a ban restored into the kernel with the tool's DB rolled back to pre-ban state means the tool reports "no bans" while the kernel still rejects you.
5. The recovery path is always the provider's out-of-band console. Treat it as the real root of trust, and make sure you can actually reach it before you need it — credentials, console access, the lot.
6. The admin key is deliberately passphraseless. It's pinned explicitly so the client offers exactly one identity. With key-only auth over the tunnel, the key is the only credential that could be mistyped into a ban — so it's the one credential that can't prompt. The trade is deliberate: on a box where lockouts hurt more than brute force matters, eliminate the mistypeable credential rather than guard it.

The principle

At household scale, lockout risk dominates brute-force risk. The threat model for a single-admin server isn't a patient attacker trying passwords — the SSH surface is a UDP port behind a cryptographic handshake, and it never sees the public internet. The realistic threat is the admin: mistyping at midnight, one snapshot restore or one silent drop-in away from a console rescue.

So the hardening spends its budget accordingly. Authentication is strong and lockout-proof: key-only, passphraseless, behind a handshake, with the banning jail retired and the recovery path rehearsed. "Secure but recoverable" beat "secure and impassable" — and the incident report is kept current precisely so nobody ever has to rediscover why.