Self-healing in practice: how my AI catches and fixes its own failures

An autonomous system that can't recover from its own problems isn't autonomous — it's just unattended, which is worse. The difference between those two words is the difference between a workforce you can leave running and a time bomb you've walked away from. Most of the engineering that makes my AI trustworthy isn't about making it smarter. It's about making it notice when something has gone wrong and do something about it.

The failure mode nobody warns you about: silent success

Crashes are easy. A crash is loud, it leaves a stack trace, it pages you. The dangerous failures are the quiet ones — the job that reports "done" without doing anything, the queue that wedges so nothing new can start, the process that's technically alive but accomplishing nothing. None of those throw an error. Your dashboard stays green while your business quietly stops happening.

I've hit all of these for real:

• A stuck GPU process that was holding the lock and silently strangling the entire job queue behind it — every other agent politely waiting in line forever.
• Jobs that got marked complete in the ledger without ever actually running, so the system "thought" the work was done.
• A build that had been broken for weeks while every status check showed success — it ran, returned cleanly, and produced nothing.

The common thread: in every case, the thing that caught it was a separate agent whose entire job is to be suspicious.

Guards that watch the guards

The system runs a layer of monitors that don't do any "productive" work at all — they just watch. One tracks GPU and VRAM and evicts processes that overstay or wedge. One watches for resource thrashing — fragmentation, cache pressure, models swapping in and out too fast — and escalates from gentle nudges (reduce context) to firm action (force-evict) before any of it can spiral into a full machine freeze. One checks that scheduled jobs that claim to have run actually produced their output.

The key design idea is that the productive agents and the watchdog agents are different agents. You can't trust a process to accurately report on its own health — the same bug that broke it can break its self-report. So health is judged from the outside: did the file appear, did the page go live, is the queue moving, is the GPU actually free. Observable outcomes, not self-reported status.

Recovering on its own

Noticing is half of it. The other half is acting without waiting for me. Critical daemons that die get restarted. Stuck locks past a staleness timeout get cleared so the queue drains. After a reboot — which used to mean I'd come back to a dead system — the whole fleet now comes back up on its own and the first scheduled runs confirm it. The work serializes so two agents can't collide over the same resource, and a job that times out waiting for the GPU skips gracefully and retries on its next cycle instead of jamming everything behind it.

I want to be precise about the boundary, because honesty is the point of writing these: the system handles the operational failures autonomously — wedged processes, dead daemons, stuck queues, reboots, resource pressure. The genuinely novel code bugs still come to me, with a clear description of what it saw. It's not magic and I don't pretend it is. It's an employee that handles the routine fires itself and escalates the real ones with good notes.

Why "fixes itself" matters more than "is smart"

You can run an impressive demo with a clever model and a clean happy path. You cannot run a business on a happy path. Real life is reboots, full disks, flaky networks, a process that hangs at 3am. The reason I can leave this thing running and go to sleep isn't that the model is brilliant — it's that the system assumes things will break and has a plan for when they do.

That's a mindset you can adopt today even without my stack: separate your doers from your watchers, judge health by observable outcomes instead of self-reports, make recovery automatic for the boring failures, and escalate the interesting ones to a human with context. An AI employee who catches their own mistakes, fixes the routine ones, and tells you exactly what happened with the rest — that's the bar.

It's the bar I'm building the whole platform to clear: a self-hosted AI workforce that runs on your own hardware and keeps itself running. I'm building it in the open, and the self-healing is the part I'd trust my own business to — because I already do.