Worst case, I lose three minutes: how I made my AI's memory impossible to kill

If my laptop died right now — dropped in a lake, drive failure, stolen — I would lose, at most, about three minutes of work. Not three days. Not "whatever was in the last backup." Three minutes. That number is the single most important design decision in the whole system, because everything my AI knows about my business lives in its memory, and memory you can lose is memory you can't trust.

Memory is the business, so memory is the thing you protect

An autonomous AI workforce isn't valuable because of the code — code is replaceable, it's on a dozen machines. It's valuable because of what it has learned: every project, every preference, every decision, what worked and what didn't, the accumulated context that makes it act like it actually knows my business instead of meeting me fresh every morning. Lose that and you don't have a degraded system, you have a stranger.

So I treat memory the way a bank treats money, not the way most apps treat logs. The rule, in order of priority: back up memory first, back up everything else second.

Three minutes, two locations, one watchdog

The setup is deliberately boring, because boring is what survives:

• Near-real-time off-site: any change to memory is pushed to private off-site storage within about three minutes. Not nightly. Minutes.
• A second, independent location on a slower cadence, so a single remote going down doesn't leave me with zero copies.
• A watchdog whose only job is to confirm the backups are actually happening — and to revive the backup process and email me if it ever stops.

That last one is the part most people skip, and it's the part that actually matters.

The bug that taught me to watch the watchdog

Here's the war story. The backup watchdog itself silently died — and stayed dead for nine days before I noticed. The cause was subtle and a little embarrassing: it identified its own running process by a process ID, and the operating system had recycled that ID to a completely unrelated program. So the watchdog looked up, saw "a process with my ID is running," concluded "I must be healthy," and went back to sleep — while the actual backup work hadn't run in over a week.

Nothing was lost that time, because the near-real-time push was still going. But it was a perfect lesson: a backup you don't verify is a backup you don't have. The fix was to check identity properly, not by a recyclable number, and to make the health check assert that backups actually landed recently rather than that some process is nominally alive. Same principle as everything else I build — judge by observable outcomes, never by self-reported status.

The other quiet failure: the backups that fought each other

A second one worth sharing: at one point multiple backup processes were all trying to push to the same remote at the same time, and they collided — stepping on each other's commits and stalling the whole thing for hours. The lesson there was about coordination: now the pusher pulls-rebases-and-retries on conflict instead of blindly racing, so concurrent backups queue cleanly instead of deadlocking. Resilience isn't one feature; it's a hundred small "what happens when two of these run at once" decisions.

Why I sleep fine

People hear "autonomous AI running my business unattended" and reasonably think: that sounds terrifying, what if it all vanishes? The honest answer is that the vanishing is the failure mode I engineered against hardest, first, and most paranoidly. The worst realistic case is three minutes of lost context and a fresh restore from off-site — a one-command operation I've actually tested, not a hope.

If you're running anything autonomous, steal the priority order more than the specifics: protect the irreplaceable thing first, back it up off-site on a tight cadence, keep a second copy, and — most importantly — put a watchdog on the backup that verifies real outcomes and screams when they stop. Then test the restore, because a restore you've never run is a theory, not a safety net.

That paranoia is baked into the platform I'm building — a self-hosted AI workforce that runs on your hardware and is genuinely hard to kill. I'm building it in the open, and "it survives anything, worst case you lose three minutes" is a promise I can make because I've already watched it hold.