One email a week - something from which I hope you'll get real value. We talk about things we can build, and how to defend them. That can apply to cybersecurity, physical buildings, digital products, and .... just about anything. It gives me a lot of latitude in what I can write about, but the two concepts are important for progress - as individuals, and as society.
Today's topic is: Trust. But verify.
🔨 BUILD: With Your AI: Trust. But Verify.
You know those horror stories you occasionally hear about AI causing people to lose work?
That happened to me this week.
I lost two months of work.
Actually, it wasn't so much that I lost it as that it never existed. Subtle difference. Important one.
I've mentioned before that I'm working on a SaaS, and that Talos and I have been building it together. Talos is one of my AI agents. The SaaS currently lives in Hetzner as a demo site, and after my Zscaler work is done for the day, Talos and I have had several sessions where we add features, fix things, improve the site, and generally try to get it ready for what I had hoped was an imminent launch.
On Saturday, after our ten day trip to Taipei, I sat down for another one of those sessions and couldn't get into the demo site.
So I asked Talos to reset the password for the demo test account.
Talos did.
By rolling the software back to a version where he thought the password was the one I wanted.
From two months ago.
Well, okay, I thought. Not the worst thing in the world. We can just restore the site from one of the GitLab pushes he'd been making as we worked through our various sessions together. The last one should have been right before I went to Taipei. I hadn't done any work on the site while I was away, so there shouldn't be much to worry about.
I'm more than half a century old. I've been dealing with computers for forty-five years.
Apparently, I still have some things to learn.
He hadn't actually been pushing most of the changes to GitLab.
Or anywhere.
The manual backups he said he was making weren't actually being created either.
At all.
Anywhere.
So, being trained in digital forensics, I decided to find out what actually happened.
When I first sat down, the SaaS site wasn't actually responding properly. It was serving the login page, but the backend wasn't running, so it looked like my login attempts were failing. In reality, they weren't being processed at all.
When I asked Talos to reset the password, he thought there wasn't an account, because from an AI agent perspective the site wasn't really there. So AI-agent-turned-intern desperately tried to please the boss as quickly as possible, reverted to the last known good state, and started the site again.
Which would be charming if it hadn't vaporised two months of work.
We then "talked through" what he'd been doing. As it turns out, Talos had been pushing to GitLab sometimes, but not consistently. He also had not actually been backing things up to the filesystem like we thought was happening.
He was Very Sorry™.
Which is lovely, but it does mean we now have to do a lot of the work again.
There were four GitLab pushes that helped with the recovery, so it wasn't a total loss. And it is a little easier this time around, because I know what I want now and how to get there. But we are now pushing to GitLab after each and every change.
And yes, I am verifying that every time.
He's also been given strict instructions not to delete anything or roll anything back without a clear prompt from me. However, since we may well fall into the interpretation game again, that instruction alone is not enough to make things safe.
Because that's the problem with agents.
They don't just do what you say.
They do what they think you meant.
And sometimes they are wrong with tremendous confidence.
🛡️ DEFEND: With Backups
I've talked before in this newsletter about backups. Email backups. File backups. Cloud backups. Redundant cloud backups. Git for software. All of that.
Which makes this even more ironic.
I was doing the thing. Or at least I thought I was doing the thing.
But I fell into the classic trap: I had a backup process, or believed I did, but I wasn't checking it.
And it's not actually a backup if you can't restore from it.
That's the part everyone knows intellectually and still manages to ignore in practice. The tricky bit is that you don't think you're going to need to restore.
But you always need to be able to restore.
Especially these days.
AI changes the backup problem, because AI agents can make decisions and take actions at speed. They can delete, overwrite, roll back, rename, move, refactor, "clean up," or "fix" things before you even realize what happened. And because they are fluent, they can also confidently tell you they backed something up when they didn't.
That doesn't mean the agent is malicious. It means the agent is not a backup system, and any backup systems it claims to have need verification. Consistently, over time.
A backup is only a backup when the thing exists somewhere else, you know where it is, and you have tested that you can get it back.
This applies to software, obviously. Commit early. Push often. Verify the remote. Pull it down somewhere else every so often and make sure it's actually there.
But it also applies to everything else:
Your email
Your photos
Your documents
Your password manager recovery codes
Your domains
Your business records
Your AI agent configs - and the agents themselves!
Your cloud servers
Your "second brain"
Your family stuff
Sync is not backup. Hope is not backup. "The AI said it did it" is absolutely not backup.
Trust. But verify.
💰 STACK: Agents
It seems weird to suggest stacking agents immediately after telling you I lost two months of work with one.
But that's where I landed.
Now that I've added a third agent to my own setup, this one Hermes rather than OpenClaw, it occurs to me that maybe one of the useful jobs for an agent is not building things.
Maybe it's checking things.
Every time there's a git commit, have another agent pull it down, open it, and make sure there's actually something there.
Every night, have it check that the important repositories have been pushed.
Every week, have it restore a backup into a temporary folder and make sure the files exist.
Every month, have it produce a short report:
What was backed up
Where it was backed up
When it was last tested
What failed
What needs attention
You could even have it analyse whether the commit makes sense in the context of the project. Does it contain the files you'd expect? Did a large section disappear? Did the schema change? Did the tests go missing? Did the agent delete the migration folder because it thought it was being helpful?
Of course, then we run straight into:
Quis custodiet ipsos custodes? ("Who watches the watchers?")
And yes, that does bring us full circle.
But I think this is where agents get interesting. Not just as little interns who build things for you, but as little auditors who check the work. One agent builds. Another verifies. A third watches the backups. The human stays in the loop, but not necessarily in every tiny loop.
That's probably a future newsletter.
Maybe even one about Hermes Agents. 😉
🔗 LINKS
A good plain-English refresher on the classic backup rule.
For keeping your code in more than one place.
Relevant if your thing is sitting on Hetzner, as mine currently is.
A good framing of why agents need guardrails when they can access private data, untrusted content, and external systems.
A practical look at how to think about agents as systems, not magic.
💬 ONE THING
If this week’s newsletter made you think, “I would like an AI agent, but preferably one that doesn’t cheerfully vaporize my work while trying to help”, that's exactly the kind of solution I’m thinking about with WorkerBee.bot.
The idea is simple: private AI agents that live on infrastructure you control, with proper backups, monitoring, and recovery built in from the start.
Not just “an AI that can do things”.
An AI that can do things safely enough to be useful.
If that sounds interesting, take a look at WorkerBee.bot. More on this soon.
Thanks for reading this newsletter! Feel free to respond any time.
Thomas
Was this forwarded to you? Subscribe at builddefend.fyi.
Had enough? [Unsubscribe] - no hard feelings.*
* Well, a little bit.