Nobody Applauds a Restore That Works

I told you last episode that the homelab made my career. And I was deliberately vague about which part of it actually did that, which in hindsight was a bit of a cop-out. So let me be specific now, because the answer is the least glamorous thing in the entire rack. Backups. I work in backups. That is probably one of the most boring ways to open a blog post, and I’m going to spend the next fifteen hundred words on it anyway, because it’s the one corner of this hobby where being wrong actually costs you something.

Stick with me on this one.

What Actually Runs in the Lab

I’d rather show you a real thing than talk about principles in the abstract, so here’s what’s actually running. I use Veeam Backup & Replication in my homelab, protecting the Proxmox workloads I migrated to after stepping away from VMware. That means the VMs in the cluster, the domain controllers, a handful of services I run for personal use. All of it covered. I’ve also got a Synology NAS that holds family photos, documents, the sort of irreplaceable stuff that has nothing to do with IT and everything to do with why any of this matters in the first place. Veeam’s protecting that too, writing to a secondary NAS I keep on the network.

And then there’s the offsite copy. I run Veeam Data Cloud Vault as well, and it’s encrypted, immutable, genuinely offsite, so I’ve got something that survives a local hardware failure, a power event, or anything else that could take out the primary copy. My NAS runs RAID 6, so I’ve got some redundancy there, but RAID is not a backup and I think we all know that. A power failure that blows the NAS takes out RAID 6 just as efficiently as it takes out anything else. The offsite copy solves for that.

Now, I know what you’re thinking: that sounds like a lot of overhead for a homelab. And yeah, honestly, it probably is. But this is the practice-what-you-preach thing, and it’s something I come back to constantly. I talk to customers and colleagues about this every single day. Multiple copies, multiple locations, immutable storage. And I’d feel like an absolute hypocrite if I came home and did none of it. My family photos are important enough to treat seriously. The habits are worth reinforcing. And, candidly, if I’m going to recommend this to people, I should be able to say I actually do it rather than just say I do.

The Mechanic’s Car

Here’s the uncomfortable question: do you actually test your homelab backups, or do you just have backups?

Because there’s a version of this where you’ve got a perfectly configured backup job, the jobs complete green every night, and you have absolutely no idea whether you can recover anything from them. And the people most likely to fall into that trap are the people who know the most, the ones who spend all day in this space professionally and then come home and take the shortcut, because nobody is auditing them.

That’s the mechanic’s car problem. A mechanic’s own car is usually the rust bucket in the street, because they’ve just spent eight hours working on other people’s vehicles and the last thing they want to do is come home and change their own oil. I get that. Completely understandable. But in IT, and specifically in backups, that habit will eventually cost you.

So I actually do test. Not as often as I’d like, and definitely not as rigorously as I’d run a recovery test in a production environment, but I go in periodically and restore something. A single photo. A VM. Something that forces me to actually walk through the process rather than just assume it works. And that’s the whole point. The assumption is where people get burned.

I’ve been burned by this myself, which I’ll admit freely. I had a snapshot of a lab workload, ran some scripts, played around, absolutely corrupted it, tried to roll back and the snapshot restore just didn’t work. I’m still not entirely sure why. So I fell back to the backup, which at that point was a couple of days old and wasn’t fully configured yet. I got most of it back, but not all, and I spent an afternoon redoing work I’d already done. That stings a bit when you’re the person who spends their working day telling others to test their backups. The mechanic’s car, right there, in my own driveway.

Snapshots are not backups, and I want to hammer that point because it comes up constantly, even in production environments. A snapshot lives inside the same cluster, on the same infrastructure, potentially on the same physical hardware. If that hardware fails, you don’t have a copy of anything. You have a very elegant log of changes that no longer exists. Confusing the two will eventually burn you, and I’ve seen it happen in real environments, not just my own lab.

Why Verification Is No Longer Optional

This is where I want to make a slight pivot, because there’s a bigger context here that I think is easy to miss if you’re only thinking about backups as a “data protection” problem rather than a security problem.

Everything is accelerating. I talked a bit in an earlier episode about AI and how the pace of change is genuinely dizzying right now, and the threat landscape is no different. Threat actors are better resourced and faster than they’ve ever been, and the AI tools that are helping us build and test and automate things are helping them too. There was some interesting work that came out of Anthropic’s research, their model surfacing zero-day vulnerabilities in well-maintained open source projects, and I think that should give everyone pause. If that’s the capability on the defensive side, assume the offensive side isn’t far behind.

And here’s the thing that should make you uncomfortable: your backups are the target now. Not the afterthought, the target. The whole point of a ransomware attack in 2025 is to find the backups, corrupt them or encrypt them or exfiltrate them, and then present you with a ransom note. If your backups are gone, you have no last line of defence. So the immutability matters. The encryption matters. But neither of those things matters at all if you’ve never verified you can actually restore from them.

So when I talk about verification, I mean the full picture, not just “did the backup job succeed?” but understanding boot order, boot dependencies, which workloads have to come back first before the others function, what the tiers of your environment look like and how they map to a recovery sequence. That’s the discipline. And the homelab is the place to build it, because getting it wrong there costs you nothing except an afternoon.

Running It Like It Matters

Least privilege on the backup account, that’s another one. When you’re setting up a homelab you can just hand everything domain admin rights and move on, and I understand the temptation because it’s faster and who’s going to know. But if you practise least privilege in the lab, you’ll be far more confident applying it in production, because you’ll have already worked through why it matters and what the minimum required permissions actually look like. You’ll have already made the mistakes and fixed them where it’s safe to do so.

The homelab is the one place where being wrong is free.

That framing matters to me, because the goal isn’t to run the homelab exactly like a production environment. That would be exhausting and unnecessary. The goal is to practise the discipline, to keep that rhythm going, to maintain the habits, so that when you’re in a real situation, you’ve already rehearsed it. You’ve already sat down and thought through the recovery, you’ve already felt the slight panic of a restore that takes longer than expected, you’ve already confirmed that your offsite copy is actually intact and not just a successful backup job that was writing corrupted data.

Because the alternative is being on the receiving end of a ransomware attack and reading the recovery documentation for the first time while a ransom note is on the screen. Nobody wants that. And the homelab is specifically the place to make sure it doesn’t happen, not because every homelab workload is critical, but because the knowledge is transferable and the habits are real.

There’s a reason I keep coming back to this when people ask me what the homelab has actually given me. Not the hardware, not the certifications, not even the specific product knowledge, though all of that matters. The thing it’s genuinely given me is the confidence that comes from having done the thing, not just knowing how to do it. There’s a difference, and you can feel it when you’re standing in front of a customer or a leadership team and someone’s asking whether you can recover and you know, actually know, what the answer is.

Nobody is going to applaud a restore that works. Nobody has ever called someone into a boardroom to celebrate the backup that came back clean at three in the morning and saved the business. That’s just not how any of this works. But nothing else you do matters if the restore doesn’t work. All of it, every other clever thing in the rack, every configuration you’ve optimised, every service you’re running, it’s all conditional on that last line of defence holding.

Make sure it holds. Test it. Verify it. Do it in the lab where it’s safe, so you’re not doing it for the first time when it isn’t.

As always, keep on learning.