Practising What We Preach, with Raghu

Episode nine, and I opened it by introducing Raghu as my second guest, which he immediately corrected to third. Great start. Good to know at least one person is paying closer attention to the show than I am. We recorded this one in Melbourne over a beer, which is a first for the podcast and something I’d like to make a habit of, so if it sounds a bit more relaxed than usual then that’s why.

Raghu is a mate and a colleague, the three of us all work together, so this is the second episode in a row where I’ve sat down with someone I’ve known for years and gone digging into their lab. Ben’s episode was a Pure Storage array in a garage and a power bill to match. Raghu’s lab is a handful of mini PCs he bought off eBay and breaks on a schedule, and he was very quick to say up front that his setup is nowhere near as enterprise-y as Ben’s. Neither is mine. I don’t think anyone’s is.

From “What’s a Router?” to Production Firewalls

Raghu’s origin story is one of my favourites and he tells it against himself without any hesitation. Fourteen years ago he came over to Australia to do his master’s, walked into a networking course, and asked his professor what a router actually was. The professor’s response was basically, you signed up for a networking course and you’re asking me what a router is. Great start, as he put it.

From there to configuring firewalls in production, and he ties the whole journey between those two points to how much he’s broken along the way. That’s genuinely the thesis of his approach to all of this, the privilege of being able to break anything without getting judged for it, and he came back to that phrase more than once while we were recording.

Mine started in a similar place, an old gaming PC sitting off to the side with a graphics card in it and specs I honestly can’t remember other than that they were rubbish. That’s where the whole thing started for me before it grew into networking gear and NUCs. Raghu went from old laptops to a NUC that was loaned to him, which he then melted. His words, not mine, and his own explanation for why he’s so reluctant to touch anything in production is that he’s capable of breaking things that aren’t supposed to be breakable.

After the NUC it turned into an addiction, and I mean that in the way every one of us listening will recognise. He picked up a few mini PCs off eBay, three of them to start with, six or eight cores each and sixteen gig of RAM, and then every time second hand memory showed up cheap on eBay or Amazon he’d just buy it and add it in. Without really noticing, he ended up with somewhere around 24 to 30 physical cores and close to 64 to 70 gig of RAM across the lot, running a full stack. It grew organically as his requirements grew, which is exactly how these things go, nobody sits down and designs it.

The gear is small enough that it doesn’t really impose on the house either, other than what he described as his poor handyman skills mounting them to the wall and poking a few holes in it. They’re not power hungry the way enterprise kit is, so there’s no equivalent of Ben’s monthly bill to justify to anyone.

He had a good line on justifying it anyway. There’s a cricketer he follows who says that every time you get picked in a league and get paid, you put ten percent of it back into your training so you keep growing your own worth. Investing in yourself, basically, which is the same argument Ben made about his power bill being a training budget rather than an expense. Two completely different scales of lab, same reasoning underneath.

The Cert Didn’t Teach Him That

Raghu and I have both sat through our CCSPs, which means we’ve both waded through the ISC2 acronym soup, and I wanted to know how much of the lab actually fed into that.

Not much, as it turns out, and that surprised me less the more he explained it. The CCSP is about the theory of running a data centre, everything from the building itself up through cooling and governance, so there’s not a lot in there you can go and virtualise on a mini PC. What the lab gives him instead is the technology rather than the product. If he wants to understand networking or firewalls he can drop in OPNsense or pfSense, virtualise the scenario, and see what a real environment actually looks like when it’s misbehaving.

The point of that isn’t the knowledge, it’s the empathy. When an operations person comes to him and says this is a problem for me, he can either pretend to understand it or he can say yes, I’ve done that, it’s annoying. You can hear which one of those lands in a room. He was pretty honest that fourteen years in the industry isn’t long enough to have seen the ebbs and flows of it, so the lab is how he fast forwards that, going and living through a problem before he opens his mouth with advice about it.

My own takeaway from the cert was a bit different. I came at it as a technical person who’d done firewalls and networking, badly enough that I’ve broken more of them than I’ve fixed, and the part that was genuinely new was the governance and compliance layer sitting over the top of all of it. Being able to look at a firewall rule and understand what it means to a governance person, and then translate that back to the techies, was the thing I actually got out of it.

We also worked out on air that we learn the same way, which explains a fair bit about both our labs. Some people are happy to dive into the middle of something and understand it at a high level and move on. Neither of us can do that. We want to start at the foundations, build the thing, break the thing, and understand what makes it work by finding out what makes it stop working. Hand me a textbook and nothing happens. Let me go and wreck it and it sticks.

The Vacuum Cleaner Incident

So I asked him whether there’s anything he deliberately relaxes in the lab from a security point of view, because I’ve certainly got form there.

Mine was ad blocking. I rolled out a blanket block across the entire network, not just the lab, which promptly broke a handful of apps on my wife’s phone. Paramount stopped working, the Woolworths app stopped working for reasons I still don’t fully understand, and at that point it stopped being a lab decision and started being a household one. I relaxed it. My home network also sat as a flat layer two for a long time simply because I couldn’t be bothered setting it up properly, which I’ve now done, but only because I eventually needed to.

Raghu’s version of that story is better than mine because his consequence wasn’t an inconvenience, it was the vacuum cleaner. Back when his lab was still flat, he hardcoded an IP address on a VM he’d spun up and it turned out that address already belonged to the robot vacuum. He didn’t know there was a problem until there was a problem, which arrived in the form of his wife asking whether he’d done something to the network, because the vacuum was saying it couldn’t connect. Oh. Probably, yeah.

He put it better than I could, that’s worse than breaking production. It’s pretty impressive how quickly a segmentation strategy crystallises once a vacuum cleaner is involved.

These days he runs a pfSense box between his home router and the lab so the blast radius stays inside the lab, and that’s the bit that carries straight across from work without him having to think about it. Everything else inside that boundary is fair game. He rebuilds half his lab every fortnight, sometimes weekly, because someone will mention a technology, he’ll think that’s so cool, and off he goes to break it. He’s moved vCenters around while learning networking on the fly and locked himself out entirely, to the point where the only way back in was booting off a live ISO.

Then there’s the Terraform one, which is the story I’ll be thinking about for a while. He went to have a crack at infrastructure as code so he could rebuild faster, wrote out a state file describing what he wanted, and added a VM to it. Terraform, doing exactly what Terraform is designed to do, took that as the desired end state and wiped everything that wasn’t in it. One VM left standing. His reaction was basically, okay, now I’ve learned something. I suggested automating his rebuilds and he reckons he’d break that too, and on the evidence I think he might be right.

Practising What We Preach

The line from this episode I keep coming back to is that you haven’t learned something until you’ve broken it and then had to fix it, and fix it, not rebuild it. That distinction matters. Blowing it away and starting again teaches you the install process. Actually repairing the thing teaches you how it works.

The other half of that is being able to do it without duress, and there’s no better place to practise that than somewhere that doesn’t cost you a client or an SLA or your job. Practising in production is not for everyone. Some people thrive under that kind of pressure and he was very clear he isn’t one of them, he’d rather be a response guy than a react guy, and the only way you get there is by breaking your own gear often enough that breaking things stops being frightening.

It shows up in the work in ways that aren’t really about technology at all. When we do presentations on ransomware, the most powerful thing either of us can say isn’t a technical explanation, it’s that we’ve been through it ourselves. People sit up at that. What was it like? It was awful, and that’s exactly why I’m standing here telling you about it in the hope that you don’t have to find out. You don’t get to say that convincingly unless somewhere along the line you actually broke something badly enough to mean it. Or as Raghu framed it, save ten minutes for ten people and you’ve handed back an hour you never had to spend.

He didn’t need a homelab to be good at his job, and I don’t think he’d claim otherwise. He built one because knowing a thing and having somewhere to go and do it badly, repeatedly, with nobody watching, are two very different things. That’s the whole reason my cupboard exists too.

That’s why we practise what we preach. Do it in the test lab, that’s what it’s there for.

Thanks for listening, and as always, keep on learning.