Build & Defend Newsletter 28

One email a week - something from which I hope you'll get real value. We talk about things we can build, and how to defend them. That can apply to cybersecurity, physical buildings, digital products, and .... just about anything. It gives me a lot of latitude in what I can write about, but the two concepts are important for progress - as individuals, and as society.

Today's topic is: giving everyone (every customer) their own AI.

A couple of weeks ago I said I'd talk more about the hardware choices you can make to keep your AI local. This week I actually had to make those choices myself, with real money, so this is that newsletter - written from the receipts rather than from my imagination.

Back in week 12 we talked about making your AI local, and the answer then was mostly "run a small model on the machine in front of you". That still works. But I've been building a SaaS - yes, during the "SaaSpocalypse" where the whole promise is that the users will have dedicated servers and dedicated AI. So .... yeah, there's that. (I'll announce the SaaS soon.)

But then .... I have to figure out what it actually COSTS to do such a thing.... I ran into the version of this problem that has a budget attached: if every customer needs their own AI, what does that actually cost, and what do you actually get?

The short version is that I was wrong about nearly everything until I measured it.

🔨 BUILD: Your Own AI, On Your Own Box

The obvious idea is to run the model on the same machine as the app. That died quickly. The app server has 16 GB of RAM and is already running the web app, a database, a job queue and a web server. A good open model wants most of that just to hold its weights. There are potentially some models that run in 16GB, but since it's not yet 2036, I had to be more realistic. So: two machines per customer. One runs the software. One runs the AI. They talk to each other and to nobody else. (That's part of the SaaS benefits - local-to-you-but-in-the-cloud-for-everything.)

Dense v. Sparse

My first instinct was to grab the biggest well-reviewed model that would fit in memory. There was a lovely new 30-billion-parameter model released last week that fits in about 18 GB. NICE. Except….

It would have been unusably slow, and here's the reason - which is the single most useful thing in this newsletter if you're doing this at home:

On a CPU (not a GPU), speed is determined by the number of parameters the model actually uses per word. Memory is determined by the total.

A "dense" model uses all of its parameters for every single word it produces. A 30B dense model at 4-bit has to pull about 17 GB through memory to generate one word. Your memory is the bottleneck, and that maths is brutal.

A "Mixture of Experts" model is different. It's much bigger overall, but only wakes up a small slice of itself for each word. The model I settled on (Gemma 4 26B-A4B) has 26 billion parameters in total but only uses about 4 billion at a time. It takes 15 GB of RAM to hold, and generates at the speed of a far smaller model.

Same memory budget. Roughly five times the speed. Simply because of how it's shaped.

(Fun fact: the Mixture of Experts idea isn't a 2020s invention at all - it dates back to a 1991 paper co-authored by Geoffrey Hinton, who has since become rather better known.)

What It Actually Does

I rented a machine - 8 CPUs, 32 GB of RAM, no graphics card at all - and measured it rather than guessing. Real numbers, on the box I'd actually buy:

Reading your question ~50 words/sec
Writing its answer ~16-21 words/sec
A full back-and-forth about 40 seconds
Reading a PDF 59 seconds

Not instant. But this thing digests a PDF, correctly, pulling all the relevant data - on a computer with no graphics card, that costs about €165 a month, sitting in a data centre that will only be reachable by a potential customer. Five years ago that sentence would have been science fiction. I'm not entirely sure it isn't now.

Four Times Faster?

Here's the one I want you to steal.

Modern models "think" before they answer - they generate a load of internal reasoning you never see, then write the actual reply. For a chat, that's genuinely useful. For pulling data from a PDF, it's completely wasted.

My first measurement: 262 seconds, of which about 80% was the model musing to itself about the list of data from the PDF. It was basically having a conversation in its little silicon head - an entirely unnecessary one at that.

I turned reasoning off for that one task. Result: 59 seconds instead of 262 - and byte-for-byte the same answer. Not "close enough". Identical. And 4 times faster.

Four times faster, for free, because I asked it to stop thinking about something that didn't need thinking about. If you're running anything locally and it feels slow, check whether it's reasoning at you when you only wanted it get information from a form.

🛡️ DEFEND: One User, One Machine (Well, Two Machines)

The original plan - written months ago, by me (well, AI) - was that every customer's server would connect to one shared AI machine over a private mesh network. It's the efficient design. One model, loaded once, serving everybody. The code had already been written.

Two problems:

The first is that the shared machine was the computer on my desk. Fine for testing. Absurd for actual customers.

The second: a shared AI machine means every customer's most sensitive data passes through one box. That is not a small compromise. That's the promise of my SaaS being quietly false in the one place nobody would look.

So I deleted it. Every potential user now gets their own machine, on their own private network, with exactly two members: their app machine and their AI machine. The AI has no public address at all. It isn't firewalled off from the internet - it simply isn't on it.

Beware the Efficient Design

The shared version was cheaper, simpler, and less to maintain. Every argument was in its favour except the one that mattered.

This is worth watching for in your own projects, because it rarely announces itself as a privacy decision. It arrives as an architecture decision, or a cost decision, and the privacy consequence is a side effect three steps down that nobody writes on the whiteboard. "We'll just use one server for that" is a sentence worth stopping on.

The Other Thing I Removed

The old design also leaned on a third-party mesh VPN. Lovely product, genuinely. But for a company selling data sovereignty, it meant another company's coordination service sat in the trust chain.

Both machines are with the same provider, in the same place. The provider already offers a private network between them - free, and one less organisation involved. So I used that instead.

Fewer moving parts is usually also fewer things that you have to administer and fewer things that could let you down.

"Up" is Not the Same as "Working"

Last week we talked about dashboards telling you flattering lies. Here's a fresh one.

The failure that matters on an AI machine isn't the server going down - you'd notice. It's the server being perfectly healthy while the AI silently isn't. Every light green, and the customer's AI just... stops answering.

So the health check asks the model itself whether it's alive, and reports which model is loaded. A machine running the wrong file looks completely healthy from the outside, which is exactly the kind of thing you only discover at the worst possible moment.

💰 STACK: Local AI, Boring Parts

llama.cpp - runs the model. Free, open source, runs on nearly anything. Pretend it's an OpenAI-compatible server and most software just works.

Gemma 4 26B-A4B - Google's open model. 26 billion parameters total, 4 billion active. Apache 2.0 licensed, so you can actually use it commercially.

Unsloth's quantisations - the compressed versions that make it fit in 15 GB. I'd originally picked a different model and had to drop it: excellent model, but nobody had published a trustworthy compressed version, and I wasn't about to point a potential customer-facing SaaS at something that has 1000 downloads by random researchers.

Hetzner CCX33 - 8 dedicated CPUs, 32 GB RAM, about €165/month, and available in Virginia, Oregon, Singapore and Germany. No graphics card. Turns out you don't need one!

systemd + a private network - the AI listens only on the private address. Not "blocked from the internet". Not on it. As mentioned.

The maths work out to be about 10% COGS. Hopefully people will eventually want it.

🔗 LINKS

llama.cpp
https://github.com/ggml-org/llama.cpp

Gemma 4 26B-A4B
https://huggingface.co/google/gemma-4-26B-A4B-it

Unsloth (quantised models & guides)
https://unsloth.ai

Hetzner Cloud
https://www.hetzner.com/cloud/

Geoffrey Hinton
https://en.wikipedia.org/wiki/Geoffrey_Hinton

Build & Defend Archive
newsletter.builddefend.fyi

💬 ONE THING

Measure it. I estimated the speed and I was wrong. I estimated the server prices and they were double what I'd assumed. I picked a model on reputation and had to drop it because the compressed version didn't exist.

Every one of those took an afternoon to check, and every one of them would have been a bad decision I'd have defended for months.

The arithmetic you do in your head is not a benchmark. Rent the machine for a day. €5 buys you a fact.