One email a week - something from which I hope you'll get real value. We talk about things we can build, and how to defend them. That can apply to cybersecurity, physical buildings, digital products, and .... just about anything. It gives me a lot of latitude in what I can write about, but the two concepts are important for progress - as individuals, and as society.
At some point, keeping up with AI stopped being an interest and became a part-time job for people who already have jobs.
There is a new model. There is a new benchmark. Somebody has made an agent that can allegedly run a company, book a holiday, redesign a kitchen, and perhaps replace your dentist. A former intern has released a seventeen-part thread explaining why the thing you used last Tuesday is now prehistoric.
By Thursday, someone else has tested it and found that it cannot count the letters in “strawberry.” This is not a criticism of progress. The progress is real. It is also very noisy.
The wrong answer is to ignore it all. The other wrong answer is to turn your attention into an airport departure board for every model release, funding round, and graphic containing the phrase “reasoning tokens.”
The useful answer is to build an AI Field Guide: a small, deliberately personal system for noticing what matters, testing it against real work, and forgetting the rest. And, because the robots are now extremely good at reading a lot of material quickly, use AI to help you keep up with AI. Just do not hand it the steering wheel and take a nap.
🔨 BUILD: Your AI Field Guide
Begin with the Only Question that Matters
When an AI announcement appears, your first question is not, “Is this impressive?” Almost everything is impressive in a demo. After all, people were impressed with BetaMax, and look where that got us.
Ask this instead:
What could this change for me, the people I work with, or the life I am building?
If the honest answer is “nothing, unless I decide to become a YouTube influencer who makes reaction videos about coding agents,” you have permission to kinda ignore the thing.
Your Field Guide should be organised around your territory, not the AI industry’s preferred categories. Mine would include:
work and consulting
writing and research
software and small products
family learning
personal administration
local and private AI
creative work
things I am simply curious about, because not every useful interest needs a business case
That is enough. You are building a radar, not an intelligence agency.
Follow Sources with Different Incentives
Do not get all your AI news from the people trying to sell you AI news, and definitely do not get it entirely from the people trying to sell you AI itself.
Build a small source list with four jobs:
The builders tell you what they have released
The evaluators tell you how it performed under someone else’s test
The practitioners tell you what worked in real work
The sceptics tell you what the announcement forgot to mention
Each source has blind spots. Labs announce the good bits. Benchmarks can reward a narrow kind of cleverness. Practitioners tend to generalise from their own excellent use case. Critics can become professionally allergic to enthusiasm. That's okay. You are not looking for one oracle. You are looking for useful disagreement.
Let AI Do The Reading, Summarise, But Not Decide For You
This is where an AI assistant earns its keep. Once a week, give it a short collection of links: release notes, independent evaluations, practitioner write-ups, and perhaps one sceptical take. Then ask it to produce an evidence brief, not a motivational speech.
Use a prompt like this:
Read these sources. For each development, tell me: what actually changed; what is a demonstrated capability versus a marketing claim; which of my areas it may affect; what would need to be true for me to care; what privacy, cost, or reliability questions remain; and which original links support each conclusion. If the sources disagree, show the disagreement. Do not recommend a tool unless the evidence is specific.
That last sentence is doing a lot of heavy lifting. The model can save you from reading twenty pages of release notes. It cannot know whether a new model is useful in your particular work until you give it a job that matters to you.
Keep A Test Bench
Every development that survives the first reading gets one test.
Not a ten-hour weekend experiment (though those can be fun). One real task.
Can it summarise a long client document without inventing facts?
Can it turn a rough newsletter outline into a better first draft?
Can it reason through a piece of code you are actually maintaining?
Can it help your child practise a language without becoming an unnervingly cheerful substitute teacher? (Ugh, Skye)
Can it search your private notes without sending the family archives into somebody else’s training pipeline?
Record four things:
The claim
The task
What happened
What changes next
That is your Field Guide! Straightforward and to the point. In the future, the AIs will do this for you automatically, and you may not even want that, so if you get in the habit now, it may well make things better.
Review It, But Not Compulsively
The AI world moves quickly. Your opinion does not need to move at the same speed. Your work and effort doesn't either. That's a recipe for driving yourself crazy.
Once a month, ask three questions:
What became genuinely useful?
What looked exciting but did not survive contact with real work?
What can I now stop paying attention to?
The third question may be the most valuable. Attention is not a free resource, however much the internet behaves as though it is.
🛡️ DEFEND: Defend Your Attention and Judgement
Defend Against AI News About AI News
A great deal of AI coverage is now AI summarising an AI announcement, which was itself turned into a thread by someone who may or may not have read it. By the time it reaches you, it has the informational nutritional value of a Rice Krispie. Delicious when put together in a bowl with other Rice Krispies and some sugar and milk, but by itself, easily crushed and kinda disappointing.
For anything that might alter your work, money, privacy, or workflow, go one layer closer to the source. Read the release note. Look at the evaluation. Find somebody who tried the thing on work that resembles yours.
Use AI to condense primary material. Do not use it to turn recycled opinion into another layer of recycled opinion.
Defend Against Benchmark Theatre
Leaderboards are useful. They are not the league table of reality. That's money. Elon Musk is winning. Larry Oracle, er, Ellison, is doing pretty well too and he's like 800 years old now.
A model may be remarkable at coding, mediocre at writing, excellent at one language, expensive at scale, or charmingly useless at following the constraint that makes your task possible. The Arena leaderboards and independent evaluators are useful because they expose different dimensions, not because they crown an eternal winner. Strangely, there are lawyer benchmarks which get announced with everything, because it seems society wants to replace lawyers first and fastest. 🤷
Treat a benchmark as a clue. Then test the clue.
Defend Against Tool Accumulation
Every new AI tool promises a better future in exchange for $200 a month and access to a surprising amount of your life and your compute(r).
Do not build an AI stack outside of your agents. Build an AI shortlist.
A tool stays only if it does one of three things:
saves meaningful time
improves the quality of something you already do
makes a previously unreasonable task possible
“Looks incredible in a demo” is not one of the three. Unless your job is making demos. I'm not sure if any subscribe to this newsletter though!
Defend Against Outsourcing Your Taste
The danger is not only that AI gets things wrong. It is that it gets things blandly right, at a speed where you stop noticing that you are being recommended backsplashes that even the blind wouldn't tolerate. Use AI to gather, compare, challenge, plan, and draft. Keep the final judgment for yourself, especially where tone, trust, money, privacy, family, and reputation are involved. And backsplashes.
A good Field Guide should make you more capable of deciding. If it makes you ask the model what to think about every new model, you have accidentally hired a very confident hall monitor.
💰 STACK: The Smallest Useful AI Radar
You do not need a dashboard with twelve widgets blinking like the bridge of the Enterprise, though if you're a Star Trek geek, you might want that! If you're not, start here:
One note called “AI Field Guide.” Keep categories, links, tests, and decisions together.
Three to five sources. One builder source, one evaluator, one practitioner, and one sceptical voice. Add more only when you know what job they perform.
One weekly AI brief. Feed your chosen links to an AI assistant and demand source-linked claims, disagreement, and relevance to your work.
One real-world test per week. A task you would have done anyway.
A Keep / Test / Ignore list. Every tool and development belongs somewhere.
A monthly 30-minute review. Promote what works. Kill what does not. Remove sources that make you feel busy without making you better informed.
A private-data rule. Decide in advance what you will not paste into a cloud model, no matter how beautifully it promises to summarise it.
The system is intentionally mundane. Exciting systems otherwise become Notion templates that you revisit every six months with the same expression you reserve for a loft full of insulation.
🔗 LINKS
Arena Leaderboards
For crowdsourced comparisons across text, coding, reasoning, and other model categories. Use it as a clue, not a verdict.
Artificial Analysis MicroEvals
For comparing the same real-world prompts across multiple models.
Hugging Face Daily Papers
For a direct view of what researchers and builders are publishing.
Simon Willison’s AI writing
For rigorous, hands-on analysis of models, tools, and what they actually do.
Latent Space
For practitioner-focused conversations about AI engineering and product development.
The Model-Evaluation Guidance from NIST
For a useful reminder that testing, measurement, and risk assessment are not merely things to do after a demo goes wrong.
💬 ONE THING
This week, pick three AI developments you keep hearing about. Find one primary source and one independent source for each. Give all six links to the AI tool you already use and ask it for the evidence brief described above. Then choose just one thing to test on a real task. Not ten tools. Not a new “AI productivity” workspace. One useful test. That is how you keep up without letting the news cycle move into your house.
Thanks for reading this newsletter! Feel free to respond any time.
Thomas
Was this forwarded to you? Subscribe at builddefend.fyi.
Had enough? [Unsubscribe] - no hard feelings.*
* Well, a little bit.