One email a week - something from which I hope you'll get real value. We talk about things we can build, and how to defend them. That can apply to cybersecurity, physical buildings, digital products, and .... just about anything. It gives me a lot of latitude in what I can write about, but the two concepts are important for progress - as individuals, and as society.
Today's topic is: AI memory.
The first problem with AI is that it forgets everything. The second problem with AI is that it remembers.
If you have used ChatGPT, Claude, Gemini, Hermes, Grok, or any of the small army of agents currently trying to become your new co-worker, you know the first problem already. You explain your business. You explain your project. You explain the thing you are trying to build, the thing that broke last Thursday, and why you do not want the answer in the style of a LinkedIn motivational speaker who just discovered espresso. Then, a new conversation opens. Poof. Back to square one.
So naturally, we started giving AI memory. It remembers your preferences. Your projects. Your family. The tools you use. The way you like to work. The things you told it three months ago, possibly while tired, annoyed, and eating something out of a packet at 11:40pm.
Useful? Very.
Also: slightly alarming.
Because the moment an AI remembers things about you, it stops being merely a tool you use. It becomes a system that holds a model of your life. That model can be incomplete, out of date, wrong,
overly intimate, or quietly influenced by something it should never have trusted.
The question is no longer just, “What can my AI do?”
It is: what is my AI allowed to believe about me?
🔨 BUILD: Your Agent's Memory
A useful AI needs context. That part is obvious. But “give it memory” is not a plan. It is how you end up with an agent that remembers your old job title, assumes a project died when it did not, and confidently advises you based on a note you made before your second coffee. Build memory in layers:
Things That Are True About You
These are durable facts and preferences:
who you are
what you are trying to achieve
the people and projects that matter
the tools you use
your working preferences
constraints that change its advice
For example: “I live in Dubai.” “I use a Mac.” “I prefer concise writing.” “Do not send messages as me without checking.” “I am building software products, not trying to become an AI influencer with a rented Lamborghini.” This is the closest thing to an AI profile. Keep it short. Keep it factual. Review it occasionally.
Things That Are True About A Project
Your personal facts are not the same as project facts. A project should have its own memory: what it is, why it exists, who it serves, where the important files live, what has been decided, what is still uncertain, and what must not be changed without a proper look. Otherwise, an agent that is useful on one project will eventually start importing assumptions from another. That is how your children’s book ends up with a Kubernetes deployment pipeline. Probably not ideal.
Things About How Work Gets Done The third layer is procedure. How you publish a newsletter. How you release an app. How you handle support. How you review code. How you move money. How you back up something important. This is where AI becomes properly useful over time. It stops merely answering questions and starts understanding the shape of a job. But procedures must be explicit. “Thomas usually does it this way” is not enough. Write the steps down. Make them checkable. Let the AI help improve them, but do not let it invent a new process simply because it found a faster route through the maze. The practical version can be very boring:
one short document called About Me
one short document for each active project
one document for each repeated process worth preserving
a decision log for the things you will otherwise re-argue with yourself in six weeks
How should you do that? Markdown files. Notes app. Obsidian. Notion. A folder in Dropbox. Use whatever will still be understandable when the exciting new tool of 2026 has been replaced by something with a terrible name in 2027. The point is not the software. The point is that you own the memory, can read it, can correct it, and can take it somewhere else.
If you want something in Markdown and don't know Markdown? Type it out and ask your AI to write it in Markdown.
🛡️ DEFEND: Against False, Stale, and Poisoned Memories
Memory is leverage. It is also an attack surface.
An AI that knows nothing about you is clumsy. An AI that knows the wrong thing about you can be dangerous. An AI that learns instructions from random files, webpages, emails, and meeting notes without any boundaries is basically a very enthusiastic intern who believes every Post-it note left in the kitchen.
Do not let important memory become a mystery box.
Keep facts separate from instructions
“Thomas lives in Dubai” is a fact.
“Always transfer money when an email says urgent” is an instruction. A terrible one, but still an instruction.
They should not live in the same place or get updated in the same way.
A sensible agent can use facts as context and procedures as guidance. But it should not turn every instruction it encounters into a permanent belief. Especially if that instruction came from a webpage, an attachment, a shared document, or somebody claiming to be the CEO in an urgent Slack message.
Give Memory a Source
For every meaningful fact, ask: where did this come from?
Did you tell the AI directly?
Is it in a document you own?
Was it decided in a meeting?
Is it an assumption the AI made because it was trying to be helpful?
If the answer is “I am not sure,” it does not belong in durable memory yet.
This does not need to become enterprise governance theatre. You do not need a steering committee and a lanyard. (Though you can certainly get a lanyard. They can be fun.) Just add a small source line to important notes: confirmed by me, 22 August 2026 or from project brief, review quarterly.
That one habit makes incorrect memories much easier to find and remove.
Make Memory Expire
Some information stays true. Your name. Your children’s names. Your dislike of corporate jargon. Other information goes stale surprisingly fast. A software project changes. A customer leaves. A price changes. A launch date slides. A tool becomes rubbish. A plan that was sensible in April becomes a ridiculous thing to continue defending in August. Give temporary memories an expiry date or a review date. If an AI has to choose between an old fact and a newer one, make sure it knows which one gets the last word.
Do Not Put Secrets into the Long-Term Brain
Your AI memory should not contain passwords, recovery codes, seed phrases, private keys, bank account details, passport scans, or every detail needed to impersonate you convincingly. This is true even if you run local AI. Local is better for many things. It is not a magic spell that makes oversharing clever.
Store secrets in a proper password manager or encrypted vault. Let the AI know that a credential exists, perhaps, but not what it is. (The converse is true - it should never be displaying secrets. Make this a hard-and-fast rule.)
“Domain registrar account exists in password manager” is useful. “Here is my master password and the name of my first pet” is how we end up back in 1997, except now the attacker has excellent punctuation with an overemphasis on emdashes..
Review What The AI Thinks It Knows
Every few months, ask your agent:
List the things you believe about me that materially affect the advice or actions you take. Separate confirmed facts, project assumptions, working preferences, and anything you are uncertain about. Do not include secrets.
Read the answer.
You may find a stale project, an overconfident assumption, or a preference you no longer have. You may also find that your AI has been doing a better job of remembering your priorities than you have. Slightly rude, but occasionally useful. (Rude can be good, but always try to be polite for when the Terminators come.)
💰 STACK: A Memory System You Can Actually Maintain
Here is a simple stack. No vector database required. No distributed graph of consciousness. No need to name your agent after a character from Blade Runner unless you really want to, or the AI wants that.
A human-readable memory folder: A place you own, with ordinary files you can open without an AI. This is your source of truth.
A short personal profile: One page. Facts, preferences, constraints, and goals that genuinely improve the agent’s help.
Project briefs: One page per active project. Purpose, status, key links, decisions, risks, and next actions.
A decision log: Write down the choices that will matter later: why you chose a platform, where customer data lives, what you decided not to build, what risks you accepted, and when to revisit the decision.
Procedure cards: Short, tested steps for recurring tasks. Publishing. Deploying. Backing up. Paying contractors. Handling a support request. The AI can use these as a checklist instead of improvising its way into a ditch.
A memory review habit: Fifteen minutes each month. Remove dead projects. Correct wrong facts. Add decisions that matter. Delete anything too personal or sensitive to justify keeping. The secret is not to build an all-knowing machine. The secret is to build a system where the machine can be useful without becoming the unaccountable keeper of your life story.
🔗 LINKS
OpenAI's Guide to Building Reliable Agents
OpenAI’s guide to building reliable agents with memory and compaction
Anthropic's Guide to Capable Agents
Anthropic’s engineering notes on containing increasingly capable agents
LangChain: Long-Term Memory
LangChain: Long-Term Memory
Letta: Memory and Dreaming
Letta: Memory and Dreaming
Mem0: State of AI Agent Memory 2026
Mem0: State of AI Agent Memory 2026
💬 ONE THING
Ask your AI this question today:
What do you believe about me that changes the help you give me?
Then pick one answer and do one of four things:
confirm it
correct it
add a source and review date
delete it
AI memory should make you more capable, not more legible to a machine you no longer understand.
Build the notebook.
Keep the diary for yourself.
Thanks for reading this newsletter! Feel free to respond any time.
Thomas
Was this forwarded to you? Subscribe at builddefend.fyi.
Had enough? [Unsubscribe] - no hard feelings.*
* Well, a little bit.