I Built an AI That Remembers Everything. Including My API Keys.

Late in August I was digging through a backup of my own memory database, looking for something completely unrelated, and found a credential sitting there in plain text. Not in a config file. Not in an environment variable where I had carefully put it. In the store that my assistant reads from before it answers any question I ask it.

I had built that leak myself, months earlier, in a small pile of code I was quite pleased with at the time.

A memory layer is a database with a friendly name

Some context on the system, because the shape of it matters more than the specific tools.

I run four products off a 16GB M1 Pro laptop, with an older 8GB M1 in the corner as an always-on worker. The thing that makes that survivable is a memory layer: a local vector database, plus a folder of markdown files holding the longer notes. Every prompt I type gets the five most relevant memories injected into it automatically, before it reaches the model. That half works well, and it is the reason I can open a project I have not touched in six weeks and not have to re-explain it from scratch.

The other half is the write side. At the end of a session, a hook runs, pulls out the facts worth keeping, and saves them. Decisions, gotchas, the state of a build, the exact command that finally worked at 1am. I do not review those writes. Not reviewing them was the entire point of automating it.

Nobody ever said what a fact was allowed to contain

That is the root cause and it is embarrassingly boring. A hook persists facts automatically, and at no point did I, or anything else in the system, define what a fact could be.

Think about how a technical session actually ends. A debugging session ends with the request that finally returned a 200. A deploy session ends with the variable that was missing. A session spent wiring up somebody’s API ends with the line where the credential finally got read correctly. All of that is genuinely useful context, and all of it goes through the same extract-and-save path.

The extraction step was doing exactly what I asked it to do. Keep the thing that made it work. It has no opinion at all about whether the thing that made it work is a secret.

One hundred and eighty-one rows

Once I saw the first one I stopped everything else and swept the whole store. 181 database rows and 36 markdown files had secrets in them in plain text.

That number is what changed my mood about the whole thing. One leaked credential is an accident you fix in five minutes. 181 rows is a system that has been quietly doing this for as long as it has been running, across every project, at the end of every session, while I read the tidy summaries it produced and thought the automation was going great.

I am not going to show you any of them. Not a truncated one, not a prefix, nothing. I should probably have been running a scan for this on a schedule from the first week, and the reason I was not is that it never occurred to me that my own notes were a place secrets could live.

Filtering on read is a curtain over one window

My first instinct was wrong, and I want to be honest about how attractive it was.

The damage happens on retrieval, so put the redaction on retrieval. Everything goes through one search function, that function is easy to patch, and you can watch it work immediately. Ten minutes of work, nothing sensitive ever reaches a prompt again.

Here is the shape of it, written out generically rather than copied from my code:

// Illustrative, not my actual implementation.
function search(query) {
  const rows = db.query(query);
  return rows.map(redact);   // looks safe, is not
}

That holds right up until anything reads the data by another route. And there are always other routes. A debug script. sqlite3 on the command line. The sync job that copies markdown files into the database. A backup. A retrieval path I write next month and forget to send through the same helper. The secret is still in the row the entire time. All I did was put a curtain in front of one of the windows.

The only chokepoint that holds is insert()

So the redaction moved to the write path, inside insert(). Nothing gets into memory without passing through it.

// Illustrative, not my actual implementation.
function insert(row) {
  row.content = redact(row.content);
  return db.write(row);      // there is no other door
}

The property I wanted is that there is no way to add a row that skips redaction, short of somebody writing raw SQL against the file by hand. Every hook, every sync job, every manual save funnels into the same function. When a new write path shows up later, it inherits the behaviour without anyone having to remember to be careful, which matters a bit more than it sounds, because the person who has to remember is me at 1am.

Read-path filtering protects today’s readers. Write-path redaction makes the stored row boring, which is what I should have wanted from the start. I think the reason I reached for the read filter first is simply that it demos better. You patch one function, you run one query, you see clean output, you feel finished.

I had a clean database and a dirty copy of it one folder over

This is the part I nearly missed, and it is the part worth taking from this piece if you take one thing.

Before running the sweep I did the responsible thing and took a backup. Then I ran the sweep, watched the rows come back clean, checked the markdown files, and felt done.

The backup was taken before the sweep. Which means it held every original, unredacted, in full, in the same folder tree, protected by nothing, and completely invisible to the fix I had just shipped. For a while my actual state was a clean database, a dirty snapshot of the same database sitting next to it, and a strong feeling of having solved the problem. If I had stopped there I would have written a confident little post about write-path redaction and still been leaking.

Two more steps to actually be finished. Delete the pre-sweep backup properly, not into a trash folder that syncs somewhere. Then rotate the keys, because a credential that has been sitting in plain text on disk for months, inside files that get backed up and synced and occasionally read out to a model, is not a credential you get to keep using. Rotation is tedious and it is also the only step that makes the earlier exposure stop mattering.

The general form of this: any time you redact data in place, go and ask where else that data exists. Backups, exports, logs, the snapshot your editor keeps, the sync target, the copy you mailed yourself while debugging. Cleaning one location is a change in one location and nowhere else.

I do not know for certain that I caught every copy. I got the obvious ones. Sync history is a thing I have thought about and not resolved, which is a slightly uncomfortable sentence to publish.

What I would go and check in your system tonight

If you are shipping a RAG pipeline, an agent with memory, a chat product that saves transcripts, or basically anything that writes text into a store on the user’s behalf, there is a decent chance this bug is in there and nothing has told you.

Four questions worth an hour of your evening. Does anything write to your store without a human reading the write first. Are session transcripts or tool output a source of writes, because a secret lands in a transcript without anyone ever typing it into a field marked “secret”. Is your redaction implemented as a display concern or a retrieval concern rather than a storage one. And when you last cleaned the store, what happened to the copy you made before you cleaned it.

The last one gets people. It got me.

What still bothers me is that the memory system did not do anything wrong. It was told to keep what mattered from each session, and by any honest definition a working credential is the thing that mattered in that session. I have not come up with a rule for that which does not also throw away half of what makes the system useful. So for now there is a redactor with a list of patterns in it, doing its job on the way in, and it will hold until the day something arrives in a shape the list has never seen.