The logs were beautiful.

Timestamped. Structured. Every tool call, every argument, every response, every latency number in milliseconds. I had been careful about this. I had built this part before.

Then I went looking for one user’s history.

Every request in the file came from the same small set of IP addresses. Not one of them belonged to a person. They belonged to OpenAI.

My backend had never been talking to a user. It had been talking to a platform that was talking to a user. And I had been logging that conversation faithfully, in detail, for weeks.

The thing I already knew

A while back I worked on a healthcare platform heading toward regulatory clearance. Serverless Python, an AI component, patient data, and a compliance regime that did not care about my opinions.

I learned one thing there I have not been able to unlearn.

“It worked in testing” is not an answer.

Sooner or later somebody asks you to prove what the system did. For a named person. On a named day. If you cannot produce that, whether it worked stops being the question.

So when I started building an AI travel and lifestyle agent for an editorial client — much lower stakes, nobody’s health involved — I logged from day one. I thought I was being disciplined.

The discipline did not save me. The failure was not carelessness. It was architectural, and it was completely silent.

Why this trap is invisible

In a normal web application, the request is the user. Headers, cookies, session, IP — all of it describes the person who caused it. Every instinct you build over years of backend work rests on that.

A Custom GPT calling your API through Actions breaks it.

The platform receives the user’s message. The platform decides to call your endpoint. The platform makes that call, from its own infrastructure, on its own schedule. What arrives at your door is a machine that was sent on an errand.

Nothing errors. Nothing warns you. You get well-formed requests at plausible times carrying sensible arguments. The logs look correct because they are correct. They are an accurate, detailed record of one company’s servers talking to mine.

A log of events with no subject is not an audit trail. It is noise with timestamps.

Per-user OAuth is what fixed it — the identity has to be established at the authorisation layer and carried through the call, because it is never going to arrive by accident. Worth knowing: Custom GPT Actions support per-user OAuth without an OpenAI review process for private or link-shared GPTs. There was no platform limitation here. Only my assumption.

The easy half and the hard half

Logging what happened is the easy half. Tool called, arguments in, response out, duration. Anyone can produce that, and most systems do.

The hard half is why.

An itinerary is not an output. It is a chain of choices. Why this restaurant and not the other four in the same neighbourhood. Why this order. Why this one dropped at the last step. Reconstructing that afterwards from a stored result is not difficult — it is impossible. The information was never written down.

So we write it down. Three things, specifically:

The plan itself, kept and graded. Every generated day plan is stored and reviewed by the client’s editorial team, who grade its quality. Not automated scoring — actual editors reading actual plans and saying whether they would send this to a reader.

The selection reasoning. Each place in a plan carries the reason it was chosen. Not a summary of the plan. The per-item rationale.

The gaps. When the agent had nothing — no coverage for a neighbourhood, no options matching a constraint — that gets logged too. Most systems only record what happened. Recording what could not happen turned out to be the more useful half, because it maps directly onto where our content is thin rather than where our code is broken.

Three reasons I do not fully trust my own trace

This is the part I would want to read, so it is the part I will write.

The reasoning is a self-report, not a causal trace. When the model explains why it picked a place, that explanation is generated text. It is produced after the fact, by the same system, and there is no guarantee it reflects the actual mechanism of selection. It is closer to an interview than an instrument. Useful, plausible, and not evidence. Anyone claiming their agent logs “why” should be asked which of those two they built.

Human grading does not scale and does not agree with itself. A small editorial team is a good signal and a bad measurement. There is no inter-rater agreement figure. There is no rubric that survives contact with a genuinely ambiguous plan. It tells us something real. It does not tell us something stable.

Gap logs only catch loud failures. They record the times the agent knew it had nothing. They do not record the times it had almost nothing and produced a confident, well-formatted plan anyway. That second category is larger and considerably more dangerous, and by construction it does not appear in the gap log at all.

The cost that does not show up in the latency graph

Transparency costs latency and storage. Everybody knows this and it is mostly a solved trade-off.

The cost people skip is this one: a complete trace of a regulated workflow is itself regulated data.

Log everything, and you have built a second system holding the sensitive material of the first, usually with worse access controls, longer retention, and none of the review that the primary datastore went through. The instinct to log everything for compliance creates a compliance problem wearing compliance’s clothes.

Which means the interesting design question is not what to log. It is what to deliberately not log, and being able to say why.

What I measure, and what it cannot tell me

The success criteria for this phase are product criteria. Do people come back unprompted. What proportion of plans would the client send to a reader with no edits. Would anyone pay for it.

Those are the right metrics for whether the product works.

None of them tell me why the agent chose what it chose.

A product can clear every one of those numbers and remain completely unauditable. Those are independent properties, and conflating them is how you end up with a system that is performing well and cannot be explained — which is a fine position until the first time someone asks you to explain it.

Where this leaves me

I do not have audit stories. Nothing has launched. No regulator has asked me for anything.

That is the entire point.

The time to find out your trace cannot identify anyone is while you still have the time to fix it. Not when somebody is standing over your shoulder asking what the system did for a specific person on a specific day.

My logs were perfect for six weeks. They were also useless, and nothing in them said so.