AI By Michael

What Should Your AI Gateway Remember?

An insurer’s claims assistant reads an adjuster’s notes and drafts a request for missing documents. During a rollout, the claims tool starts returning a timeout. The platform team can see the failed call in its shared trace viewer. What it shouldn’t see there is the claimant’s medical note, copied into an exception by the client that handled the timeout.

That’s the distinction to settle before rolling out an AI gateway: the shared trace should show enough to locate and investigate a failure. Sensitive content should stay with the team that owns the workload, available through a separate, authorized investigation path when there’s a reason to see it.

Two November 9 announcements on X put different sides of this design in view. Langfuse presented a gateway for model calls and tracing. Appwrite presented MCP1 as a shared entry point for agent tools. These are launch posts from two vendors. Public X visibility is incomplete, and the posts tell us what the vendors are promoting, not how widely enterprises have adopted the pattern.

One run, two places to leak

Langfuse’s beta announcement says its gateway can record usage and cost without prompt or response content; full capture is opt-in. MCP1’s documentation describes a separate kind of gateway, one that checks and records tool calls. Those announcements describe different traffic. In a claims workflow, the same medical note can start as a tool result, become input to a model and show up again in an application error if the client copies part of the request into its diagnostic message.

That’s why switching off prompt logging at the model gateway is only one part of the work. The tool client, application logger and trace exporter can each emit a copy of the same data. Even a read-only tool can create a durable copy elsewhere through its instrumentation. If the platform team can search that copy, a successful access check at the claims system no longer tells you who can read the note.

Give platform operators a trail to follow

For this workflow, let the central trace carry an opaque run ID, the approved tool and model route, the operation’s status and duration, a bounded error code and usage where available. On a timeout, that should be enough to tell whether the model stalled or the claims lookup failed, and whether other calls on the same route are failing. Keep free-text exceptions, claim identifiers, prompts and tool payloads out of that shared record. A random run ID can link events without giving the platform operator a search key for a person’s file.

Define those fields where telemetry is emitted, then enforce the contract at export. OpenTelemetry’s guidance recommends minimizing collection and describes Collector processors for filtering and redaction. Use those as a second line of defense. An attribute allowlist alone won’t save a pipeline if an allowed message field or log body still carries raw text. Have the instrumented client emit a known error code instead of forwarding an upstream error verbatim.

The claims team should be able to follow the run ID inside its own restricted environment and review the original record under its normal access rules. If it needs the exact assembled prompt to diagnose a formatting bug, start with a synthetic reproduction. When that fails and real content is necessary, scope any separate capture to a named incident, a small set of requests and a defined expiry. Give it its own access and retention rules, including any copies made for debugging or evaluation.

What you give up by keeping traces lean

Full capture makes some failures easier to reproduce. With a slim trace, an operator can spot a bad deployment or a failing tool, but may need the claims team to work out why the model took a particular note as an instruction. That handoff adds time, and the exact input may be gone by the time the incident is reported. Set an escalation path and practice the handoff before the first incident. If only the claims team needs the original text to debug a case, keep that text out of the viewer everyone uses to monitor the rollout.

Test the timeout, then search the exports

Before rollout, put distinct synthetic markers in a claim note and an upstream error response. Run one successful request, force a tool timeout and retry, then exercise the denied-call path. Search the shared trace store, exported logs and any enabled evaluation sink for both markers. Pass only if neither marker appears and an operator with ordinary platform access can still identify the failing component and hand off the run to the claims team. Have an authorized claims reviewer verify that the run can be located there. If you can preserve privacy only by making the incident impossible to investigate, the trace contract still needs work.

Sources