r/PhalaNetwork • u/Upstairs-Win-8843 • 13d ago
What does a TEE actually protect when you send a prompt to an AI service?
Suppose you send a private contract to an AI service. The connection uses HTTPS, and the operator says it does not keep your data. Once the request reaches the server, though, some program has to read the contract. Who can inspect that program's memory? What stops an update from adding a log of every prompt?
Those are useful questions for confidential computing. The answer depends on where the model runs, which parts of the system the TEE protects, and what the client verifies before sending anything.
I maintain an open-source AI gateway. The forwarding example below comes from that work, but the same trust-boundary questions apply to other services.
Consider two deployments. In the first, the model and inference process run inside a protected VM. With Intel TDX, the VM's private memory and processor state are isolated from direct host access under the hardware's security model. The guest OS and application remain trusted components. A program inside the VM can still leak data deliberately or through a bug. Intel's introduction explains this boundary.
For a CPU inference example, that boundary can include the model computation. If the service sends tensors to an ordinary GPU, or sends documents to an external retrieval database, those paths need their own protection. The diagram needs to follow the data beyond the box labelled “TEE.”
In the second deployment, the protected VM runs an AI gateway. It receives the prompt, calls an external model provider, and returns the answer. Here the TEE can reduce exposure to the gateway's hosting infrastructure. The model provider still receives the content required for inference. Its retention policy and the identity of its model remain separate trust questions.
Before sending the contract, the client needs evidence about the endpoint. One approach is to bind a fresh client challenge and the current TLS certificate into the hardware attestation. The client checks that evidence against independently approved software measurements and a platform security policy, then sends the request over the verified connection.
The connection binding matters: a valid quote from one machine is little help if the prompt goes to another. The reference measurements matter too. Accepting whatever measurements the server supplies would let the server choose its own passing grade. Open source gives people code to review; the build and measurement chain must still connect that review to the running service.
Now consider a different dispute: the contract stayed private, but the bill looks wrong.
Suppose a provider reports 10,000 input tokens, including 2,000 cached tokens, and 1,000 output tokens. Use illustrative rates of $2 per million ordinary input tokens, $0.20 per million cached tokens, and $8 per million output tokens. The charge is:
8,000 × $2 / 1,000,000
+ 2,000 × $0.20 / 1,000,000
+ 1,000 × $8 / 1,000,000
= $0.0244An implementation that charges all 10,000 input tokens at the ordinary rate and then adds the cached-token charge would produce $0.0284. A TEE can faithfully run either calculation. Checking the arithmetic requires the counters, their meanings, the applicable price snapshot, and the billing rules.
This is where the public-verification idea associated with blockchains becomes useful. A user should be able to inspect the evidence behind an accounting result instead of relying entirely on a balance displayed by the operator.
In the implementation I work on, the gateway commits usage and billing records to signed batches on a five-minute publication cycle. Each batch identifies the previous batch hash. The signing key has its own attestation binding, so a verifier can connect a receipt to an approved runtime as well as check that its bytes have not changed. Checking a batch's signature, reproducing each charge, and verifying the runtime identity are separate checks.
The public records contain accounting metadata. They exclude prompts, answers, and authentication credentials. User pseudonyms change between batches, although timing and usage patterns can still link activity. Publishing auditable bills requires a deliberate decision about what becomes public.
There is an important gap in this design. A valid signature establishes what a key committed to; it does not establish that every request was recorded. A chain of hashes can expose a broken link in a presented history, but an observer may still receive an old prefix or one of several divergent histories. A single attested writer provides no consensus about the latest state. These cryptographic building blocks also predate blockchains.
A crash creates another boundary. A local journal can preserve events already committed to it, and a failed upload can be retried with the same signed bytes. Neither mechanism recovers an event that was never committed, nor does local storage alone prove that the entire stored history has not been rolled back.
The practical result is a more specific set of questions to ask of an AI service: which endpoint received the prompt, which software was approved, what evidence supports the charge, and which parts of the story still depend on an external provider or operator?
The concrete gateway example is TeeRouter, which I maintain and run on dstack/TDX. Its receipt format and verification limits are public. This article was prepared with AI assistance.
For others building TEE services, how do you handle the history problem? In particular, what do independent observers retain or compare so that a valid but stale history is detectable?


