Privacy in LLM applications

An LLM feature is not one processing activity. It is a chain of copies: the prompt, the retrieved context, the tool results, the completion, the trace, the eval set, the provider's abuse log. Each copy has its own location, owner and lifespan, and each is in scope for access, deletion, retention and transfer rules.

This guide walks the chain from the user's keyboard to the model provider and back, names the privacy obligation that attaches at each step, and ends with a checklist you can hand to the team that owns the feature. It is written for privacy leads and the engineers they work with. The detail lives in five companion articles:

• Personal data in LLM prompts and traces

  Detection, redaction, pseudonymization and trace retention, at engineering level.

• Access and deletion requests against LLM data

  Logs, vector stores, eval sets, caches and the hard case of model weights.

• Model providers as processors

  What OpenAI, Anthropic, Google, Microsoft, AWS and Mistral say about retention, training and regions.

• Self-hosting the LLM stack for data residency

  Which layers you can bring in-house and what each one actually buys you.

• Where the EU AI Act and GDPR overlap

  Roles, timelines after the 2026 omnibus, FRIA vs DPIA, Article 22.

Nothing here is legal advice. The mechanics are general; how they apply to your feature depends on your purposes, your contracts and the jurisdictions you serve.

The data flow, stage by stage

Take a typical retrieval-augmented support assistant: a customer types a question, the app pulls the customer's account record and some help-center articles, calls a hosted model, maybe calls a tool (look up an order, open a ticket), and returns an answer. Along the way, an observability SDK records the whole exchange. Here is where personal data lands.

Two things stand out. First, the same sentence a user typed can exist in six or seven places within a second of being sent. Second, the longest-lived copies are usually not the ones the product team thinks about: traces, eval sets and fine-tuning files outlive chat history because nobody set a retention rule for them.

If you already maintain a data map, the LLM feature needs entries for each of these stores, not one line for "AI assistant". The data mapping guide covers how to structure those records; the identifier and request-handling fields matter most here.

Which obligations attach where

The GDPR does not have an AI chapter. The ordinary rules apply to each copy. The useful exercise is to go stage by stage and ask the six questions below.

Lawful basis and purpose

Answering a user's question with their own data usually rests on contract necessity or legitimate interest, the same basis as the underlying service. The harder cases are secondary uses: copying production conversations into eval sets, using them to fine-tune, or letting the provider train on them. Each is a new purpose that needs its own basis and has to be compatible with what users were told under Articles 13 and 14 of the GDPR. The EDPB's Opinion 28/2024 on AI models treats development and deployment as separate phases that each need their own legitimate-interest assessment if that is the basis relied on.

Data minimization

Every field injected into a system prompt is a disclosure to the model provider and a copy in your traces. Minimization in an LLM app is mostly prompt engineering discipline: send the order status, not the whole customer record; retrieve the three relevant chunks, not the whole ticket history. It is also logging discipline. The prompts and traces article covers redaction before data leaves your process.

Retention

Article 5(1)(e) storage limitation applies to traces and eval sets as much as to the CRM. Set an explicit time-to-live per store, and check that the provider's retention for the same content is compatible with what your privacy notice says. Provider retention defaults and zero-retention options are compared in the vendor article.

Access, deletion and correction requests

A data subject access request covers chat history, traces, eval rows and anything else that can be linked to the person. Deletion covers the same, plus the harder questions of vector stores and models. Chatbots also raise rectification: if the model states something false about a person, that person may ask for it to be corrected. The practical approach is in the DSAR article; your general request workflow is covered in the DSR automation guide.

Processors, sub-processors and transfers

A hosted model provider acting on your instructions is a processor and needs an Article 28 contract; the DPA guide lists what that contract must contain. The observability vendor, vector database host, eval platform and gateway vendor are processors too, each with their own sub-processor lists. If any of them process data outside the EEA, you need a transfer mechanism under Chapter V of the GDPR: an adequacy decision (the EU-US Data Privacy Framework for certified US companies), standard contractual clauses, or another Article 46 tool. The General Court upheld the Data Privacy Framework on September 3, 2025 (Latombe v Commission, T-553/23); an appeal to the Court of Justice (C-703/25 P, lodged October 31, 2025) was still pending as of October 2026, so keep SCCs in the contract as a fallback.

Watch for the case where the provider acts as a controller for part of the processing. Several providers reserve the right to use content for abuse monitoring or, on some plans, model training. Mistral's DPA (effective July 27, 2026), for example, lists abuse monitoring and, unless the customer has opted out, model training as purposes for which Mistral acts as controller (Mistral DPA, section 2.3). That is a disclosure you have to account for in your notice and your records.

Impact assessments

Article 35 requires a DPIA where processing is likely to result in a high risk. An LLM feature that processes customer conversations at scale, uses a new technology, and may handle special-category data that users volunteer will meet more than one of the screening criteria most supervisory authorities use. Even where a DPIA is not strictly required, writing one forces the stage-by-stage inventory above. If the feature is also a high-risk AI system under the EU AI Act, the deployer may owe a fundamental rights impact assessment as well; how the two fit together is covered in the AI Act article.

Where teams usually get caught out

• Tracing captured everything by default. Many LLM observability SDKs record full inputs and outputs unless told otherwise. The OpenTelemetry GenAI conventions mark message content as opt-in precisely because it "is likely to contain sensitive information including user/PII data" (OpenTelemetry GenAI semantic conventions), but vendor SDKs set their own defaults.
• Eval sets built from production. A developer copies fifty real conversations into a test suite to reproduce a bug. Two years later they are in a git repository with no retention rule and no way to find them during a deletion request.
• Stateful provider features. Zero data retention agreements typically do not cover stateful endpoints. OpenAI's documentation, for instance, lists (as of October 2026) Files, Vector stores, Assistants, Threads, Batches, Fine-tuning and Evals as not eligible for zero data retention (OpenAI, Your data). Turning on a hosted vector store quietly moves documents into a store that persists until you delete it.
• Residency that covers inference but not storage, or the reverse. Providers split "where inference runs" from "where data is stored at rest", and global routing options often trade residency for price or availability. Read the specific setting, not the marketing page.
• Agent memory. Agents that persist "memories" about users across sessions create a profile store that nobody put in the data map.
• Feedback buttons. A thumbs-down with a comment can send the full exchange to a provider under different terms. Anthropic's commercial retention article, for example, states a five-year retention for feedback submissions as of October 2026 (Anthropic Privacy Center).

A control checklist for an LLM feature

Use this as a review gate before launch and again whenever the feature adds a new data source, tool or provider.

1. Inventory every store. Chat history, traces, application logs, vector database, source document bucket, eval sets, fine-tuning files, agent memory, caches, provider-side stored objects. Each gets an owner, a location, a retention rule and a note on how to find one person's data in it.
2. Record the purposes and bases. Serving the user, quality evaluation, fine-tuning and provider abuse monitoring are separate lines in the ROPA. Update the privacy notice to match.
3. Minimize prompts. Review what the system prompt and retrieval layer inject. Remove fields the model does not need. Filter retrieval by the requesting user's permissions so one customer's documents never reach another customer's prompt.
4. Redact or pseudonymize before logging. Decide whether traces need raw content at all. If they do, mask identifiers in the SDK before export and keep any re-identification key in a separate, access-controlled vault.
5. Set retention per store. Short TTLs on raw traces, longer only on aggregated metrics. Eval sets reviewed and pruned on a schedule. Hosted files and vector stores given expiry where the provider supports it.
6. Choose provider settings deliberately. Confirm training is off for your plan, decide whether you need zero data retention or modified abuse monitoring, pick the regional endpoint or data zone, and check which features you use fall outside those arrangements.
7. Paper the vendors. DPA with each model provider, observability vendor, vector database host and gateway vendor. Sub-processor lists reviewed and subscribed to. Transfer mechanism in place for each non-EEA flow.
8. Make requests answerable. Tag every stored record with a stable user or customer ID so access and deletion can be run by query, not by grepping text. Document how deletion propagates to embeddings, eval sets and backups.
9. Assess. DPIA where the screening criteria are met. AI Act classification (are you a provider or deployer, is any use high-risk, which Article 50 transparency duties apply).
10. Secure the stores. Traces and eval tools contain the most concentrated personal data in the stack and are often the least protected. Apply the same access control, SSO, audit logging and encryption you would to a production database; the security controls guide covers the baseline.
11. Test for leakage. Before launch, try to get the assistant to reveal another user's data, the system prompt's injected fields, or retrieved documents the user should not see. Repeat after significant prompt or retrieval changes.

Questions that come up

Need this built?

If you would rather have this implemented than read about it, tell us about your project and we will reply with questions or a rough plan.