The AI Liability Chain Part 1 - What happens to user data when you build on third-party AI?
It's become incredibly easy to spin up your own AI chatbot. For instance, you can instruct your coding agent to build a health app that functions as a personal instant GP. Users can ask your chatbot about symptoms they're experiencing and the chatbot spits out possible causes, remedies and medical knowledge. You don't need to train your own foundational AI model or build the next Open AI for health, all you need are their API keys. But once you start sending real user information through that model, the simplicity of building the product starts to hide much more complicated questions: where does that information actually go? Which systems can access it, where is it stored, and how can you design the product so that users can trust how their information is handled?
The answers depend on the architecture of the product you're building. The way you design the information flow in your system and how you connect your app to third-party models will determine the privacy and safety risks that emerge. I will attempt to break down all of this - how third-party AI models can work inside your product, where user data goes, and the privacy and safety risks to consider with various architectural choices.
The architecture
Let's take a look back at our hypothetical health app, which we'll call Remedi, and assume that it uses OpenAI for its own AI chat bot. When a Remedi user types: "I've had a headache and a fever for two days. What could be causing it?" and taps Send, here's roughly what happens:
User → Remedi app → Remedi's backend → OpenAI → Remedi's backend → Remedi app → User
The user (let's call her Amy) types her message into the Remedi app (the frontend) and the app sends Amy's message to Remedi's own servers, which sit behind the scenes. Think of these servers like the app's control room: they receive her question, handle the data, and talk to other systems on her behalf.
The Remedi app sends Amy's question to OpenAI, whose AI model reads it and works out a reply. For Remedi's servers and OpenAI's servers to "talk" to each other, they use something called an API. Think of it as a messenger that carries requests and answers back and forth between two systems.
When Remedi sends your message to OpenAI, that's called an API request - Remedi is basically saying, "here's a question, please answer it." The app proves to OpenAI it's really them asking by including an API key, which is kinda like a password that identifies Remedi. When OpenAI's model works through Amy's question to come up with an answer, this step is called inference, but you can just think of it as "the AI thinking." The answer travels back the same way it came, through the API, to Remedi's servers, and finally back to Amy.
That's the journey in a nutshell. That's how the message travels. But what happens to the user's information along the way, and where could things go wrong?
Following the data
Let's take a closer look at the journey and what happens to the user data at each point:
1. The user: what information are we collecting?
When Amy types into the Remedi chat "I've had a headache and a fever for two days. What could be causing it?" and hits send, it seems like a simple enough question. However, this is a piece of health information. And if Amy signed up for the app previously, Remedi may know her name, email address, age, location, and history of questions asked. Those are all identifiers (PII)/Personal data.
So before the message goes anywhere, we need to ask: what information does Remedi actually need in order to answer Amy's question? For example, it's probably not necessary to know Amy's age just to answer a question about a headache. This idea - only collecting and using what is necessary - is called data minimisation.
Remedi could strip out some of the identifying information before Amy's data goes any further. Her name, for example, could be replaced with something like User 84721. This is pseudonymisation. Remedi can still work out who User 84721 is, so the data isn't truly anonymous. Anonymisation goes further: the information can no longer reasonably be traced back to Amy.
The distinction matters because the easier it is to connect sensitive information back to a real person, the more damaging it could be if that information is leaked, stolen or ends up somewhere it shouldn't.
2. The frontend: data leaves the user's device
After she sends her message, it sits on Remedi's frontend. The frontend then sends that message to Remedi's backend, usually over HTTPS, which encrypts the data as it travels between the user's device and Remedi's servers.
This is where things start to get a little more complicated as it's the first point where data starts leaving Amy's device. The app itself might also have other tools quietly running behind the scenes; like analytics, which track how Amy uses the app, or crash reporting, which logs what happens if something breaks.
A few things that could go wrong at this point. Those background tools could end up collecting information they shouldn't. Remedi could also send more information than it actually needs, or send it to the wrong place.
3. The backend: assembling the AI request
The backend of the app does a bunch of things, including building the actual request it will send to the 3rd-party AI model. The backend decides what info the model needs in order to answer Amy. This can include: her last question, or a full history of her questions, a system prompt, some particulars from her account, and supplementary information from a certain section known as RAG.
So what looks like one simple question from Amy's side can actually be a much bigger package.
The way information is packaged by the app to send to the external AI model can pose a few risks here. Remedi might include more than it needs to (Amy's entire chat history, for example) when only her latest message actually matters. Or the extra background information could accidentally pull in things it shouldn't. There's also the question of who can see all of this sitting on Remedi's servers - developers, admins, or other connected tools may have access to it.
4. The API request: sending the data to OpenAI
Once the user data is packaged and Remedi assembles the request, it sends it to OpenAI through the API. This is where the data leaves Remedi's systems and reaches a third party. The user's message (and potentially their conversation history, system instructions and RAG context) all travel with it.
The big question is, what does OpenAI do with this data? OpenAI says API inputs and outputs aren't used to train its models by default. But that doesn't necessarily mean the data isn't retained. Depending on the API and configuration, data may be retained for things like abuse monitoring. Eligible customers can use Zero Data Retention (ZDR) on supported endpoints, which means prompts and responses aren't retained after processing.
So what could go wrong?
The data is now in someone else's infrastructure. There could be a security breach, unauthorised access, a retention setting that keeps data longer than expected, or a problem with another service involved in processing the data. And even if Remedi has configured everything correctly, it's still relying on another company to handle sensitive information.
A carefully considered question in using someone else's AI model is what happens to the data when they are no longer under Remedi's control. (I will leave aside who is responsible for what happens after that for the moment - I shall come to that in Part 2 of this article series.)
5. The model: processing the information
OpenAI passes Amy's question to the model Remedi has chosen, which uses everything it's been given (her question, previous messages, Remedi's instructions and any other context) to generate a response.
But there are two things to worry about here. What goes into the model, and what comes out. We've already looked at why the information going in needs to be protected. The answer coming out creates a different problem.
AI can be very confidently wrong. It could misunderstand Amy's symptoms, invent something entirely, or suggest something that causes harm. A convincing answer isn't necessarily a correct one.
So how much can we actually trust what the model tells us?
6. The response: back to Remedi
The moment the model has worked its magic and generated a reply, it is passed back to the Remedi server by way of the API. At this point Remedi could simply display the reply to the user, or it could run more checks, apply business rules, store the conversation, or send the reply on to another service.
Another danger lurks here. Remedi might show an unsafe or incorrect answer because it doesn't bother to check it first. Then again, the answer might contain sensitive data that is logged or stored somewhere without Amy's knowledge. All of which is in Remedi's hands.
7. Storage and logs: what gets left behind?
Developers need to know what's happening inside their applications, especially when something goes wrong, so they use logs. Logs can automatically record when a request was made, whether it succeeded, how long it took and any errors that occurred. But depending on how Remedi is built, logs could also capture Amy's messages or the AI's responses. This matters because logging can create another place where Amy's health information is stored, potentially even with a third-party logging or monitoring service.
Logs can lead to one message making a surprising number of duplicates. For instance, a person may have removed their conversation from the app without realising it may be preserved in logs or backups. And the more places the data is located, the bigger the likelihood of it being exposed.
Closing
When building an app like Remedi on top of someone else's AI model, you're crafting the path your user's data travels down. Where the path leads, who can see the data along the way and what happens if something goes wrong depends on the choices made in the architecture. The AI model is really just one small part of a much bigger system. The technical question starts to merge with the legal question of who is actually responsible when something goes wrong. With a health app like Remedi, there is also a question of whether it constitutes a medical device and the legal considerations that come with that.
In the next part of this article series we will talk about how things can go wrong and who is actually responsible for it.
Originally published on MAED.
Glossary
API - The way two different pieces of software talk to each other. Remedi, for example, uses an API to send a request to OpenAI and get an answer back.
API key - A secret key that lets a service know who's making the request and whether they're allowed to use the API. A bit like a password for an application.
API request - A request sent from one system to another. In our example, Remedi sends OpenAI a request asking the model to generate an answer.
Backend - Everything happening behind the scenes of an app. The backend receives requests, runs the app's logic and talks to databases and other services.
Frontend - The bit you actually see and interact with: screens, buttons, forms, text boxes and everything else that makes up the interface.
Server - A computer (or, more commonly, a group of computers) that receives requests and does work for an application. Remedi's backend runs on servers.
Database - Where an application keeps organised information it needs to come back to later, like accounts or conversation history.
AI model / LLM - The technology actually generating the answer. LLM stands for Large Language Model.
Inference - The process of giving an AI model some information and having it generate an answer. When Amy asks Remedi a question and the model produces a response, that's inference.
System prompt - Instructions that sit behind the scenes and tell the model how it should behave. Remedi might use one to tell the model what kind of answers to give, what it shouldn't do and how it should respond.