ChatGPT

Core Requirements

1. The system should have low time-to-first-token (< ~500ms).
2. The system should be fault tolerant, recovering an in-flight stream without dropping tokens when a connection drops or a server fails.
3. The system should prioritize availability over strong consistency for conversation state (~99.9%+).
4. The system should scale under GPU-constrained capacity, with fair allocation across a tiered user base (free, Plus, Pro).

Below the line (out of scope)

• Durability of every streamed token (we persist the final assistant message, not each chunk).
• Authentication, abuse prevention, and content moderation.
• GDPR, data residency, and privacy compliance.
• Monitoring, logging, alerting, and CI/CD.

Here's how it might look on your whiteboard:

The Set Up

Planning the Approach

Before designing anything, take a moment to plan. This is a product-style question, so we'll build the design up sequentially, going one by one through the functional requirements. There are only two of them, so the high-level design will be intentionally short.

The work that actually makes this problem interesting lives in the non-functional requirements, where we have to stream tokens back fast, schedule limited GPU capacity, and keep cost under control. Those are what become our deep dives.

Defining the Core Entities

I like to begin with a broad overview of the primary entities. At this stage we don't need every column, just the nouns we'll reason about for the rest of the interview. We'll flesh out fields during the high-level design.

To satisfy our functional requirements, we'll need the following entities:

1. User: An account on the platform. Carries the tier (free vs paid), which is going to matter a lot once we get to fairness and scheduling.
2. Chat: A single conversation thread. Belongs to one user, groups an ordered sequence of messages, and carries a title and timestamps.
3. Message: One turn in a chat, either a user prompt or an assistant response. Carries the chatId, a role (user or assistant), the content, and a token count.

In the actual interview, this can be as simple as a short list like this. Just make sure you talk through the entities with your interviewer to ensure you are on the same page. We'll introduce one more entity, a Run, later once the deep dives actually need it, since it isn't something you'd naturally reach for this early.

API or System Interface

The API is the contract between the client and our system, and it's the bridge into the high-level design. We'll define one or two endpoints per functional requirement and keep moving.

First, a user starts a new chat. We use POST because we're creating a new Chat entity.

POST /chats -> { chatId }
Body: {}

Next, the user sends a prompt. We use POST because we're creating a new Message and starting the model response on the server.

POST /chats/{chatId}/messages -> Message
Body: {
  content
}

For the second functional requirement, we list a user's chats for the sidebar and load the messages for one chat. Both are GETs with cursor pagination, since a heavy user can have thousands of chats and a long chat can have thousands of messages.

GET /chats?cursor={cursor}&limit={n} -> Chat[]
GET /chats/{chatId}/messages?cursor={cursor}&limit={n} -> Message[]

High-Level Design

We'll keep this first pass deliberately naive, with synchronous calls and no streaming or queues. Starting with a working system and then breaking it is far better than starting with a "perfect" one that's hard to reason about.

1) Users should be able to send a prompt and receive an AI-generated response

When a user opens a chat, types a prompt, and hits enter, the client sends that prompt to our backend and eventually gets a response back. Let's lay out the minimum set of components to make that happen.

1. Web Client: The browser or mobile app where the user types prompts and reads responses. It's the chat UI.
2. API Gateway: The entry point for all client requests. It handles authentication, rate limiting, and routes requests to the right service.
3. Chat Service: A stateless service that owns chat and message persistence and orchestrates the call to the model. It's cheap to run and easy to scale horizontally, which matters because we'll want to scale it independently from the expensive inference layer.
4. Postgres: Our system of record, holding the chats and messages tables. The data is simple rows keyed by chatId and userId, so almost any database would work here. We reach for Postgres as a sensible default rather than because anything about the problem demands a relational store.
5. Inference Service: The pool of GPU machines that actually run the LLM. Each one runs a model server with the weights resident in its GPU memory, and those machines are what we'll call workers from here on. There's no tier in between, a request goes to a worker and that worker generates. We treat the model itself as a black box that takes a prompt in and returns a completion. This is the expensive, GPU-bound part of the system, separated from the Chat Service so we can scale and schedule it on its own.

Here's how these interact when a user sends a prompt:

1. The user types a prompt, and the client sends a POST request to /chats/{chatId}/messages.
2. The API Gateway authenticates the request and forwards it to the Chat Service.
3. The Chat Service writes the user's message to the messages table.
4. The Chat Service makes a synchronous call to the Inference Service, which runs the prompt through the model and returns the full completion once it's done.
5. The Chat Service writes the assistant message back to the messages table and returns it to the client.

The split between Chat Service and Inference Service is the one design decision to dwell on here. The chat tier is cheap and stateless, while inference is GPU-bound and expensive, and since we'll want to scale the two independently, we separate them now.

The obvious problem is that this is fully synchronous, so the client sits on that HTTP call until the entire response is generated, and a long response can take up to 30 seconds. That's 30 seconds of blank screen, which violates our TTFT requirement and feels broken.

On top of that, the Chat Service calls a GPU worker directly with no admission control. Nothing decides which requests to accept or turn away when the workers are already full. That breaks down the moment GPUs become the bottleneck. We'll fix the first problem with streaming and the second with a scheduling layer, both in the deep dives. For now, it works.

2) Users should be able to view past chats and resume a conversation with context carried across turns

Users expect to come back tomorrow, scroll their old conversations, open one, and keep going as if the model remembers everything. Two things have to happen here, a read path for past chats and context carry-over on the next turn.

We don't need any new services for this. We add the read endpoints off the existing Postgres tables and a context-loading step inside the Chat Service.

For the read path:

1. GET /chats returns the user's chats ordered by recent activity, cursor-paginated for the sidebar.
2. GET /chats/{chatId}/messages returns one chat's messages, cursor-paginated so a long conversation doesn't load all at once.

For context carry-over, when the user sends a follow-up prompt on an existing chat:

1. The Chat Service queries the messages table for the prior messages in that chatId, ordered by creation time.
2. It builds the prompt by concatenating those messages (with their roles, user vs assistant) followed by the new user message.
3. It sends that combined prompt to the Inference Service, just like the first turn.
4. The new assistant message gets written back to the messages table, so the next turn can read it too.

This is the simplest thing that works. The model sees the whole conversation every turn, so it behaves like it remembers. But sending full history every turn has two obvious problems. It breaks once a conversation grows past the model's context window, and it gets more expensive every turn since input tokens are billed per call. We'll tackle that with summarization and prefix caching in the deep dives.

That gets us a working system. It satisfies both functional requirements and has the bottlenecks our non-functional requirements warned us about. Let's go fix them.

Potential Deep Dives

With the functional requirements met, it's time to go back and satisfy the non-functional requirements. There's no one right set of deep dives here, or one order to tackle them in. In a real interview you'd have agreed on what matters with your interviewer when you outlined the non-functional requirements up front.

These are the ones I'd expect to come up for this question, roughly in the order the design pushes you toward them. We start with streaming the response and keeping that stream alive, then move to scheduling the GPUs, sharing them fairly, and keeping cost under control.

1) How do we stream tokens back to the client?

Our synchronous design makes the user wait up to 30 seconds for a blank screen to turn into a full answer. Fixing that means putting tokens on the screen as the model produces them, and there are two connections involved. One runs from the client to the Chat Service, and the other from the Inference Service back to the Chat Service.

Let's take the second one as given for now and assume the worker hands tokens back as it generates them over a server-streaming gRPC call. gRPC is a framework for service-to-service calls that runs over HTTP/2. The Chat Service makes one call, and the worker streams messages back over that open connection, one token at a time. A plain request/response call wouldn't do, since it would force the worker to buffer the entire 30-second completion and hand it back in one lump, which is the blank screen all over again. We reach for gRPC rather than WebSockets on this internal hop because we want HTTP/2's binary framing and multiplexing between our own services, and protocol buffers give us a typed contract on top of that.

That leaves the half the user actually sees, which is how the Chat Service gets that stream down to the browser.

Before we wire this up, we need a name for the thing being streamed. Our entities so far are User, Chat, and Message, and none of them fit. A message is the finished text, while what we're streaming is an attempt to produce it, and that attempt has a lifecycle of its own. It can be queued, it can run, it can fail halfway through, it can be cancelled, and it can be retried, all before there's a completed assistant message to write down.

So we add a runs table. A Run is one attempt to generate an assistant response. The Chat Service creates it with a fresh runId when the user's message lands, before inference starts. The row carries the chatId and messageId it belongs to, a status that moves from queued to streaming and then to done, cancelled, or failed, the model that served it, and the input and output token counts we'll lean on for billing and quotas.

That runId is what reshapes the message endpoint from the API sketch. POST /chats/{chatId}/messages no longer waits around for the model. It creates the Run and returns { runId } immediately, and the client follows the tokens on a separate GET /runs/{runId}/stream. Keeping that stream on a GET matters because the browser's native EventSource API only opens GET requests.

Everything we build in the deep dives ahead, the queue, the workers, and the cancellation path, refers to the same run.

That gives us a simple first version of streaming.

1. The Chat Service persists the user's message, creates a Run, and returns the runId to the client right away.
2. The client opens an SSE connection for that runId.
3. The Chat Service sends the assembled prompt to the Inference Service, and the worker streams tokens back over gRPC as it generates them.
4. The Chat Service forwards each token down the SSE connection, and the browser appends it.

The user starts reading the answer as soon as the model starts producing it rather than waiting 30 seconds for the whole thing.

2) How do we keep the stream alive across reconnects and deploys?

That works, but it assumes one fixed server sits between this user and the model for the full 30 seconds. The Chat Service instance holding the SSE connection is also the one holding the gRPC stream from the worker, so one specific instance has to sit in the middle for the entire generation. Our Chat Service tier is stateless, horizontally scaled behind a load balancer, and redeployed all day long, which makes that instance disappearing mid-generation routine.

What we want is to separate the worker producing tokens from the Chat Service instance delivering them. The worker shouldn't need to know which instance is holding the user's connection, and when a client reconnects and lands on a different instance, that instance should be able to pick up the same run where the last one left off.

Here's the full flow when a user sends a prompt:

1. The Chat Service persists the user's message and creates a Run with a fresh runId, returning it to the client right away.
2. The Chat Service hands the assembled prompt and that runId to an inference worker, which starts generating.
3. The client opens an SSE connection for that runId, and the load balancer routes it to any Chat Service instance.
4. That Chat Service instance starts a blocking XREAD on the runId stream, from the beginning for a fresh connection or from the client's last-seen entry ID on a reconnect.
5. The inference worker generates tokens and XADDs each one to the runId stream.
6. The Chat Service instance reads new entries, forwards them down the SSE connection, and the browser appends them. If the connection drops, the browser reconnects, lands on some Chat Service instance, and that instance replays from the last-seen ID before continuing live.

Notice how much step 2 glosses over. The Chat Service hands the prompt to a worker and it starts generating, as if there's always one sitting idle waiting for it. At 20k prompts per second against a pool of GPUs we can't add to on demand, there often isn't.

3) How do we route and schedule generation requests across GPU workers?

GPUs are the bottleneck. They're the most expensive resource in the system and the one in shortest supply, so how we route work to them decides both our cost and our latency under load.

The scale of that expense is easy to underestimate. A frontier model is far too big to fit on a single GPU, so its weights get split across all the GPUs in one server, typically eight of them. Serving 120k concurrent streams means standing up thousands of those servers. That puts you at tens of thousands of GPUs for this one model, and the labs running systems at this scale spend staggering amounts on compute, easily hundreds of millions to billions of dollars a year.

When the hardware costs that much, every percentage point of unused capacity is real money.

GPU batch

Request A -> token 42
Request B -> token 17
Request C -> token 91
Request D -> token 8

one forward pass

Request A -> token 43
Request B -> token 18
Request C -> finished
Request D -> token 9

next pass

Request A -> token 44
Request B -> token 19
Request E -> token 1
Request D -> token 10

4) How do we keep heavy users from taking over the GPU pool while giving paid tiers a better experience?

This is a multi-tenant system sharing one limited GPU pool, and the costs are wildly uneven. One user firing a 30k-token prompt burns far more compute than a hundred users sending one-liners. We need fair sharing so nobody can starve everyone else, and we need business priority across tiers so paying customers get a better experience when things are tight. A flat requests-per-minute cap can't express either of those.

5) As conversations get longer, how do we control inference cost without making the assistant feel forgetful?

Recall our high-level design replays the entire conversation into the model on every turn. It works, but its cost and latency grow with the chat.

A 50-turn chat at ~500 tokens per turn means we're shipping ~25k input tokens on the next prompt, all billed per call. Worse, it has a hard ceiling. Once the conversation grows past the model's context window the request simply can't fit.

Real assistants usually just surface this, telling you the chat has gotten too long rather than failing silently. But leaning on that as your only answer means the product stops working for the power users who chat the most. It's fine for a five-turn chat and untenable as a general approach. What we want is to keep the assistant feeling like it remembers without paying to re-read the whole transcript each time.

SELECT * FROM "messages"
WHERE "chatId" = $1
ORDER BY "createdAt" DESC
LIMIT $2

Cancelling a run and reclaiming the GPU

The one operation on a Run we haven't shown yet is cancellation, and it's the clearest payoff for having made the run first-class. When the user hits stop, the client makes a plain HTTP call to POST /runs/{runId}/cancel. SSE only pushes one way, so anything the client needs to send mid-stream travels over a separate request like this one. The Chat Service flips that Run's status to cancelled and publishes a cancel signal on a control channel keyed by the runId. The worker checks that channel between token batches, and the moment it sees the signal it drops the sequence and stops generating.

Cancellation immediately frees capacity that would otherwise stay occupied for the rest of the 30-second generation. GPU capacity is the most limited and expensive resource in the whole system, so reclaiming it the instant the user stops caring is real money back.

Worth being clear that closing the tab is not a cancel. We built the Redis Stream and SSE reconnect precisely so a dropped connection isn't read as the end of a run, and like ChatGPT we keep generating in the background. The user can reopen the chat and reconnect to the stream, or just refetch the finished message from Postgres once it's done. Cancellation has to be an explicit signal from the user, never an accident of the network.

Final Design

After applying the "great" solutions, the design has grown well beyond the naive synchronous version we started with. Here's roughly how it all fits together:

Some additional deep dives you might consider

There's plenty we couldn't fit here. A few more directions worth thinking through on your own:

1. Safety and moderation: We put content moderation below the line to keep the focus on serving, but plenty of interviewers will want to see it. The usual shape is a cheap classifier on the prompt on the way in, and a second pass on the output as it streams. The output pass is where it gets tricky, because you've usually already streamed some tokens to the user by the time the check trips. You have to decide whether to moderate in chunks before flushing each one, or pull the message back after the fact.
2. Why one model spans a whole server of GPUs: We said the weights get split across the GPUs in a server without saying how. A frontier model's weights don't fit in a single GPU's memory, so you split them across GPUs. You can use tensor parallelism, where each GPU holds a slice of every layer, or pipeline parallelism, where each GPU holds whole layers and hands off to the next. Both lean on fast interconnects, NVLink between GPUs inside a server and InfiniBand between servers, and that interconnect can become its own bottleneck.
3. Speculative decoding: For some interviews, especially at the senior and staff level, you'll go much deeper into inference internals, and this is one you should know. Because decode runs one token at a time, a small cheap "draft" model can guess the next few tokens and the big model verifies all of them in a single forward pass. When the draft guesses right you get several tokens for the cost of one step, a real win for both time-to-first-token and throughput.
4. Cheaper requests through routing and caching: Not every prompt needs the biggest model. Routing simple queries to a smaller model, and caching responses for semantically similar prompts (keyed off an embedding rather than the exact string), both cut cost without the user noticing. This pairs naturally with the tiered fairness deep dive, where free traffic is the first to get routed down.
5. Multimodal input and cross-chat memory: We scoped to text in, text out. Real ChatGPT also takes images and audio, which changes tokenization and makes the input much larger to process, and it remembers facts about you across separate conversations. That cross-chat memory is a retrieval problem, embedding past messages and pulling the relevant ones into the prompt, rather than the single-conversation summarization we did here.

What is Expected at Each Level?

Okay, that was a lot. You may be thinking, "How much of that is actually required from me in an interview?"

Mid-level

For this question, a mid-level candidate will have clearly defined the API endpoints and data model. They'll have landed on a working synchronous high-level design that handles sending a prompt and viewing past chats with context carried across turns. I want to see them recognize that a 30-second blank screen won't fly and reach for a push-based streaming model like SSE, even if it takes some prompting.

They should understand that GPUs are the bottleneck and at least propose putting a queue in front of the workers. They may not get to continuous batching or backpressure on their own.

Senior

For this question, senior candidates are expected to speed through the high-level design. That leaves time to cover at least two of the streaming fanout, GPU scheduling, and fairness deep dives in detail. You should be able to explain the SSE-vs-WebSocket choice from the one-way nature of token streaming, and how queueing plus continuous batching keeps the GPU busy.

I also expect a senior candidate to recognize that cost grows with conversation length and to propose summarization or truncation, even if they don't reach prefix caching unaided.

Staff+

Staff+ candidates should drive at least three deep dives with real depth and bring the GPU economics into the conversation without being asked. The back-of-envelope is that ~120k concurrent streams means tens of thousands of GPUs and a seven-figure daily bill. That scale is what justifies continuous batching and backpressure in the first place.

They reach prefix caching for context cost on their own. They also separate fairness across users (cost-aware per-user budgets) from priority across tiers (tier-weighted queueing). Beyond that, I expect at least one non-obvious serving choice explained well, whether that's continuous batching, KV-cache reuse, or why free traffic degrades first.

At this level, interviewers may also expect some inference details, such as speculative decoding. It has become common enough in production serving systems to be fair game, even though it is less fundamental than the scheduling and caching ideas above.