Q: How does an LLM remember the conversation? (context and history management)

Here's something that surprises almost everyone: it doesn't. LLM APIs are stateless. Every API call starts from zero, and the model has no memory of anything you sent before.

So how does ChatGPT hold a conversation? Behind the scenes, the app resends the entire conversation history with every new message. The model reads the whole thing again, from the first message to the latest one, and generates the next reply. Then it forgets everything again. What feels like memory is re-reading, every single turn.

When you build AI applications, managing this is your job. In your code, you keep a list of messages. Each time the user says something, you append it to the list, send the full list to the API, get the reply, and append that too. That list is the conversation, and it lives on your side, not the model's.

Two consequences follow, and they shape real applications. First, cost: because the model re-reads everything each turn, long conversations get more expensive with every message (prompt caching helps here). Second, limits: the conversation has to fit in the context window, so production apps trim old messages, summarize earlier parts of the conversation, or store important facts somewhere more permanent.

This is called context management, and it's one of the first real engineering skills in working with LLMs. It's also usually the moment engineers realize an LLM is not a magic being that knows them. It's a stateless function: everything it "knows" about you is in the text you sent it.