What Is a Context Window in AI? How It Works

Have you ever been deep in a long chat with an AI assistant and noticed it forgot something you said at the start? That's usually down to the context window. It's one of those terms that gets thrown around in product announcements, and it helps to know what it really means for the work you do.

The context window as an AI's working memory

Anthropic's documentation defines the context window as all the text a language model can reference when generating a response, including the response itself. It describes this as a kind of working memory, and it's separate from the huge body of data the model was trained on.

IBM puts it similarly: the amount of text, in tokens, that the model can consider or remember at any one time. It determines how long a conversation can run before earlier details drop out.

Tokens, the unit a context window is measured in

Models don't count words; they count tokens. IBM explains that a token can be a character, part of a word, a whole word or a short phrase. There's no fixed conversion, but IBM says a decent estimate is roughly 1.5 tokens per word in English.

Languages differ. IBM gives an example where an English sentence and its translation into Telugu, which had fewer characters, produced over 7 times as many tokens. So if you work in several languages, the same window may hold less in some than in others.

What takes up space in a context window

It isn't only your latest message. Anthropic says everything in the request counts: the system prompt, every message in the conversation, tool results, images and documents, and the model's own output.

• Your instructions and the whole chat history.
• Any files, pasted documents or images you add.
• The assistant's earlier replies.
• Hidden instructions the tool adds behind the scenes.
• The answer it's writing right now.

This is why a long chat with big attachments can use up the window faster than you'd expect.

What happens when a context window fills up

IBM says that when input goes beyond the window, it must be truncated or summarised for the model to proceed. Anthropic notes that chat interfaces can manage this on a rolling first in, first out basis, meaning the oldest material falls away first. Anthropic also describes automatic summarising of earlier parts of a conversation as one way tools keep long sessions going.

In practice, if an assistant suddenly ignores a rule you set early in the chat, the context window is a good first suspect.

Is a bigger context window always better?

Not automatically. IBM says a larger window can bring more coherent responses, longer conversations and the ability to handle longer documents in one go. But Anthropic is clear that more context isn't automatically better: as the number of tokens grows, accuracy and recall degrade, a phenomenon it calls context rot. Curating what's in the window matters as much as how much room there is.

IBM lists further drawbacks. Compute requirements scale quadratically with sequence length, so very long inputs are demanding. Responses can get slower as context grows. And models tend to perform best when relevant information is near the beginning or end of the input, with information in the middle often getting lost.

How to work well within a context window

1. Put the important material first or last. IBM's point about the middle being weaker is a handy reminder.
2. Start a fresh chat for a new topic. Old, unrelated text just takes up room.
3. Paste only what's needed. A relevant page beats an entire manual. For long documents, Anthropic's guidance suggests placing the long material above your question.
4. Summarise as you go. Ask the assistant for a short recap of decisions, then begin a new chat with that recap.
5. Repeat key rules. If a rule matters, restate it near your request rather than trusting the start of a long chat.

If you're after accurate answers about your own business, giving the tool the right facts in the window is half the job, which I cover in what to do when ChatGPT is missing your business details.

Context window versus training data and memory

These get mixed up all the time. The context window is what the model can see right now. Training data is what it learned beforehand, which I explain in my piece on where ChatGPT gets its information. Some assistants also offer a saved memory feature across chats, and what that does depends on the tool, so check its help pages.

Window sizes vary by model and account and change over time, so I won't quote numbers here. The tool's official model page is the place to check. If you'd like a hand choosing and setting up the right assistant for your team, that's part of my work as an AI consultant.

Frequently asked questions

What is a context window?

It's the amount of text, measured in tokens, that an AI model can consider at once, including your messages, any attachments and its own reply.

What is a token?

A token is the unit models use to read text. It can be a character, part of a word, a whole word or a short phrase. IBM suggests about 1.5 tokens per word in English.

Why does my AI forget things from earlier in a chat?

Older material can fall out of the window, or get less attention as the conversation grows. Starting a new chat with a short recap usually helps.

Is a bigger context window always better?

No. Anthropic says accuracy and recall can degrade as tokens grow, which it calls context rot.

Is the context window the same as training data?

No. Training data is what the model learned beforehand. The context window is the working space for the current conversation.

How big is the context window of my AI tool?

It varies by model and account and changes over time, so check the tool's official model information.