What is a context window in LLMs and why does it matter?

ontext window is the maximum number of tokens a large language model (LLM) can process in a single session. It includes both the tokens from your input (prompts) and the tokens the model generates in its responses. It matters because it defines the limit of the model's "memory." Once the context window is full, the model can no longer take in new information without losing earlier parts of the conversation. For example, Claude Opus 4.5 has a context window of 200k tokens.

What is context rot and how can I avoid it?

Context rot is the gradual decline in the quality and accuracy of an LLM's responses as the context window fills up, especially when unrelated or noisy topics are introduced into the conversation. To avoid it, keep your sessions focused on a specific subject with relevant data. Avoid jumping between unrelated topics in the same chat, as this adds "noise" that forces the model to filter through irrelevant information, leading to less accurate and more "forgetful" answers.

What are tokens and how do they affect my interactions with AI models?

okens are the basic units that LLMs use to process text. Each word or even part of a word is converted into a numerical token ID using the model's internal dictionary. Both your input and the model's output consume tokens, and together they count toward the context window limit. Understanding tokens is important because the more tokens you use on unnecessary or noisy input, the less room remains for meaningful responses, which can directly impact the quality of the AI's output.

Understanding Context Window

In our previous posts, we talked about prompting techniques Basic Prompting Techniques and Reasoning Prompting Techniques. Now, let’s understand: What is context? Why is it so important? And how does it work? There are three main concepts we need to grasp.

1. What are tokens and how do they impact context?

Each time we ask an LLM a question, it calculates the number of tokens involved. Every model has a maximum limit of tokens—known as the Context Window—that it can process at any given time.

How does it work? When an LLM is trained, a "dictionary" of tokens is created. For example, the word "hello" might be assigned the ID 57. When the LLM receives text, it breaks the string down, finds the corresponding IDs in its dictionary, and processes those numbers.

2. What is the Context Window?

As we mentioned earlier, each model has a limit on the number of tokens it can handle in a single session. This is called the Context Window. For example, Claude ops 4.5 has a max context window of 200k tokens.

Why is it important? It means you cannot overload the LLM with infinite questions or data. You are always limited by the maximum capacity of the model's "memory."

3. What is "Context Rot" and why does it matter?

We’ve covered the technical side, but what is "context" in a practical sense? Context represents the connections between subjects discussed in a conversation. When we start a new session with an LLM, we should aim to stay focused on a specific subject with specific data. This helps avoid Context Rot.

What is Context Rot? Context rot occurs when an LLM is forced to process too many tokens or too many unrelated topics. As the window fills up, the LLM's responses often become less accurate or "forgetful." Furthermore, adding unrelated tangents to a chat increases noise, which accelerates this decay in quality.

There is a interesting research about the subject - link

Example: High-Noise Context (Messy):

📝Markdown
1
2
3
4
5
6
7
8
9
John: Hey Dani, how’s it going? 
Dani: Good! How about you? 
John: Doing well, just a bit sleepy. I stayed up until 2 a.m. finishing "Stranger Things". 
Dani: I’ve never seen it—is it any good? 
John: It’s great! I’m a huge sci-fi fan. Have you seen any of the Marvel movies? 
Dani: Yeah, but they’re not really my thing. 
John: No way! I love Spider-Man. I used to watch the animated show all the time as a kid. 
Did you ever see that one?
....

In this exchange, John jumped between three subjects: Stranger Things, Marvel, and the Spider-Man TV show. If John later asks a specific technical question, Dani has to filter through all this unrelated "noise."

Example: Clean Context (Focused):

📝Markdown
1
2
3
4
5
6
7
8
9
10
11
John: Hey Dani, how’s it going? 
Dani: Good! How about you? 
John: Doing well, just a bit sleepy. I stayed up until 2 a.m. finishing Stranger Things. 
Dani: I’ve never seen it—is it any good? 
John: It’s great! I’m a huge sci-fi fan. It starts with a group of kids 
playing Dungeons and Dragons, and the cool part is that the monsters from 
the game actually come to life. 
Dani: How many seasons are there? 
John: There are five, and each episode is about an hour long. 
They’re all connected, so every episode really impacts the next one. 
Dani: Sounds interesting! Maybe I’ll check it out.

Loading...

Vibki