What the Context Window Is, and What to Do About It

You’ve probably felt this without having a name for it. You start a chat with an AI and the first answers are great. An hour later, in that same conversation, it’s gotten worse. It forgets something you told it earlier. It suggests an idea you already turned down. The replies get vaguer. It feels like the model quietly got dumber as you went.

It didn’t. You filled up its context window. Once you know what that is, you can stop it from happening, and the fix is easy.

What it is

Every AI chat has a working memory. Think of it as the AI’s desk. Everything it can see while answering you has to fit on that desk: your instructions, anything you pasted, and the entire conversation so far, your messages and its own replies. That space is the context window, and it’s a fixed size.

Two things are worth knowing about it.

It’s separate from what the AI “knows.” The model learned an enormous amount during training, and that knowledge is permanent. The context window is just what it happens to be looking at right now.

And it doesn’t carry over. When you start a new chat, the desk is wiped clean. The AI remembers nothing from your last conversation. That conversation just isn’t on the desk anymore. (It’s the same reason an AI can feel like a brilliant new colleague who forgets everything overnight, which I get into in Your AI Has No Idea What Your Company Does.)

The size is measured in “tokens,” which are just chunks of text, roughly three-quarters of a word each. You don’t need to count them. You just need to know the desk has an edge, and you can run out of room.

Why a full window hurts

Two reasons. The first you’d guess; the second surprises most people.

First, a fuller window is slower, and if you’re paying per use, pricier. There’s more to read and process on every single message.

Second, a full window makes the AI less accurate. That’s the opposite of what most people assume. It feels like more context should mean a better-informed answer. Past a point, it doesn’t, and the model starts to lose track. When you cram a long document in, models are noticeably worse at using the information buried in the middle of it than the parts at the start or the end. Researchers even have a name for it, “lost in the middle,” and it’s a measured, repeatable effect, not a hunch.

The picture I keep in my head: you brief a sharp new hire by dropping a 200-page binder on their desk and saying “it’s all in there.” You technically gave them everything. But the sentence that actually mattered is on page 104, and they’ll skim right past it. A clean one-page brief beats the binder every time. Same with AI. Less, but relevant, beats more.

So when a tool advertises a giant context window (“a million tokens!”), take it as a ceiling, not a target. The amount it can hold and the amount it can think clearly across are not the same number.

What to do about it

This is one of the easiest problems in AI to manage. A handful of habits cover most of it.

Start a new chat when you switch topics. The single most effective thing you can do, and the one people resist most. A fresh chat is a clean desk. If you’ve finished one task and you’re moving to another, don’t keep going in the same window. Open a new one. You’ll feel the difference in the first reply.

Give one chat one job. Long conversations that wander across five topics are exactly how windows fill up and answers drift. Decide what you want out of a session before you start, stay on it, and wrap up once you have it.

Don’t paste more than you need. If your question is about one section of a document, give it that section, not the whole PDF. Every extra page you add is one more thing the real answer has to compete with.

Put the permanent stuff in a permanent place. Anything that’s true in every conversation (who you are, how you like things written, what your project is) shouldn’t be retyped each time. Most tools let you set it once: “custom instructions” or a “project” in ChatGPT and Claude, a CLAUDE.md file in Claude Code. Set it once and it loads automatically without clogging the chat. I wrote a full guide on this: The Setup That Makes AI Stop Being Generic.

Turn off tools you’re not using. If you’ve connected apps or plugins, each one quietly takes up room in the window before you’ve even typed. Keep what the task needs, switch off the rest.

When you really must keep going, summarize. Sometimes a session has to run long. You can ask the AI to write a short summary of what matters and start a fresh chat from that, instead of dragging the whole history along. In Claude Code there’s a command for it, /compact, that does this in place. Just remember the summary is the AI’s best guess at what’s important, so it can drop a detail. Don’t treat it as perfect memory.

If you use Claude Code (it runs in the terminal, a desktop app, and the browser), two small extras help: a bar shows how full the window is as you work, and /context shows you what’s eating the space, so you can catch a bloated file or a tool you forgot to switch off.

How I do it

In case it’s useful, here’s my own approach. I give each session a single, specific goal before I start. Not “work on the website” but “add the context-window guide and wire it in.” I track all the changes in one pull request, and the moment the goal is met I merge it, delete the branch, and archive the session. The next task gets a clean session of its own.

It looks like project tidiness, and it is. But it’s also context management without trying. A session scoped to one goal never gets the chance to bloat or drift. The work ends before the window does.

The short version

Most of the time, when an AI seems to get dumber, it hasn’t. Its desk got cluttered. A small, clean context beats a big, full one almost every time, so clear the desk often and put on it only what the task actually needs.

---

Next: to set up the permanent instructions that keep stable context out of your chat window, see The Setup That Makes AI Stop Being Generic. For why company knowledge, not raw context, is what makes AI useful at work, read Your AI Has No Idea What Your Company Does.