Tokens, context window and memory: what happens inside when you talk to an AI model

📺 Watch the video on Instagram

You already know AI needs context to give good answers. What people usually don't explain is what happens inside, and why giving it too much context can be just as bad as giving it too little.

Everything becomes tokens

Everything you send to a model, your question, attached files, the conversation history, gets converted into tokens before it's processed.

A token is roughly 4 characters. In practice:

Every model has a limit on how many tokens it can process in a single conversation. That's called the context window.

Sounds huge. But in real projects, with large files, detailed system instructions and long conversations, that limit shows up sooner than you think.

What nobody tells you: the model doesn't fail, it degrades

Here's the part most people don't know: when the context fills up, the model doesn't throw an error. It doesn't warn you. It simply starts paying less attention to whatever is in the middle.

The instructions at the beginning get diluted. Important details you mentioned 20 messages ago get "forgotten". Response quality drops, and you assume the problem is your prompt.

Short context     → focused model, precise answers
Long context      → scattered attention, diluted instructions
Saturated context → the model "forgets" what's in the middle

This phenomenon is known as lost in the middle: models pay more attention to the beginning and the end of the context, and less to whatever is buried in the center.

The answer isn't always more context. It's the right context.

That's why many tools implement context compression: a way to automatically summarize long conversations, files or documentation so the model keeps only what matters. It cuts noise, saves tokens and lets the AI focus better on the task at hand.

Context window ≠ persistent memory

This is the most common confusion, and it's worth making it crystal clear.

The context window is what the model has available in this conversation. When you close it, it's gone. The next time you open a chat, the model starts from scratch, even if you talked to it for hours the day before.

Persistent memory is different: it's information stored outside the model and injected into future conversations. Products like Claude.ai or Cursor implement it in different ways, but it isn't a native capability of the model; someone had to build it on the outside.

Understanding that difference changes how you design your AI workflow. If you rely on the model "remembering" something from a previous session without persistent memory enabled, you'll end up repeating context all the time, or getting answers that ignore important information.

Three practical rules for managing context

With all of that clear, the rules are simple:

1. Include only the files that are relevant to the task. Don't paste the whole repo. The more noise, the worse the focus. Pick the files the model actually needs for the specific task you're giving it.

2. If the conversation gets long, start a new one with a summary of the context. Instead of piling up messages in the same thread, summarize the key points and start a fresh chat. The model will have more attention available from the start.

3. Put important instructions at the beginning and the end, not in the middle. The lost in the middle effect is real. Whatever is buried among many messages gets diluted. What sits at the beginning and end of the context carries more weight.

Coming up in the next episode

Now that you know how context works, the natural next step is to structure it permanently: so the agent has the right context from the very first message, without you having to paste it every time.

In the next video we show you exactly how to do that.