Why AI forgets the start of a long chat, and how Nivana keeps it

What a context window is

A language model has no memory between messages. Every time you send one, the app sends the whole conversation again, and the model reads all of it before it writes a word. The amount it can read at once is its context window: in Anthropic's words, "all the text a language model can reference when generating a response, including the response itself", a kind of "working memory" for the model.¹ It is measured in tokens, roughly three quarters of a word each in English.

Windows have grown from a few thousand tokens to hundreds of thousands, so it is tempting to think the problem has gone away. It has not, for two reasons. A long chat with attached documents, code and images still fills a window faster than you would expect. And a model reading a very long input does not read all of it equally well.

Lost in the middle

In 2023, researchers at Stanford and elsewhere tested how well models use information placed at different points in a long input. Performance was often highest when the relevant fact sat at the very beginning or the very end, and it "significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models".² The study was published in Transactions of the Association for Computational Linguistics.

NVIDIA researchers built a benchmark called RULER to check whether advertised context lengths hold up. Of the long-context models they tested, all of which claimed 32,000 tokens or more, "only half of them can maintain satisfactory performance at the length of 32K".³ In July 2025, Chroma evaluated 18 models and reported the same pattern even on simple tasks: "model performance varies significantly as input length changes".⁴ They called it context rot.

A bigger window means the model can accept more text. It does not mean the model will use every part of that text well.

What chat apps usually do about it

• Cut the oldest messages. Simple and cheap, and the reason an assistant forgets your opening instructions.
• Send everything until it errors. The chat works until it suddenly does not, and long chats get slow and expensive along the way, because every turn pays to resend the whole history.
• Summarise. Older turns are replaced by a summary. This is the right idea, but a vague summary ("the user discussed their project") loses exactly the details you will ask about later.

How Nivana keeps a long chat

Short chats go to the model exactly as you wrote them. Once a conversation passes about 80,000 tokens, Nivana keeps the most recent part of it (about 24,000 tokens, and never fewer than the last four messages) word for word, and folds everything older into a running summary that the answering model reads before your latest message.

• A ledger, not a recap. The summary is organised into sections and is written to keep the things people come back for: names, numbers, decisions, preferences and open questions.
• Prepared ahead of time. When a chat gets close to the limit, the summary is built in the background while an answer streams, so crossing the line does not add a pause.
• Stable between turns. The point where old turns become summary moves in steps of a few messages rather than every turn. That keeps the start of the prompt the same from turn to turn, which lets providers reuse their prompt cache and keeps long chats cheaper.
• Files are remembered too. Once an attached file is old enough that resending it is wasteful, the chat keeps a short description of it instead, so the model still knows what you shared.

Older turns land at the start of the prompt in a compact form instead of piling up in the middle. Recent turns stay word for word at the end, where the research says models read best.

Getting the most out of a long chat

• Restate what matters when you change direction. One line like "keep the budget at $5,000" puts it at the end of the context, where it is read most reliably.
• Start a new chat for a new topic. A fresh chat is cheaper and keeps unrelated history from crowding out what you need.
• Use folders, not one endless chat. Related chats can sit together without sharing a context window.