A large context window is not memory. Here is what belongs in context for one task, what belongs in persistent memory instead, and how to curate both on purpose.
A context window of a million tokens can hold your entire codebase, every open file, and the full back-and-forth of a session, and none of it survives past that session. Context is what a model can see right now. Memory is what it can recall later, on its own, without you pasting it back in. Conflating the two is one of the most common setup mistakes with AI coding assistants: teams treat a large window as a substitute for a memory system, then wonder why the assistant re-derives the same architectural decision a different way every few weeks.
The cost of that confusion is not just annoyance. Every time an assistant has to re-derive why a service is structured a certain way, or guess at a naming convention instead of being told, it spends tokens on rediscovery instead of the actual task, and it has a real chance of guessing wrong. A wrong guess that reads as confident is how drift starts: one change follows a different pattern than the last, nobody catches it in review, and months later the codebase has three ways of doing the same thing. Larger windows make this worse in one specific way: they make it cheaper to skip curation, so more unrelated material ends up in front of the model by default, not less. This post covers what belongs in the context you hand an assistant for a single task, what belongs in a persistent memory layer instead, and why the difference matters more as a project grows.
What actually fills a context window
A context window is a token budget, and everything in a request competes for the same space: the system prompt, the files the assistant has read, the output of any commands it has run, retrieved documentation, and the conversation so far. None of it is free and none of it is permanent. Close the session and the window empties. Open a new one and the assistant starts from whatever you, or your tooling, decide to put back in. Nothing about the mechanism favors older, more important facts over newer, less important ones; a decision from three months ago and a typo fixed five minutes ago carry equal weight unless something outside the window tells the model otherwise.
- Open files and diffs the assistant is actively editing
- Tool output: test results, build logs, linter warnings from this session
- Retrieved snippets: documentation, prior code search results, related functions
- Conversation history: every prior turn in the current chat
- Whatever instructions or spec you pasted in for this specific task
Context is not memory
Context is meant to be temporary. It exists to give a model what it needs to finish the task in front of it, then disappear, the same way working memory clears once a person finishes one task and moves to the next. Persistent memory is built for the opposite job: it holds facts that stay true across many sessions and many months, things like why a table is denormalized on purpose, which library the team standardized on, or what actually broke last time someone touched the payment webhook. A bigger window makes the first kind of storage roomier. It does nothing for the second kind, because a window that resets between sessions was never built to keep anything.
A million-token window and a zero-token memory system produce the exact same amnesia the moment you close the chat. Size solves a different problem than persistence does.
What belongs in context for a given task
Context should be curated per task, not maximized. The instinct to paste in as much of the repository as the window can technically hold works against you: relevant instructions get diluted by irrelevant code, and recall quality inside a long context degrades well before the token limit is reached, especially for details buried in the middle of a long prompt. Curate context the way you would brief a contractor on a single ticket: give them exactly what this task needs and nothing that belongs to a different one.
- The specific files the task touches, not the whole module
- The spec or ticket describing what "done" looks like
- The relevant slice of your conventions, not the entire style guide
- Output from the last failing test or the last review pass, if this is a fix
- Directly related code the change has to stay consistent with
What belongs in persistent memory instead
Persistent memory is for facts that hold regardless of which task is active this week. These are decisions and constraints an assistant should already know before it opens a single file, not things it has to rediscover by reading the whole codebase again. Keeping them out of the window by default and loading them only when relevant is what keeps a session lean while still starting from knowledge instead of guesswork. Think of it as the difference between a résumé and a diary: memory should read like a résumé, a short list of durable facts you would tell a new hire on day one, not a diary of everything that has ever happened in the repository.
- Architecture decisions and the reasoning behind them
- Naming, style, and structural conventions for the project
- Known gotchas, deprecated paths, and tech debt to route around
- Past incidents and how they were actually fixed
- Domain vocabulary and business rules that are not obvious from the code
Two different failure modes
Getting this wrong produces two distinct problems, not one. Too much undifferentiated context inside a single session causes drift: the assistant loses track of what matters, mixes conventions from unrelated files it happened to read, and produces a diff that runs but does not match how the rest of the codebase is built. Too little persistent memory across sessions causes repetition: the same architectural questions get re-asked and re-answered differently each time, decisions quietly diverge, and nobody notices until two parts of the system disagree with each other. This is also why review quality depends on curated context: a multi-agent review given a clean, relevant diff finds real issues, while one buried in noise wastes its attention on files that were never part of the task.
Before a task starts, ask what the assistant would need to know that no file in the diff would tell it. That gap is usually exactly what belongs in memory, not in the prompt.
A persistent memory layer, not a bigger window
The fix for the cross-session problem is not a larger context window; it is a separate, durable place to keep the facts that outlive any one session. A private, persistent memory layer like MemX is built for exactly this: it stores the project facts, conventions, and decisions you choose to keep, keeps them private and portable, and makes them reloadable across sessions instead of re-derived from scratch every time. That is a different kind of tool than a longer window. A window is rented space for the current task. A memory layer is somewhere you actually own the contents of.
| Dimension | Raw context window | Persistent memory layer |
|---|---|---|
| Lifespan | Cleared when the session ends | Persists across sessions |
| What it holds | Whatever is pasted or read this turn | Curated facts you choose to keep |
| Retrieval | Manual: you re-paste it each time | Reloadable on demand |
| Cost as the project grows | Same background re-explained every session | Stored once, loaded when relevant |
| Best suited for | This task's files, spec, and recent output | Architecture, conventions, and past decisions |
Curate both, on purpose
A spec audit is one of the more effective places to enforce this discipline, because it forces the task's context to be written down before any code exists: what this change needs, and what it can assume is already known. TLM Forge runs that audit before implementation starts, so the context handed to the coding step is curated rather than dumped in wholesale. Read more on how a tight spec keeps context small for a single task in spec-driven development, and treat your project's persistent memory as the set of facts a spec is allowed to assume rather than something it has to re-explain every time.
Frequently asked questions
01Is a longer context window the same thing as memory?
No. A longer window gives a single session more room to hold files and conversation, but it still resets when the session ends. Memory is separate storage that persists between sessions and gets reloaded on request, regardless of how large or small the window is.
02How much should I put in an assistant's context for a single task?
Only what that task needs: the files it touches, the spec describing what done looks like, and any directly related code it has to stay consistent with. Anything true across every task, like architecture decisions or conventions, belongs in persistent memory instead, referenced rather than pasted in full.
03Does a memory layer replace code review?
No. A memory layer keeps an assistant from forgetting your project's facts; it does not verify that a given change is correct. That is still a job for tests, an independent reviewer, and a review gate before anything merges.