"Memory" gets used loosely enough in agent discussions that it's worth pulling apart into what it actually refers to, because short-term working memory and long-term memory across sessions are genuinely different engineering problems with different solutions - and a lot of confusing agent behavior traces back to one being used where the other was needed.
Short-term memory: state within a single run
Short-term memory is everything the agent has seen so far in the current task - the conversation, the tool calls it's made, the results it's observed. This lives entirely within the model's context window during that one run, and it's what makes the plan-act-observe loop coherent: the model can only "loop back to plan again" sensibly because it still has the observation from the previous step in front of it. Once the run ends, this state is gone unless something explicitly saves it - which is exactly what long-term memory is for.
Long-term memory: state across runs
Long-term memory is information that needs to persist beyond a single run - a user's stated preferences, facts learned in a previous conversation, a record of past actions taken. This has to be stored somewhere outside the model's context window (a database, a file, a vector store) and explicitly retrieved and re-inserted into context at the start of a new run. If you've been through this site's RAG 101 series, this retrieval step is literally a RAG pattern applied to an agent's own memory instead of a document corpus.
Short-term memory disappears at the end of a run. Long-term memory has to be stored externally and deliberately pulled back in.
Why conflating the two causes weird behavior
A very common source of confusion: expecting an agent to "remember" something from a previous session when nothing was ever written to long-term storage - the previous run's context window is simply gone, and there's no mechanism by which information crosses that boundary unless someone built one. The fix is never "make the model remember better"; it's building the explicit save-and-retrieve step, because the model has no persistence of its own between separate runs.
What "state" adds on top of memory
Beyond conversational memory, many agents also need to track structured state - which step of a multi-step process it's on, what values it's already collected, what's still pending. Frameworks like LangGraph represent this explicitly as a state object that gets updated at every step of the graph, which is a more structured version of the same idea: some information needs to persist and evolve across the loop's iterations, separate from the raw conversation history.
The cost of memory that's too long
More context isn't free - a context window stuffed with the entire history of a long-running agent gets slower, more expensive per call, and can actually degrade the model's reasoning by burying relevant recent information under a pile of older, less relevant context. Well-designed agents actively manage this: summarizing older parts of the conversation, only retrieving the specific long-term memories relevant to the current task rather than dumping everything back in, and periodically trimming what's carried forward.