How language models, context, and memory fit together

Start with the layers, not the magic

“AI” is often used for a stack of different things. A language model predicts a continuation from the tokens it receives. A chat product wraps that model in instructions, file handling, safety rules, interface choices, and sometimes search or tools. An application may add retrieval, databases, permissions, memory, routing, and monitoring. A result can fail in any one of those layers.

The transformer architecture described in Attention Is All You Need is a foundation for modern language models. That does not mean a model consults a hidden encyclopedia when it answers. Its learned parameters shape likely continuations; they do not provide a reliable, inspectable record for every factual claim.

Tokens are the working units

Models process tokens rather than pages or ideas. A token can be a word, part of a word, punctuation, or another text fragment. Tokenization varies by model family, so character counts are not exact token counts. Long inputs consume space that could otherwise hold instructions, retrieved evidence, tool results, and the answer.

A context window is best treated as working space, not permanent memory. Material outside it is unavailable unless the application retrieves or summarizes it. Material inside it can still be overlooked, especially when the prompt is crowded, repetitive, contradictory, or poorly structured. “It was somewhere in the conversation” is not an evidence policy.

Memory is an application decision

When a product appears to remember, it may be storing a profile, a conversation summary, selected facts, embeddings, or previous messages and then placing some of that material into a later prompt. Those choices determine what is retained, who can access it, how it is corrected, and when it is deleted. The model itself is not a substitute for a governed record system.

For consequential state—customer preferences, approvals, account balances, publication status—keep the authoritative value in a database or other deterministic system. The model may help interpret a request, but code should validate the operation and the system of record should preserve the result.

Retrieval changes the evidence available

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks described systems that combine parametric and non-parametric memory. In practical applications, retrieval usually means finding passages from an approved corpus and placing them in context before generation.

Retrieval can improve freshness and traceability, but it does not guarantee either. The corpus may be stale, permissions may be wrong, the search may select an irrelevant passage, or the answer may stretch beyond what the passage supports. Evaluate retrieval and generation separately: did the system find the right evidence, and did the answer represent it faithfully?

Tools should do deterministic work

Use calculators for arithmetic, databases for records, search for discovery, and code for rules that must be repeatable. Tool access expands capability and risk at the same time. A read-only search tool is different from a tool that can publish, purchase, delete, or email. Give each tool the narrowest permission needed, validate arguments in code, and require confirmation for consequential actions.

The practical question is not “What can the model do?” It is “Which layer should own this part of the job, and how will we know it worked?”

Field questions

• Which facts must come from a current or private source rather than model parameters?
• Which result must be calculated or validated by code?
• What is the authoritative system of record?
• What material is allowed into context, and who can retrieve it?
• What can the tool change, and how can the action be reversed?
• Which logs are needed to reproduce a failure without retaining unnecessary sensitive data?