Cover - Why AI Agents Drift: Belief State Is the Real Bottleneck

In short: Many AI agents look productive but are actually drifting — confidently executing the wrong moves on a wrong picture of the situation. The bottleneck for the next phase of agent systems is not larger context windows or stronger base models; it is whether the system can construct and maintain a stable belief state. This piece argues why belief state quality is the right optimization target, proposes five proxy metrics to measure it, and lays out where to put incremental engineering resources next.

AI agents that look productive often turn out to be drifting — confidently executing the wrong moves on a wrong picture of the situation. Competition in agent systems is shifting from “whose model is stronger” toward “who can keep producing higher-quality belief state.” If you accept that framing, several seemingly unrelated problems suddenly line up: the same model behaves very differently inside different product shells; long-running agents fail not because they cannot answer but because their judgment of the situation is wrong; context windows keep growing, but system capability does not scale linearly with them; and scattered engineering pieces — skill, memory, retrieval, tool use, trace, summary — all start to matter at the same time.

These pieces serve the same goal: helping the system understand the current situation more reliably, rather than simply “seeing more.” That is what I take “the second half is about context” to mean. To put it more precisely, the relevant context here is not the prompt and not the chat history. It is the external cognitive infrastructure the system builds around the chain context → belief state → action.

What is belief state?

Belief state is the system’s internal estimate of the current situation when information is incomplete. It is not the world itself, but the system’s judgment of the world at this moment. When the system cannot see the full picture, it combines current observations, past observations, executed actions, experience, and prior knowledge into something like “here is what I think is going on.” That synthesis is the belief state. The rest of this article argues that the quality of this synthesis — not the size of the context window or the strength of the base model — is what will decide whether an agent system stays useful as tasks get longer and noisier.

1. From chatbot to agent: the real shift is maintaining state

Treating an agent as “an LLM that can call tools” leaves a lot unexplained. Tool calling is only the surface. The harder questions sit underneath: when to search, what to keep from a search, which results are noise, which history to compress, what to remember long-term, which hypothesis to keep pursuing. None of these belong to language generation. They look much more like what a cognitive system does — perception, memory, state estimation, action selection, feedback correction.

A modern agent is a cognitive system that pushes working memory, retrieval, rules, tools, and execution feedback outside the model. That matters because the model no longer has to carry the whole burden of “understanding the world.” The system around it takes on more and more of the state-management work. Agent intelligence is no longer purely intrinsic to the parameters. A growing share of it comes from how the surrounding system organizes context and state.

The familiar distinction between chatbot and agent is “a chatbot only talks, an agent acts.” That is true but shallow. The deeper distinction: a chatbot optimizes single-turn semantic mapping; an agent has to stay stable across multi-step state evolution. The chatbot’s core problem is expression. The agent’s core problem is state.

In the chatbot phase, the system works around a conversation: user input, message history, system prompt, and a reply. Context is short-lived, conversational, and implicit. In the agent phase, the system starts handling tasks like searching code, looking up docs, calling APIs, running commands, editing files, and continuing based on the results. What it has to maintain is no longer “what was said last turn” but the current goal, known facts, failed paths, current hypotheses, the next thing to verify, and the active risk constraints.

The line between chatbot and agent is whether the system explicitly maintains task state. Once task state becomes central, context stops being chat history and becomes external working memory for the task. That is what makes “context” the truly new variable in the agent era.

2. POMDP and belief state: what the agent is actually working with

POMDP — partial observable Markov decision process — is a useful frame for agents. Anyone who has played poker or worked on RL will recognize the lineage from Markov chains. The point of POMDP is not the formula. It is a very plain fact: you never see the full world. You can only guess the current situation from local evidence, take an action, and correct based on the feedback.

That is roughly how an agent operates day-to-day. A user asks you to “fix the login bug” and never mentions that the rate limiter is involved. The 500 in the log is from the gateway, but the real fault is a token expiry two layers deeper. The snippet from auth.ts looks self-contained, even though the same logic is copy-pasted in three other files. The snapshot you scrape from a checkout page does not include the JavaScript state. The agent works from these scraps — user messages, file fragments, tool results, API returns, logs, errors — and has to extrapolate the rest. The “here is what I think is going on” judgment it forms from them is the belief state.

Mapping a modern agent onto POMDP, the update chain looks roughly like this:

o_t     = user input / file fragment / log / tool result
b_t     = U(b_{t-1}, o_t)
a_t     = π(b_t)
s_{t+1} ~ T(s_t, a_t)

In plain words: the system receives a new local observation o_t, updates its current estimate of the situation to b_t, picks an action a_t based on that estimate, and the action moves the environment into the next state s_{t+1}.

The two middle equations carry the most weight. b_t = U(b_{t-1}, o_t) says belief state does not appear from nowhere — it comes from “last round’s judgment” combined with “this round’s new observation.” In today’s agent systems, U is not a single module. It is the combination of retrieval, summarization, planning, structured extraction, memory recall, and prompt orchestration. What context engineering is really doing, at bottom, is implementing this U. a_t = π(b_t) says the next action — answering, searching, reading a file, calling an API, applying a change — is chosen based on the current belief state, not on the raw input. If b_t is built crooked, even a strong policy π is just executing efficient actions on a wrong picture.

Belief state sounds like a theoretical term, but in practice it is concrete. An agent’s current belief state usually shows up in the active task description, confirmed facts, current hypotheses with priorities, intermediate tool results, stage summaries, the next plan, and the active risk and constraint information. It is not some hidden quantity inside the model’s head. In real systems it is just the “task-situation judgment” that has been organized into the current context.

The important boundary: memory is not belief state. Context is not belief state. World model is not belief state. They are ingredients or carriers for belief state, not the thing itself. A useful chain to keep in mind:

world model + observations + memory → belief state → context → action

Action is driven not by memory, prompt, or rules directly. It is driven by the situation judgment the system synthesizes from all of them. A common mistake in agent engineering today: teams optimize memory, retrieval, prompts, and tool use, but never set “belief state quality” as the explicit objective. The result is many modules pulling in inconsistent directions.

3. The second half is about belief state quality

“The second half is about context” stops being useful once it stays at slogan level. The next step: context matters because it is the main carrier of belief state.

If you treat context as “model input,” optimization tends to mean writing prettier prompts, writing more comprehensive rules, retrieving more material, or stuffing more history into a larger window. If you treat context as the carrier of belief state, the questions change. Now you ask: is the current belief correct, has it been polluted by noise, which pieces of information are helping update belief and which are dragging it down, has the system turned memory, retrieval, and tool results into a stable situation judgment, and can the system catch a wrong belief and correct it in time. The thing to optimize is belief state quality, not context capacity.

Five proxy metrics for belief state quality

Measuring belief state quality is the hardest part of the framework. There is no agreed-upon metric today. I break it down into five observable proxy metrics that can at least give you a working dashboard:

  • Key fact recall: tag N task-relevant key facts in advance for a multi-step task, and track at step k whether they are still hit by the system’s context or explicit state. If the dropout rate crosses a threshold, trigger a lookup.
  • Rollback count: how many times an agent explicitly corrects course inside a single task. Too low means the system is committing to a wrong path silently. Too high means belief is unstable.
  • Trajectory consistency: rerun the same task multiple times (same seed, same input) and measure divergence in the action sequence. Large divergence means belief is unstable.
  • Marginal information gain per tool call: the change in belief description before and after each tool call. A tool that consistently produces near-zero gain is a noise source and should be retired or gated.
  • Human agreement on key decisions: sample the agent’s key actions and have humans rate whether each action was reasonable given the belief state at that moment. The most expensive and most accurate metric.

None of the five is perfect on its own. Together they explain more than “final task success rate.” When a task fails, you can at least separate “the belief was wrong” from “the belief was right but the policy did not follow through.” The most important evaluation axis for agent systems going forward will be whether the system can keep belief state from drifting under longer task chains, more complex environments, and higher noise.

Recent academic work has started to operationalize this kind of measurement. ABBEL takes the first metric one step further by parsing each generated belief into a structured form and comparing it directly against the ground-truth posterior — turning belief quality into a usable training reward, not just a dashboard signal.

Why content creators and engineers were the first to see real gains

Real-world reactions to AI over the past two years have not been evenly distributed. The earliest, clearest, and most sustained productivity gains have come to content creators and software engineers. The shallow explanation is that the model happens to be good at writing and coding. The deeper explanation: the work context for these two roles is more easily digitized, and their belief state is easier to construct outside the model.

Their work context already lives in a digital environment. Content creators work around topics, source material, prior articles, voice and style, platform constraints, headline and cover strategy, and reader feedback. Engineers work around code, logs, errors, docs, git history, test results, configuration, and tool output. All of this is text-shaped, structured or semi-structured, searchable, and reviewable. These roles are not simpler — they are easier for a system to fully observe.

Their belief state is easier to support from outside. Having context is not enough. The system also has to be able to construct a reasonable belief state from it. Many other roles can use AI but feel less impact because the critical information lives in tacit human understanding, off-line judgment, in-the-moment negotiation, unstructured experience, implicit organizational constraints, and scene details that are hard to record. Belief state for these tasks is much harder for a system to get right. Content creators and engineers also have tacit knowledge, but most of the critical signals are visible, storable, and reviewable. The system can therefore get enough useful observations, compress history into usable state, refine its judgment as the task progresses, and pull up original evidence when needed.

When people debate “why some roles embrace AI and others do not,” the question is often framed wrong. The real question is not whether the model is smart enough for that role or whether the role is willing to use AI. It is whether the role’s critical work context is exposed to the digital system fully enough for AI to construct a high-quality belief state. If yes, AI can land real productivity early. If no, AI tends to stay at the assistive, point-solution layer and rarely takes over a workflow.

Whether a role becomes deeply AI-enabled in the future will depend less on how much further base models improve and more on whether that role’s context can be digitized and structured well enough for the system to keep translating it into high-quality belief state.

4. Common agent problems, seen through the belief state lens

Retrieval really improves observation conditions, not knowledge

RAG is often described as “plugging an external knowledge base into the model.” That is fine for an introduction but no longer enough for agents. In an agent setting, retrieval has a different role: when the current observation is insufficient, retrieval actively improves the system’s perception of the situation.

If you treat retrieval as knowledge completion, you focus on whether the knowledge base is comprehensive, whether the embeddings are strong, and how much you recall. If you treat retrieval as observation augmentation, the focus shifts to which evidence the current task is most missing, which observations would shrink the problem space the most, which retrieved results would pollute current judgment, and which evidence should go directly into context versus indexed-only. Retrieval looks more like an active observation-sampling module than a static knowledge entry point. The system is not retrieving to “know more.” It is retrieving to “guess more accurately.”

Context Rot is a belief pollution problem

Context Rot is an important concept because it punctures a common assumption: longer context means a stronger system. People expect that a larger window will get them closer to “remembering everything.” Real systems show the opposite. As context grows in long tasks, you see accuracy drop, key information get lost, noise interfere more, reasoning chains break, historical stages blur, and complex tasks degrade first while simple ones still work.

Reading this only as “the attention mechanism is not strong enough” is too shallow. I read Context Rot as a system-state problem: in long tasks, the system gradually loses a clear judgment of the current situation, and belief state is slowly polluted by noise, redundancy, and historical residue. That reading lifts Context Rot from a model-window problem to a system-state problem. This is also why the discussion has shifted toward harness engineering — keeping an agent on track at the 50th or 100th step is not a prompting problem, it is a state-management problem.

Once you see it that way, the solution direction changes. You stop asking “how do I cram in more tokens” and start asking how to do stage summaries, how to offload large tool outputs, how to convert raw text into state instead of stacking it, how to move history from context into memory, how to retrieve original evidence when needed, and how to detect belief drift. The core problem of context systems was never capacity. It is purity. Future competition in context systems will be about who can best prevent belief from being corrupted.

Why AI agents drift even when they look productive

Many agent failures are not about being unable to act. The agents are usually quite active — they search, read, call tools, write plans, and explain. The trouble is that they often keep acting on a belief that is already wrong. That is the most dangerous failure mode today: getting more and more efficient at executing actions on a wrong picture of the situation.

Several familiar symptoms become easier to read once you see them this way.

“Forgetting” is usually not failed memory. It is current belief missing key support. The information fell out of context, the history summary lost the relevant detail, memory recall missed, or the system never triggered a lookup. It looks like amnesia, but the actual problem is that the current belief is missing the evidence it needs.

“Drifting further off” is usually wrong belief driving wrong exploration. The root cause is in a backend session, but the system early on misread it as a frontend routing problem. Subsequent searching, reading, and verification all happen inside that wrong subspace. This is wrong belief allocating wrong actions, not a pure reasoning failure. There is a clean experimental version of this in the ABBEL paper: DeepSeek R1 playing Mastermind hallucinated a previous guess that never happened during a belief update, locked the wrong digit into its belief, and spent the rest of the trajectory exploring around that fabricated constraint.

“Lots of tools but mediocre results” means the tools are not folded into the belief update loop. Tools are observation channels and action channels. If the system cannot pick the right moment to call them, interpret the results correctly, compress the result into the current state judgment, or stop the result from polluting context, then more tools just create more belief noise.

Plan and reflection help because they explicitly tidy up the belief state. They force the system to keep answering four things: what is known, what is unknown, which hypotheses have been falsified, what to verify next. Reading them as “the agent thinking more like a human” misses the point — they are a belief-cleanup pass.

5. The next twelve months: five lines of evolution and the Context OS

None of these directions are mine first. MemGPT and Letta are working on tiered memory. Anthropic’s Skill system already treats skills as capability units. LangGraph’s checkpoint and state make state explicit. Cognition’s Devin trace gave a first picture of observability. LangSmith and Langfuse are doing LLM-call-level tracing. What follows is not “we should pursue these five directions.” It is the part of each direction that I think most deserves heavier investment over the next twelve months, plus where it differs from current solutions.

5.1 Memory moves from unified storage to a tiered architecture

A single storage shape will not solve long-term memory. A more realistic layering:

  • L0 In-Context: hot state inside the current window.
  • L1 Session Cache: short-term memory and stage conclusions across turns.
  • L2 Episodic Store: past episodes that can be retrieved semantically.
  • L3 Semantic Store: structured knowledge such as preferences, facts, and entity relations.
  • L4 Procedural Store: procedural memory like skills, rules, and behavioral templates.

MemGPT and Letta have early versions of L0 / L1 / L2. L3 has been heavily discussed in the RAG community. The unsolved layer is L4 — how procedural memory coordinates with current belief to decide whether to fire. Most implementations today just stuff every skill description into the system prompt and let the model pick. The real memory question is no longer “did we save it.” It is which kind of information should be saved at which timescale and in which structural form.

5.2 Memory moves from natural-language summaries to structured state assets

High-value memory in the future is less likely to be natural-language bullets and more likely to be structured state records carrying entities, relations, timestamps, sources, and confidence. The hard part is not “writing it down.” It is whether the same memory is reliably retrievable later, whether sources will conflict, whether semantic drift will appear over time, and whether belief updates have a basis for prioritization. Letta’s block-based memory is moving in this direction, but confidence and time decay are mostly missing, and cross-source conflicts are still left to the model to improvise.

5.3 Skills move from prompt documents to a capability registry

Skills are no longer just instruction documents. They are becoming registered, schedulable, traceable capability units. A skill needs more than a natural-language description — it needs explicit capability declarations: trigger conditions, required context, output contract, cost estimate, fallback strategy. Anthropic’s Skill system is moving in this direction, but trigger conditions still mostly depend on the model reading the description and deciding for itself. A capability registry is still missing one piece: a runtime that picks skills based on current belief and a cost budget. At root, this turns skills from text assets into system capability assets.

5.4 Context moves from pre-injection to just-in-time loading

This trend looks very solid to me. The old approach is to stuff in as much information as possible at task start. The more reasonable approach is to surface summaries and indexes first, let the agent decide based on current belief whether to load detailed context, and shift control of context selection from hard-coded system rules to dynamic agent decisions. Cursor and Claude Code already use read-on-demand strategies. The next bottleneck is letting the agent estimate the marginal benefit of loading a given chunk of context. Most of this is heuristic today — file names, summary similarity — and explicit information-gain evaluation is missing. Context shifting from static input to dynamic assembly is a paradigm shift, not a small optimization.

5.5 Observability becomes infrastructure, not just a debugging aid

If belief state really becomes the central object of the system, debugging an agent has to change. You no longer just check whether the final answer is correct. You also need to see which parts make up the current context, why a skill was loaded, when memory was read and written, whether belief drifted across turns, and which tool output started polluting the current judgment. LangSmith, Langfuse, and Helicone already provide tracing, but most of it is still at the LLM-call level. Belief-level observability is largely absent — nobody draws you a timeline showing “belief drifted at step 7 because of this tool result.” Without this kind of observability there is no reliable agent engineering. Once context becomes invisible and belief becomes invisible, the system is back to a black box.

The real watershed: who looks more like a Context OS

Compressing the five directions further: the watershed for future agents probably will not be who writes more like a person, who answers more fluently, who has a longer window, or who hooks up more tools. The more likely watershed is who looks more like a stable Context OS.

This framing is not new. The Operating System Moment of AI Agents already argued that context windows behave like volatile memory and that “memory management — context engineering — is both the hardest problem and the biggest opportunity.” What I am adding here is the choice of optimization target: a Context OS does not just need a better memory hierarchy. It needs an explicit objective — belief state quality — that everything else (memory, retrieval, JIT loading, observability) gets evaluated against.

In a real product Context OS looks like this: a customer-support agent recalls that this customer hit the same Redis key collision last month and fixes it the same way, pulls only the few related tickets into context instead of the whole history, and on day three of a long thread notices it has started answering questions about the wrong product line and pulls itself back. Phrased as a capability checklist, the system can keep gathering high-value observations, maintain tiered memory, assemble context dynamically, isolate noise and redundancy, detect belief drift, roll back and correct from wrong states, and turn long-term history into reusable state assets.

“The second half is about context” points to a new division of labor in the system, not an extension of prompt engineering. In this division of labor, the model handles local reasoning and action generation, retrieval supplements observations, memory handles cross-time storage, context handles current expression, belief state handles current judgment, and the agent runtime stitches all of this into a sustainable execution loop. The competitive focus shifts from “how smart is the model” to “can the system keep from drifting over time.”

6. Where to put incremental resources: a more realistic strategic call

Pushing the analysis one step further leads to a more concrete and more carefully phrased strategic claim. The claim is not “models do not matter,” and certainly not “you can skip base models.” More precisely: for some time to come, the optimal allocation of incremental resources is probably not to keep doubling down on base models, but to keep model capability current while investing earlier and more heavily in each business’s context engineering and belief state infrastructure. The emphasis on “incremental” rather than “all” matters because this is not an either/or choice.

This is also where the argument connects to a thesis I made elsewhere. In Why OpenClaw Won: The Real Battle in AI Products Is the Environment, Not the Model, I argued that what drives value in AI products has less to do with raw model strength and more to do with the environment the product lives in. Belief state quality is the technical realization of that environment thesis: the environment matters because it is what the system uses to construct and maintain its judgment.

Base models are important. Without a strong enough ceiling, complex capabilities are off the table. But the main bottleneck for many business scenarios today is no longer raw model answer quality. It is the inability to obtain enough useful context, the inability to compress history into high-quality state, the inability to combine retrieval, memory, and tool results into a stable belief state, persistent context rot polluting judgment in long tasks, and the lack of unified context infrastructure across business lines. Pouring more energy onto the model side under those conditions does not give the highest marginal return.

The counterargument worth answering directly: base models set the ceiling

This counterargument is valid and has to be acknowledged. If long-context reasoning in the model is unstable, no amount of well-designed JIT loading will save you. If tool-calling accuracy is too low, observability tells you where things went wrong but cannot save the task itself. If the model uses structured memory recall poorly, even an elegant tiered memory design will not perform as expected. So “investing primarily in context engineering” does not mean “abandon the base model.”

But ceiling and bottleneck are different. The bottleneck in most business scenarios today is nowhere near the frontier model ceiling. GPT-5, Claude 4.6, and Gemini 3 are good enough at single-step reasoning, tool calling, and long-context handling. What actually blocks businesses is context organization, state maintenance, and belief correction. Even one more generation of frontier model will not automatically solve those if the engineering work is missing.

A more realistic resource allocation:

  • Keep the base model current. If you can train your own, stay close to frontier-model pacing. If you are already far behind, just use frontier models and redirect the saved compute into context-asset accumulation in your business.
  • Reinvest the increment into context engineering. This is the part where most companies can see ROI inside twelve months.

There is now direct empirical support for this allocation. The ABBEL paper trains a Qwen2.5-7B model with reinforcement learning specifically on belief construction, and shows it outperforms a Qwen2.5-14B model running with full context on a 16-objective question-answering task — by roughly 6× on Exact Match score while using a quarter of the memory. The capability gap came from training on belief quality, not from a larger model. That is the resource allocation argument made concrete.

Model investment has a few inherent characteristics: heavy spend, long cycle time, partial controllability, and a gap to external frontier models that is not easy to close quickly. Context engineering and belief state infrastructure tend to have a different profile: closer to specific business problems, easier to form a clear quality loop around, easier to observe real ROI on, and reusable across multiple business scenarios once accumulated.

This is the same shift in scarcity I discussed in When Execution Becomes Infrastructure, Judgment Becomes the Scarce Resource. Once execution becomes commodity, the bottleneck moves to judgment. By the same logic, once frontier model access becomes commodity, the bottleneck moves to whether your system can construct a high-quality belief state on top of it.

Context systems also have first-mover advantage and network effects

Model capability has scale effects too, but a working context system tends to create three kinds of moats inside specific business scenarios:

  • First-mover advantage: whoever first accumulates the high-value context assets in a scenario sets the entry barrier early.
  • Network effects: the more users use the system, the richer the memory, rules, workflows, and state assets become, and the better subsequent experience gets.
  • Stickiness: once a user’s task history, preferences, collaboration relationships, and knowledge assets are inside the system, switching cost rises noticeably.

Model capability acts as a general-purpose substrate. A context system embeds directly into specific business processes and becomes part of the business itself. Whoever does context engineering first does not just get “better current experience” — they get a real shot at converting that experience advantage into data assets, memory assets, workflow assets, user stickiness, and long-term network effects. A context system is not just an engineering optimization. It can itself be part of the business moat.

Closing

Compressed into one sentence: the next phase of AI agents is not won by longer context, it is won by more stable belief state.

The sentence shifts the center of discussion from “what the model can do” to “whether the system can keep from drifting over time.” Context is the carrier, not the goal. What needs to be maintained is the quality of the system’s judgment about the current situation. Whoever can construct, maintain, and correct that judgment more stably is more likely to ship the next generation of usable agents. Whoever cannot turn it into engineering capability will see their model advantage dissipate quickly inside real business.

The core engineering question for agent systems going forward: how do you turn belief state construction into infrastructure that is explicit, observable, and able to keep evolving with the business it serves.