Agent Architecture in Partially Observable Stochastic Environments (POMDPs)

An agent’s architecture determines (i) what internal information it keeps, (ii) how it updates that information from observations, and (iii) how it uses the internal state to choose actions under uncertainty. In partially observable and stochastic environments, the key architectural distinction is whether the agent represents uncertainty explicitly and maintains memory across time.

A standard formalization is the POMDP model. In a POMDP, the agent does not observe the true environment state $s_t$ directly; it receives observations $o_t$ that are generated stochastically from $s_t$, and the environment transitions stochastically from one state to the next. Therefore, optimal decision-making depends on the agent maintaining a distribution over possible states—typically called a belief state.

Architectural implications:

• Memory & state estimation: Architectures with no memory (purely reactive) cannot, in general, recover hidden state information; they act from $o_t$ only, which is typically insufficient in POMDPs. Architectures with explicit belief estimation (e.g., Bayes filter / particle filter) can.
• Inference module: An “inference” component (filtering) computes/upgrades the belief state using the observation model and transition dynamics.
• Decision module: A policy can be conditioned on belief state rather than on raw observations.
• Planning horizon & computation: Some architectures approximate the belief-state optimal control problem with limited lookahead, while others compute exact/near-exact solutions.

Below is a compact view of the information flow that architecture choices control.

1) What “architecture” changes, formally

In a POMDP, an agent’s quality depends on replacing the unobservable $s_t$ with an internal representation that is computable from history. Architecture controls:

1. Belief update mechanism

   • If the agent has access to $P(o_t\\mid s_t)$ and $P(s_t\\mid s_{t-1}, a_{t-1})$, it can perform a Bayes filter: $b_t(s) \\propto P(o_t\\mid s)\\sum_{s'} P(s\\mid s', a_{t-1})\\,b_{t-1}(s')$
   • If it does not have models, it must learn approximate updates or use a learned memory module.

2. State representation

   • Explicit belief state: architecture maintains $b_t$ as a distribution (tabular, factorized, particle-based).
   • Implicit belief: architecture uses recurrence (e.g., RNN/LSTM) so internal hidden state acts as a learned summary of the history.

3. Policy conditioning

   • Belief-conditioned policy: $\\,\\pi(a_t\\mid b_t)$
   • Observation-conditioned policy (reactive): $\\,\\pi(a_t\\mid o_t)$

The critical limitation: $\\,\\pi(a_t\\mid o_t)$ generally cannot represent the information lost by not tracking uncertainty over hidden states. In contrast, belief-conditioned policies can act differently depending on how likely each hidden state is under the same observation.

Key terms: POMDP, belief state, Bayes filter, observation model.

2) Common architectural patterns and their consequences

A. Purely reactive architecture (no memory)

A typical reactive agent does:

• perception: $o_t \\rightarrow$
• decision: $a_t = \\pi(o_t)$

Effect in POMDPs: two different hidden states may produce the same observation distribution, and the reactive policy cannot disambiguate them over time. Stochasticity worsens this because the same observation may correspond to different histories.

B. Reactive-with-memory (implicit belief via recurrence)

Modern agents often use recurrent networks:

• update internal state: $h_t = f(h_{t-1}, o_t, a_{t-1})$
• act: $a_t = \\pi(h_t)$

Effect: the architecture can learn to approximate a belief state (or useful statistics) even without explicit transition/observation models. Performance depends heavily on capacity, training signal, and whether the recurrence captures sufficient history.

C. Deliberative architecture (explicit inference + planning)

A classic structured approach:

• belief update (Bayes filter / particle filter)
• planner/policy uses $b_t$ (sometimes with lookahead)

Effect: can be theoretically aligned with POMDP optimal control when models are available and computation is adequate. When models are approximate or planning is truncated, it becomes an informed approximation.

Practical Example: Robot navigation with localization uncertainty

Scenario: A robot moves in a 1D corridor with walls (hidden exact position). It moves stochastically (motor noise) and senses obstacles with uncertainty (sensor noise). The robot only receives noisy distance/observation readings, not the true position.

Architectural choices

1. Reactive controller (no memory):

   • Input: current noisy reading $o_t$
   • Output: action $a_t$
   • Failure mode: the same reading can occur at multiple positions; because movement is stochastic, the robot can “oscillate” or systematically drift.

2. Belief-based architecture (explicit inference):

   • Maintains belief $b_t(x)$ over positions $x$
   • Uses a transition model for motion noise and an observation model for sensor noise
   • Chooses actions based on $b_t$
   • Often can recover by accumulating evidence across time

Mini-model (conceptual)

• Hidden state: $s_t = x_t$ (true position)
• Actions: $a_t \\in \\{\\text{left},\\text{right}\\}$
• Transition: $P(x_{t+1}\\mid x_t, a_t)$ due to motion noise
• Observation: $P(o_t\\mid x_t)$ due to sensor noise
• Belief update: $b_{t}(x) \\propto P(o_t\\mid x)\\sum_{x'} P(x\\mid x', a_{t-1})\\,b_{t-1}(x')$

What changes due to architecture?

• Reactive: cannot represent “I might be at location A with probability 0.6 or B with probability 0.4.” It makes the same choice whenever $o_t$ looks similar.
• Belief-based: can decide “move right if right is more likely” and can also take actions that reduce uncertainty when needed (e.g., move to a region where observations disambiguate positions).

Key terms: particle filter, motion model, sensor noise, localization.