Reinforcement Learning, Part ii: Partial Observability, Offline Data, and DTRs

Reinforcement Learning: i. Foundations and learning methods · ii. Partial observability, offline data, and DTRs.

An agent may lack access to the environment's full state, or it may have to learn without collecting new interactions. These are different restrictions: partial observability concerns information at decision time, while offline learning concerns access to training data. We derive the corresponding belief-state and off-policy value identities, then connect finite-horizon policy learning to dynamic treatment regimes (DTRs).

Recap: What the fully observed Bellman equation assumes

In Part i, a fixed stationary policy satisfies

Here $s$ is an observed sufficient state, $P$ its transition kernel, $r$ expected immediate reward, and $\gamma<1$ the discount factor. Partial observability changes what information can replace $s$; offline learning changes how the quantities in this equation can be estimated.

1. Separate the information model from data access

Part i assumed a fully observed Markov decision process (MDP). Its conditional dynamics depended on the current state and action rather than the earlier history. Two separate changes are possible:

A partially observable Markov decision process (POMDP) assumes Markov dynamics for a hidden, or latent, state. The current observation need not contain enough information to predict the next observation and reward given the action. Offline datasets can arise from fully or partially observed processes. These distinctions follow Kaelbling et al., Section 3 and Levine et al., Sections 2.1–2.2.

2. Partial observability: Redefine the state and history

Let $S_t\in\mathcal S$ be a hidden state, $O_t\in\mathcal O$ an observation, and $A_t\in\mathcal A$ an action. Rewards $R_{t+1}\in\mathcal R$ are observed. For this section, take all four spaces to be finite and $0\le\gamma<1$. The finite reward set is a simplifying assumption for the sums, not part of the general definition of a POMDP.

The information available before action $A_t$ is

An admissible policy uses this history, with randomization independent of the hidden state conditional on $H_t$:

We specify a joint transition–observation–reward kernel

This asserts the latent controlled Markov property and time homogeneity. The initial state has distribution $\rho_0$, and $O_0$ is drawn from an initial observation kernel $E_0(o\mid s)$. Together, $(\mathcal S,\mathcal A,\mathcal O,\mathcal R,K,\rho_0,E_0,\gamma)$ specifies the model used here.

For a fixed policy, $\mathbb P^\pi$ denotes the trajectory distribution generated by $S_0\sim\rho_0$, $O_0\mid S_0\sim E_0(\cdot\mid S_0)$, $A_t\mid H_t\sim\pi_t(\cdot\mid H_t)$, and $(S_{t+1},O_{t+1},R_{t+1})\mid S_t,A_t\sim K(\cdot\mid S_t,A_t)$. Probabilities in the belief derivation use this distribution. Conditioning on a realized history averages only over the remaining latent uncertainty and subsequent draws, not over a new history.

The common separate transition/emission notation is a special case. For example, under the corresponding conditional independence assumptions,

The joint kernel retains any information about the hidden state carried by observed rewards. The policy still uses its history rather than assuming that a single $O_t$ summarizes the past.

Click to enlarge · Scroll sideways on small screens

Deriving the belief update

Define the belief state

For an initial observation of positive probability, Bayes' rule gives

First, the policy's information restriction implies that observing the selected action adds no information about $S_t$ beyond $H_t$:

whenever the conditioning event has positive probability. The law of total probability and the kernel assumption therefore give

Next, define the probability of the next observed pair:

Conditioning on $(O_{t+1},R_{t+1})=(o,r)$ and applying Bayes' rule yields

This update is defined when $c>0$; its value at an impossible observation can be assigned arbitrarily. In the transition/emission special case with no additional reward information, this reduces to the usual prediction followed by observation weighting in Kaelbling et al., Section 3.3.

Why the belief is a sufficient state

Write $B(b,a,o,r)$ for the updated probability vector. Conditional on $H_t,A_t=a$, the pair $(o,r)$ has probability $c(b_t,a,o,r)$ and the next belief equals $B(b_t,a,o,r)$. Both depend on $H_t$ only through $b_t$. Thus the belief process is a controlled Markov process.

For example, let $U_h(b)$ be the optimal expected return with $h$ decisions remaining, with $U_0=0$. Backward induction on this sufficient state gives

Terms with $c=0$ contribute zero. Finally, to prove optimality, consider any history with belief $b$. Its next-outcome distribution is the one derived above, so no randomized action choice can exceed the displayed maximum. Applying the same argument to the remaining $h-1$ decisions proves the claim by induction. A belief-based action attaining the maximum reaches this bound. This is the finite-horizon belief-MDP argument in Kaelbling et al., Section 3.4.

The belief recursion is exact when the specified kernels describe the process. If those kernels are unknown, estimating a belief representation introduces an additional statistical problem.

Example: The same observation can require different actions

Let a fixed hidden state $S\in\{0,1\}$ have prior probability $1/2$ for either value. Before a terminal decision, observe three conditionally independent signals, each equal to $S$ with probability $0.8$. There are no informative rewards or choices during this passive observation period. The final action guesses $S$ and earns one for a correct guess and zero otherwise.

For a sequence with $n_1$ ones and $n_0$ zeros, conditional independence and Bayes' rule give posterior odds

The histories $(1,1,0)$ and $(0,0,0)$ have the same current observation $O_2=0$, but

The terminal action values are $Q(b,1)=b(1)$ and $Q(b,0)=1-b(1)$. The optimal actions are therefore 1 and 0, respectively. A policy using only $O_2$ cannot implement both decisions. A policy using the posterior belief can.

3. Offline is a data-access restriction

For the next derivation, return to a fully observed finite MDP. Use a finite horizon $T$ and bounded rewards, so the discount may now satisfy $0\le\gamma\le1$. Policies may depend on the history and decision time. This is a new finite-horizon setup, not an application of Part i's infinite-horizon theorem with $\gamma=1$.

Suppose a behavior policy $\beta_t(a\mid H_t)$ generated $n$ independent trajectories. The logged dataset is

Rewards and states within a trajectory need not be independent. Offline RL learns a policy using this fixed dataset without further environment interaction for training. Off-policy learning instead describes the relationship between the policy generating data and the target policy; it does not prohibit collecting more data. Levine et al., Figure 1 and Section 2.2 illustrate the distinction.

Click to enlarge · Scroll sideways on small screens

The behavior policy determines which trajectories enter the dataset, whether or not the learning algorithm estimates that policy. We make its role explicit in the value identity next.

An off-policy value identity

For a compact derivation, additionally take a finite reward set and write $p(s',r\mid s,a)$ as a probability mass function. Let $p_\beta$ and $p_\pi$ be trajectory probabilities under behavior and target policies, with the same initial distribution and environment kernel:

Let $P_\beta,P_\pi$ be the trajectory distributions with these mass functions. The dataset satisfies $\tau_i\overset{\mathrm{iid}}\sim P_\beta$, while policy value averages $\tau\sim P_\pi$. Write $\mathbb E_\beta=\mathbb E_{\tau\sim P_\beta}$ and $\mathbb E_\pi=\mathbb E_{\tau\sim P_\pi}$, with policies and environment fixed. Neither expectation averages over the process of fitting a policy.

Assume target-policy coverage: $p_\pi(\tau)>0$ implies $p_\beta(\tau)>0$. In particular, every target action on a target-reachable history must have positive behavior probability. Cancellation of the common environment factors gives

Let $G(\tau)=\sum_{t=0}^{T-1}\gamma^t r_{t+1}$. First substitute the ratio, then sum over trajectories:

The second equality uses coverage. This is an identity for a fixed target policy and true trajectory probabilities, consistent with the importance-sampling treatment in Levine et al., Section 3.1. If behavior probabilities are fitted, or the same data select the target policy, additional estimation and data-dependence arguments are required.

Missing coverage is an identification problem

Consider one state, one decision, and two actions. The behavior policy always chooses action 0 and obtains reward zero. Two environments agree on every possible logged observation:

The logged-data distributions are identical, but the optimal actions differ. Without additional structural information, no amount of data from this behavior policy can distinguish the environments. A policy choosing action 1 lacks an identified value from these logs.

Even with positive coverage, probabilities can be small. For a fixed time $t$, let $d_t^\pi(s,a)$ and $d_t^\beta(s,a)$ denote the target and behavior state–action distributions. If

then, for any real function $f_t$ on state–action pairs,

This follows by multiplying the pointwise probability inequality by $|f_t|$ and summing. Large $C_t$ can amplify errors measured under the logged distribution. The inequality explains one role of coverage; it is not by itself a finite-sample policy-learning guarantee.

4. Partial observability is not the same as hidden confounding

The belief update in Section 2 used

This is justified when the agent selects actions from recorded information and independent randomization. It can fail for historical data when the behavior policy uses additional information unavailable in $H_t$.

For example, if a decision maker observes the hidden state and chooses $A_t=S_t$, then conditioning on the action reveals that state. A posterior that ignores this selection is incorrect. Likewise, the recorded conditional distribution of the next outcome given $(H_t,A_t)$ need not equal its distribution when the action is externally assigned.

To identify intervention values from partially observed logs, we must therefore specify what information the behavior decision maker used. A recurrent representation of $O_t$ or an estimated belief can summarize recorded information, but cannot by itself justify the assumptions needed for causal identification.

5. Dynamic treatment regimes: A finite-horizon connection

Now use treatment stages $t=1,\ldots,T$. Let $L_t$ be information observed before treatment $A_t$, and let $Y$ be a bounded terminal outcome, with larger values preferable. For explicit formulas, assume finite covariate and treatment spaces and fully observed trajectories from independent individuals. Define

Each $d_t$ maps the available history to a feasible treatment. This is the setup of Schulte et al., Sections 2–3 and our earlier Q-learning note.

Scroll horizontally to inspect the full figure →

What replaces the latest-state Markov assumption?

A history-based DTR does not require

The transition may depend on all recorded past treatments and covariates. We can still use finite-horizon dynamic programming by taking the whole history, including the stage, as the state. This does not assume that the latest covariate $L_t$ summarizes the past. The history space grows with $t$, so this construction does not provide a low-dimensional state representation.

A terminal-outcome DTR corresponds to the following finite-horizon RL reward convention:

Here RL's action at time $t-1$ is treatment $A_t$, and its state before that action is $H_t$. Intermediate utilities can instead be included as rewards when that is the chosen scientific objective.

Causal identification requires separate assumptions

Let $Y^d$ denote the outcome under regime $d$, and define its causal value by $V(d)=\mathbb E[Y^d]$. Let $\mathcal W^*$ collect the potential covariates and outcomes under all fixed treatment sequences.

In this section, $P_{\mathrm{obs}}$ denotes the logged longitudinal distribution. Unmarked conditional means and treatment probabilities use $P_{\mathrm{obs}}$. By contrast, $\mathbb E[Y^d]$ averages the regime's potential outcome over the baseline population. Equivalently, it averages the terminal outcome under the intervention distribution $P^d$.

The assumptions below identify $P^d$ from the baseline distribution and the observed conditional covariate and outcome distributions. The treatment rules $d_t$ replace the logged treatment assignments. For the standard no-interference setting, assume:

• Consistency: recorded covariates and outcome equal their potential counterparts under the treatments actually received.
• No interference: one individual's treatments do not affect another individual's potential covariates or outcome; treatment versions are well specified.
• Sequential exchangeability: $A_t\perp\!\!\!\perp\mathcal W^*\mid H_t$ at every stage. This is a strong sufficient no-unmeasured-confounding assumption.
• Positivity: $e_t(a\mid h)=\mathbb P(A_t=a\mid H_t=h)>0$ for treatments required by the target regime on its reachable histories.

These are the identification conditions in Schulte et al., Sections 2–3, not consequences of labeling histories as states. Evaluating one regime requires support for its own treatment choices. Optimizing over all feasible regimes requires support for every candidate treatment on each reachable history.

In the present finite-support setting, positivity gives a positive minimum over the required probabilities. It does not specify how small that minimum may be. Bounds across a sequence of data-generating models often impose a common lower bound $e_t\ge\kappa>0$. With infinite history spaces, pointwise positivity need not imply such a uniform bound even for a single model.

Under these assumptions, externally assigning the treatments preserves the observed conditional covariate/outcome kernels. For two stages, write

The g-formula is

Consistency links observed and potential variables, exchangeability identifies the intervention kernels, and positivity ensures the required conditionals are supported by the observed distribution.

The DTR recursion is a finite-horizon Bellman recursion

For the unrestricted class of history-based regimes, define

The maximizing rules are $d_t^*(h)=\operatorname{first}\arg\max_aQ_t^*(h,a)$, using a fixed action ordering. First, the final rule maximizes the identified conditional terminal mean. Next, its optimized continuation value becomes the outcome averaged at the preceding stage. Repeating this argument backward proves optimality by the tower property and the fact that a maximum dominates any competing treatment choice. The earlier article gives the complete two-stage optimality and estimation-error proofs.

The correspondence is therefore precise: learning a finite-horizon DTR from existing longitudinal data is a history-based offline policy-learning problem, with an explicitly causal value target and identification assumptions. DTRs can also be studied in adaptive or prospectively randomized settings; the entire field is not defined by offline data access.

6. Which distinction determines the method?

These choices can coexist. A model-based algorithm can learn offline; an actor–critic algorithm can use beliefs; and a DTR can be learned by Q-learning or by the direct-learning methods discussed earlier. The information model, data-access restriction, causal assumptions, and optimization method answer different questions.

References

• Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101, 99–134. Sections 3.2–3.4 and Figure 2 motivate the belief-state construction and controller diagram.
• Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Figure 1, Sections 2.1–2.4, and Section 3.1 motivate the data-access distinctions, coverage discussion, and importance weighting.
• Phillip J. Schulte, Anastasios A. Tsiatis, Eric B. Laber, and Marie Davidian (2014). Q- and A-Learning Methods for Estimating Optimal Dynamic Treatment Regimes. Open manuscript. Sections 2–3 supply the histories, causal conditions, and backward recursion used for the DTR connection.