Offline Policy Learning from Logged Data

Q-learning chooses treatments by comparing expected outcomes, starting with the final decision and working backward. We derive this recursion from the causal value formula, prove its population optimality, and show how Q-function estimation errors affect the learned regime's value. We follow Schulte et al. (2014), Sections 2–3 and 5.1, specializing the proofs to binary treatments; the numerical example is constructed for this note.

Recap: The regression result used in Q-learning

For a square-integrable outcome $U$ and covariates $Z$, conditional expectation gives the squared-error minimizer $m(Z)=\mathbb E[U\mid Z]$. For any fixed square-integrable predictor $g$,

The expectations use the joint distribution of $(Z,U)$. This identity explains the regression targets in Section 5; causal assumptions are additionally needed to interpret them as treatment comparisons.

1. From one treatment decision to a treatment rule

Let $X$ denote the covariates available before treatment, $A\in\{0,1\}$ the treatment, and $Y\in\mathbb R$ an outcome for which larger values are preferable. A deterministic treatment rule is a measurable function $d:\mathcal X\to\{0,1\}$.

Write $(X,A,Y)\sim P$ for the observed-data distribution and $P_X$ for its baseline marginal. For observed quantities, unmarked $\mathbb E$ and $\Pr$ refer to $P$. Conditional means average over the corresponding conditional outcome distribution; a rule $d$ is held fixed. The notation $X\sim P_X$ does not impose a parametric model for the covariates.

For now, suppose that the conditional mean outcome under treatment $a$ is identified by

Under the identification assumptions stated in Section 2, the rule's value is

Define the baseline mean and treatment contrast:

For binary $a$, the following is an identity, not a linear-model assumption:

The maximizing rule is

Ties are assigned to treatment $0$. For any competing rule $d$,

Here L is linearity of expectation and cancellation of $m(X)$; S uses the sign of $\Delta$. The loss from a wrong recommendation is $|\Delta(X)|$, not an unweighted classification error.

2. Two-stage data and the causal target

Let $L$ be observed between the two treatments. The record and decision histories are

Only information observed before a decision may enter its rule:

In this two-stage setup, $P$ now denotes the joint observed distribution of $O$, and $O_1,\ldots,O_n\overset{\mathrm{iid}}\sim P$: the records are independent draws from the same population. Subscripts identify marginal and conditional distributions, such as $P_X$ and $P_{L\mid X,A_1}$.

Both treatments are feasible at each relevant history. Outcomes are fully observed and have finite second moments under the interventions considered. Conditional equalities hold almost surely on the relevant support, so they may exclude histories with probability zero.

A dynamic treatment regime, or DTR, consists of two measurable rules,

We optimize over all measurable history-dependent rules, without cross-stage parameter restrictions.

Potential outcomes and regime value

Write $L(a_1)$ for the intermediate covariate that would be observed under first treatment $a_1$, and $Y(a_1,a_2)$ for the final outcome under the treatment sequence $(a_1,a_2)$. Under a regime $d$, define

Our target is the expected final outcome in the baseline population,

Because the first treatment affects the intermediate covariate, the law of $H_2$ changes with $d_1$.

Expectations and probabilities involving potential variables use their underlying population distribution, whose observed-data marginal is $P$. In particular, $\mathbb E[Y^d]$ averages the potential outcome over individuals in the baseline population. Equivalently, write $P^d$ for the distribution of $(X,A_1^d,L^d,A_2^d,Y^d)$ under the regime and $V(d)=\mathbb E_{P^d}[Y^d]$.

The regime determines treatment assignments in this average. It therefore differs from the unweighted observed mean $\mathbb E_P[Y]$. Below, the identification argument expresses the regime value using observed conditional distributions and $X\sim P_X$.

Assumptions connecting the records to interventions

Use the following assumptions, specializing Schulte et al., Section 2.

1. C: Consistency. Treatments are well defined, $L=L(A_1)$, and $Y=Y(A_1,A_2)$.
2. No interference. A patient's potential outcomes do not depend on other patients' treatments.
3. E1–E2: Sequential exchangeability. For the potential-outcome collection

as

4. Positivity. Define the logging propensities

require $e_t(a\mid h)>0$ for both treatments at almost every history reachable under the candidate regimes. Exchangeability is not testable from an observational regression; positivity cannot be replaced by extrapolation.

3. Deriving the two-stage value formula

Define

Intermediate law. For any measurable set $B$,

Outcome mean. First apply E1 to the joint pair $(Y(a_1,a_2),L(a_1))$, then condition on $L(a_1)=l$:

Value formula. Fix $x$ and write $a_1=d_1(x)$. By the tower property (T) and the two identification identities (I),

Integrals become sums for discrete $L$. Positivity makes the conditional treatment comparisons identifiable; positive point masses are not required for continuous covariates.

4. Backward induction: optimize the last decision first

Define the population Q-functions by

At stage 2,

At stage 1, integrate the optimized continuation:

Then

The maximum is inside the expectation because $d_2$ may depend on the realized $L$.

Theorem 1: Optimality of backward induction

Under the assumptions in Section 2, the rules just defined satisfy

Proof. Write $Q_2^d(h)=Q_2^*\{h,d_2(h)\}$. We use identification (I), pointwise maximization (M), monotonicity of conditional expectation (CE), and substitution of the definition of $Q_1^*$ (D).

First, at every relevant history,

Consequently,

For $d=d^*$, $Q_2^{d^*}=M_2$ and $Q_1^*(x,d_1^*(x))=\max_aQ_1^*(x,a)$. Both inequalities are equalities. $\square$

This is the two-stage specialization of Schulte et al., Section 3, not a finite-sample guarantee for fitted regressions.

Why regressing the recorded outcome on the first treatment is different

Let $Q_1^{\mathrm{log}}(x,a_1)=\mathbb E[Y\mid X=x,A_1=a_1]$. By iterated expectation,

The recorded outcome averages over the logging policy's second treatment, whereas Q-learning uses the best second treatment. Their first-stage treatment contrasts can therefore differ, even with perfect regression estimation.

5. From population Q-functions to fitted regressions

The backward recursion specifies conditional means, but those means are unknown. We estimate them by regression, writing $q_t$ for a working regression model and $\widehat Q_t$ for its fitted version. The squared-error identity recalled above identifies each regression target.

Lemma: The squared-error regression target

Let $(Z,R)\sim P_{Z,R}$ with $R$ square-integrable, and set $m(Z)=\mathbb E[R\mid Z]$. Within this lemma, a conditional expectation given $Z=z$ integrates $R\sim P_{R\mid Z=z}$; an unconditional expectation integrates $(Z,R)\sim P_{Z,R}$. The candidate $g$ is fixed, not fitted on the same random pair. For any square-integrable $g$,

The expansion uses linearity and measurability of $m(Z),g(Z)$ given $Z$; the cross term vanishes because $\mathbb E[R-m(Z)\mid Z]=0$. Taking expectations yields

Apply the lemma to $(R,Z)=(Y,(H_2,A_2))$ and then $(R,Z)=(M_2(H_2),(X,A_1))$. The unrestricted regression targets are exactly $Q_2^*$ and $Q_1^*$. Restricted regression families introduce approximation error.

Fit the second-stage regression

For each stage, choose fixed feature maps $u_t(h)\in\mathbb R^{r_t}$ and $v_t(h)\in\mathbb R^{s_t}$. A useful linear working model is

Both $z_t$ and $\theta_t$ have $p_t=r_t+s_t$ entries, and $q_t=z_t^\top\theta_t$. The treatment contrast under this working model is $v_t(h)^\top\psi_t$.

At stage 2, ordinary least squares minimizes

Let $Z_2$ be the $n\times p_2$ matrix whose $i$th row is $z_2(H_{2i},A_{2i})^\top$, and let $\mathbf Y=(Y_1,\ldots,Y_n)^\top$. Differentiating the criterion gives

If $Z_2$ has full column rank, the solution is $(Z_2^\top Z_2)^{-1}Z_2^\top\mathbf Y$. In computation, a least-squares solver avoids explicitly forming the inverse. Set $\widehat Q_2(h,a)=q_2(h,a;\widehat\theta_2)$ and recommend

Construct the first-stage pseudo-outcome

For each recorded second-stage history, predict both treatment outcomes and retain the larger prediction:

The pseudo-outcome is a fitted prediction of the best conditional outcome at the second stage. It is not a measurement of what would have happened under an unobserved treatment. When computing it, keep the recorded $A_{1i},L_i$ in $H_{2i}$.

Regress the pseudo-outcome on first-stage information

Fit the first-stage working model by

With $Z_1$ defined analogously and $\widetilde{\mathbf Y}_1$ the vector of pseudo-outcomes, the normal equations are

Finally, set $\widehat Q_1(x,a)=q_1(x,a;\widehat\theta_1)$ and

We fit the rules backward so that the first-stage regression accounts for optimal continuation, then apply the fitted rules in treatment order. Although these least-squares criteria do not use propensity weights, their causal interpretation still requires exchangeability and positivity. See Schulte et al., Section 5.1.

Theorem 2: From Q-function error to value regret

For an integrable function $g$, define expectation under first-stage rule $d_1$ as follows.

Draw $X\sim P_X$, assign $A_1=d_1(X)$, and draw $L\sim P_{L\mid X,A_1=d_1(X)}$. The induced $H_2$ distribution is the one used by $\mathbb E_{d_1}$, rather than the logged $H_2$ marginal. Thus

Let $\widehat d_t(h)\in\arg\max_a\widehat Q_t(h,a)$, with ties assigned to $0$. If, on all relevant histories and treatment options,

where $\|\cdot\|_\infty$ denotes the essential supremum over the reachable-history support and the finite treatment set, then

First, we decompose the regret by adding and subtracting $V(\widehat d_1,d_2^*)$, then applying the value formula:

Both summands are nonnegative by pointwise optimality. Equivalently, by the binary sign identity in Section 1,

Next, we bound each treatment comparison. Fix $t,h$, and abbreviate $a^*=d_t^*(h)$ and $\widehat a=\widehat d_t(h)$. Add and subtract fitted values:

The outer terms use the assumed uniform error bound; the middle term is nonpositive because $\widehat a$ maximizes $\widehat Q_t$.

Finally, substitute these bounds into the regret decomposition and use monotonicity of expectation:

Conditional on the training data $\mathcal T=(O_1,\ldots,O_n)\sim P^{\otimes n}$, the fitted rules and Q-functions are fixed. The value expectations average over a fresh individual. The bound holds whenever the stated error bounds hold; it does not show that least squares achieves a particular error rate.

Convergence in probability refers to repeated training samples $\mathcal T\sim P^{\otimes n}$. If both uniform errors converge to zero in that sense, the difference between the optimal and learned values also converges to zero.

How second-stage error enters the first-stage target

For a fixed fitted $\widehat Q_2$, define

The maximum is 1-Lipschitz in the sup norm:

Conditional Jensen's inequality and the last bound give

If $\|\widehat Q_1-\overline Q_1\|_\infty\le\delta_1$, the triangle inequality yields

The first-stage error therefore has two sources: error in fitting its regression and error inherited from the estimated second-stage continuation.

6. An exactly solvable example

For an artificial population, let $X\sim\operatorname{Bernoulli}(1/2)$ and

The outcome is

where $\varepsilon$ is an independent standard normal error. Treatments and $L$ are generated in their time order using independent random draws conditional on the stated histories. This construction satisfies sequential exchangeability.

Solve the second stage

Since the error has conditional mean zero,

The contrast $\Delta_2=1.5-2l$ gives

Substituting this choice yields

The second equality uses $l\in\{0,1\}$.

Average over the intermediate response

The first-stage target averages $M_2$ over $L$ conditional on the first treatment:

Hence $\Delta_1(x)=1.4x-0.8$ and $d_1^*(x)=x$.

The optimal value is

Compare with the logged continuation

To see why the raw first-stage outcome regression differs, let the logging mechanism use

All treatment probabilities are $0.2$ or $0.8$.

At $x=0$, calculate the raw outcome regression by summing over $L$ and the logged $A_2$:

The raw regression prefers $a_1=1$, although $Q_1^*(0,0)>Q_1^*(0,1)$. At $x=1$, the logged means are $3.24$ and $4.66$. Its resulting first-stage rule always selects $1$, and its value after optimal second-stage continuation is $(2.5+4.8)/2=3.65<4.05$.

7. A reproducible Python implementation

The working regressions below contain the true Q-functions of this example. Python 3 and NumPy are required.

import numpy as np

rng = np.random.default_rng(17)

def mean_y(x, a1, l, a2):
    return 2 + x + a1 * (1.4 * x - 0.3) + 0.5 * l + a2 * (1.5 - 2 * l)

def simulate(n):
    x = rng.binomial(1, 0.5, n)
    p1 = 0.2 + 0.6 * x
    a1 = rng.binomial(1, p1)
    l = rng.binomial(1, 0.2 + 0.1 * x + 0.5 * a1)
    p2 = 0.2 + 0.6 * ((1 - a1) * l + a1 * (1 - l))
    a2 = rng.binomial(1, p2)
    y = mean_y(x, a1, l, a2) + rng.normal(size=n)
    return x, a1, l, a2, y, p1, p2

def z2(x, a1, l, a2):
    return np.column_stack([np.ones_like(x), x, a1, l, x * a1, a2, a2 * l])

def z1(x, a1):
    return np.column_stack([np.ones_like(x), x, a1, x * a1])

x, a1, l, a2, y, _, _ = simulate(5000)
zero, one = np.zeros_like(x), np.ones_like(x)

# Fit the terminal outcome regression and optimize its continuation.
theta2 = np.linalg.lstsq(z2(x, a1, l, a2), y, rcond=None)[0]
pseudo = np.maximum(z2(x, a1, l, zero) @ theta2,
                    z2(x, a1, l, one) @ theta2)
theta1 = np.linalg.lstsq(z1(x, a1), pseudo, rcond=None)[0]

def d1(x):
    q0 = z1(x, np.zeros_like(x)) @ theta1
    q1 = z1(x, np.ones_like(x)) @ theta1
    return (q1 > q0).astype(int)

def d2(x, a1, l):
    q0 = z2(x, a1, l, np.zeros_like(x)) @ theta2
    q1 = z2(x, a1, l, np.ones_like(x)) @ theta2
    return (q1 > q0).astype(int)

# In this artificial population we can compute the learned regime's value exactly.
exact_value = 0.0
for xv in (0, 1):
    av = int(d1(np.array([xv]))[0])
    p = 0.2 + 0.1 * xv + 0.5 * av
    for lv in (0, 1):
        bv = int(d2(np.array([xv]), np.array([av]), np.array([lv]))[0])
        exact_value += 0.5 * (p if lv else 1 - p) * mean_y(xv, av, lv, bv)

# Evaluate the frozen regime on independent logged records with known propensities.
x, a1, l, a2, y, p1, p2 = simulate(100000)
prob1 = np.where(a1 == 1, p1, 1 - p1)
prob2 = np.where(a2 == 1, p2, 1 - p2)
matches = (a1 == d1(x)) & (a2 == d2(x, a1, l))
ipw_value = np.mean(matches * y / (prob1 * prob2))

print("First-stage choices for X = 0, 1:", d1(np.array([0, 1])))
print("Exact value of learned regime:", round(float(exact_value), 4))
print("Held-out IPW estimate:", round(float(ipw_value), 4))
print("Optimal population value:", 4.05)

First-stage choices for X = 0, 1: [0 1]
Exact value of learned regime: 4.05
Held-out IPW estimate: 4.0246
Optimal population value: 4.05

Why the held-out weighting estimates regime value

Condition on the training data so that the learned rules are fixed. For an independent evaluation record, write

At stage 2, averaging over the logged treatment cancels its probability:

The first-stage weight performs the analogous cancellation. Applying iterated expectation gives

Averaging over $X$ recovers the identified value formula for $\widehat d$. Thus the ordinary, unnormalized inverse-probability-weighted average used in the code is conditionally unbiased for $V(\widehat d)$ when the logging propensities are known and the stated identification and integrability conditions hold. Estimated propensity scores require additional analysis; the exact cancellation is not a finite-sample unbiasedness result for arbitrary fitted weights.

8. Scope of the results

Theorem 1 concerns population optimality; Theorem 2 is conditional on uniform Q-function error bounds. Linear Q-functions need not remain linear after maximization and averaging. Ordinary regression Q-learning is not doubly robust, and generated pseudo-outcomes do not justify naive first-stage standard errors. Treatment ties can create nonregular inference; see Schulte et al., Section 8. We assume full histories, complete outcomes, and a terminal-outcome objective, not a general deep or long-horizon offline-RL setting.

9. Extending the recursion to more stages

The same population construction applies to $T$ stages with a terminal outcome. Let $H_t$ contain all information available before $A_t$, and let $\mathcal A_t(h_t)$ be the finite set of feasible treatments. Under corresponding consistency, exchangeability, positivity, and moment conditions, define

At every stage, choose a treatment maximizing its Q-function, using a fixed convention to resolve ties. The fitted version works backward by replacing each future maximum with its fitted counterpart and regressing it on the current history and treatment.

For cumulative stage rewards, use the corresponding reward recursion; do not add a reward twice if it is already included in the target outcome.

References and further reading

• Schulte, P. J., Tsiatis, A. A., Laber, E. B., and Davidian, M. (2014). Q- and A-Learning Methods for Estimating Optimal Dynamic Treatment Regimes. Statistical Science, 29(4), 640–661. Journal article, open manuscript. Sections 2–3 provide the causal framework and recursion; Section 5.1 describes regression-based Q-learning.
• For the squared-error regression principle used in fitting, see Conditional Expectation under Squared-Error Loss.
• For the derivatives and least-squares calculations, see Vector and Matrix Derivatives and Linear Regression.