Direct Learning of DTRs, Part i: Doubly Robust Value Search

Direct Learning of Dynamic Treatment Regimes: i. Doubly robust value search · ii. DTRESLO · iii. SDSS.

The Q-learning note learns rules by estimating optimal continuation outcomes. Here we compare candidate regimes through estimates of their values. We develop the two-stage regime-specific criterion of Zhang et al. (2013), Section 4, equations (4)–(6), explain when its bias vanishes, and relate uniform value-estimation error to policy regret.

Recap: Optimal continuation versus a candidate's continuation

In two-stage Q-learning, the first-stage target uses an optimal second decision:

Here the conditional expectation averages the intermediate history under the observed conditional distribution. To evaluate a candidate regime, we instead use its specified second rule inside that expectation. Section 1 defines this candidate-specific target.

1. The value of a candidate regime

Observe independent trajectories $O=(X,A_1,L,A_2,R)$, where $H_1=X$, $H_2=(X,A_1,L)$, $A_t\in\{0,1\}$, and $R$ is a bounded terminal outcome. A dynamic treatment regime (DTR) is a pair of measurable rules $d_t:\mathcal H_t\to\{0,1\}$. Its value is

Assume consistency, no interference between individuals, sequential exchangeability of treatment given the recorded history, and true propensities

These are the identification conditions used in the Q-learning note. We use standard Borel history spaces, a regularity condition that allows measurable versions of the conditional expectations below.

Let $O_i\overset{\mathrm{iid}}\sim P$ be the logged records. Unless marked otherwise, $\mathbb E$ and $\Pr$ use this observed-data distribution, with $d$ and the working functions fixed. Write $\mathbb P f=\mathbb E_{O\sim P}[f(O)]$ and $\mathbb P_n f=n^{-1}\sum_{i=1}^n f(O_i)$ for the population and sample averages. By contrast, the causal value $\mathbb E[R\{d_1,d_2\}]$ averages the potential outcome under the candidate regime over the baseline population. Equivalently, it averages the terminal reward under the intervention distribution $P^d$.

For a fixed candidate $d$, define its continuation regressions:

Unlike $Q_1^*$ in Q-learning, $q_1^d$ averages the outcome under the candidate's second rule, not an optimal second rule. Changing $d_2$ generally changes $q_1^d$.

At fixed $(h,a)$, $q_2$ integrates $R\sim P_{R\mid H_2=h,A_2=a}$. At fixed $(x,a)$, $q_1^d$ integrates $L\sim P_{L\mid X=x,A_1=a}$, with $H_2=(x,a,L)$. The outer expectation of $v_1^d(X)$ is only over $X\sim P_X$. In the bias formula, $\mathbb E_X$ has this same marginal meaning; conditioning on $X,A_1$ leaves the intermediate history random, not the first treatment.

2. Inverse weighting, then augmentation

Let $I_t^d=\mathbf1\{A_t=d_t(H_t)\}$. Conditioning first on $H_2$ cancels the second propensity:

The tower property and first-stage probability cancellation then give

Inverse weighting uses treatment probabilities alone. To also use outcome predictions, we add a regression prediction and weight its residual. First consider one treatment, with working propensity $p(a\mid h)>0$ and working outcome regression $m(h,a)$. Neither is assumed correct. For a rule $d$, set

Conditioning and collecting terms yields the exact bias factorization

The error vanishes if $p=e$ or $\bar v=v$: either the treatment-probability model or the selected-treatment outcome mean is correct. We use this same conditional cancellation to construct the two-stage criterion.

3. A two-stage augmented value criterion

Let $p_t(a\mid h)\ge\bar\kappa>0$ be working propensity functions. Let $m_2(h,a)$ and $m_1^d(x,a)$ be bounded working outcome regressions, and define

All working functions are held fixed in the population calculation. Define

Equivalently, rearranging the three terms gives

In this rearrangement, inverse weighting is supplemented by the two regression corrections. Replacing the working functions by fitted ones gives the empirical criterion $\widehat V_n(d)=\mathbb P_n\widehat\Phi_d$.

4. Main theorem: Double robustness

Define selected-treatment regression errors and propensity ratios

Under the stated conditions, the population bias satisfies

Consequently, $\mathbb P\Phi_d=V(d)$ if either both propensity functions are correct or both selected-treatment continuation regressions are correct. This equality holds for fixed working functions. It does not establish finite-sample unbiasedness when we fit those functions and evaluate the criterion on the same records.

First, condition on $H_2$. Because $W_2^d$ is nonzero only when $A_2=d_2(H_2)$,

Thus, conditional on the second-stage history, the remaining bias is the product of a propensity-ratio error and an outcome-regression error, as in Section 2.

Next, propagate this conditional mean to the first stage. Define

The tower property and definition of $q_1^d$ give

Finally, condition on $X$ and use first-stage probability cancellation with the working denominator:

Take expectations and subtract $V(d)=\mathbb E[v_1^d]$ to obtain the displayed bias formula. If $p_1=e_1$ and $p_2=e_2$, both factors $1-\beta_t^d$ are zero. If $\bar v_t^d=v_t^d$ for both stages, both $\delta_t^d$ are zero. $\square$

The outcome-correct case concerns the continuation regressions of this candidate $d$. Substituting optimal Q-functions for every candidate's continuation regressions does not establish this fixed-regime double robustness. Zhang et al. distinguish their regime-specific estimator (6) from the fixed-Q alternative (7).

5. Joint learning from an estimated value criterion

Let $\mathcal D$ be a fixed candidate class, for example a finite grid of threshold-rule pairs. On a training sample, fit the treatment-probability and outcome-regression functions. These are called nuisance functions because they help estimate regime value but are not the treatment rules we seek. On an independent evaluation sample of size $n$, define

Condition on the training sample $\mathcal T$. A fresh evaluation record has $O\sim P$, independent of $\mathcal T$; $\mathbb P\widehat\Phi_d=\mathbb E_{O\sim P}[\widehat\Phi_d(O)\mid\mathcal T]$ integrates only that record, leaving the trained functions fixed. The evaluation records are conditionally i.i.d. from $P$, and $\mathbb P_n$ below averages those records. Define

Assume these suprema are measurable and finite; a finite class satisfies the measurability requirement directly. The first error measures variation between sample and population averages. The second measures bias from the fitted treatment-probability and regression functions. The third measures how far the numerical search falls short of the best sample criterion.

Proposition: Uniform value accuracy controls policy regret

For every $d\in\mathcal D$, add and subtract the two estimated values:

Taking the supremum over $d$ proves

The fixed-regime bias formula controls $\mathsf B_n$ only when its errors are controlled uniformly over the candidate class. Pointwise double robustness alone does not prove this supremum vanishes. If all three errors vanish, the learned regime approaches the best value in $\mathcal D$; an unrestricted-optimum comparison also includes $V^*-\sup_{\mathcal D}V$.

Algorithm: Regime-specific doubly robust search

Input: Trajectories $\{O_i\}$ split into training indices $\mathcal I_{\mathrm{tr}}$ and evaluation indices $\mathcal I_{\mathrm{ev}}$; finite candidate class $\mathcal D$; propensity lower bound $0<\bar\kappa\le1/2$.

Output: A treatment regime $\widehat d\in\mathcal D$.

1. Fit $\widehat e_1,\widehat e_2,\widehat q_2$ on $\mathcal I_{\mathrm{tr}}$, enforcing $\widehat e_t\ge\bar\kappa$.
2. for each $d=(d_1,d_2)\in\mathcal D$ do
3. Set $\widehat v_2^d(h)\gets\widehat q_2(h,d_2(h))$.
4. Regress $\widehat v_2^d(H_2)$ on $(X,A_1)$ using $\mathcal I_{\mathrm{tr}}$ to obtain $\widehat q_1^d$.
5. Set $\widehat v_1^d(x)\gets\widehat q_1^d(x,d_1(x))$.
6. Construct $\widehat\Phi_d$ using the fitted propensities and continuation regressions.
7. Evaluate $\widehat V_n(d)\gets |\mathcal I_{\mathrm{ev}}|^{-1}\sum_{i\in\mathcal I_{\mathrm{ev}}}\widehat\Phi_d(O_i)$.
8. end for
9. Select $\widehat d\in\arg\max_{d\in\mathcal D}\widehat V_n(d)$ with a fixed tie-breaking rule.
10. return $\widehat d=(\widehat d_1,\widehat d_2)$.

With finitely many candidates, we can evaluate them all, so the empirical optimization error is zero. For a continuously parameterized candidate class, we instead search jointly over the parameters and retain the optimization error $\mathsf O_n$.

Splitting the data lets us condition on the fitted functions when analyzing the evaluation sample. We use the split to explain the calculation; we do not claim that Zhang et al. used it. Their paper discusses the cost of fitting regressions separately for each candidate. Parameterizing the rules continuously generally does not remove the objective's discontinuities.

6. A four-outcome check

Use the same treatment tree as Part ii, with treatments recoded as $0,1$:

There are no other covariates. For $d_1=1$, $d_2(0)=1$, and $d_2(1)=0$, the true selected-treatment regressions are $v_2^d(0)=2$, $v_2^d(1)=4$, and $v_1^d=4$. Let $e_1(1)=0.3$ and $e_2(0\mid A_1=1)=0.8$; the unused branch may have any positive treatment probabilities.

Compare exact population criteria, not simulated estimates:

For the last row, $\beta_1=0.3/0.5=0.6$ and $\beta_2=0.8/0.5=1.6$ on the selected branch. Direct substitution gives

The bias identity gives the same answer:

Correct propensities at both stages or correct continuation regressions at both stages remove the bias; misspecifying both collections need not. This robustness concerns value estimation, not the discontinuity of the optimization criterion. Part ii addresses the latter issue by smoothing inverse-weighted treatment agreement and checking whether the surrogate preserves optimal decisions.

References

• Zhang, B., Tsiatis, A. A., Laber, E. B., and Davidian, M. (2013). Robust Estimation of Optimal Dynamic Treatment Regimes for Sequential Treatment Decisions. Biometrika, 100(3), 681–694. DOI, open article. Section 4 distinguishes the regime-specific DR estimator (6) from the fixed-Q augmented estimator (7).
• Offline Policy Learning from Logged Data: identification and Q-learning.
• Part ii: DTRESLO · Part iii: SDSS.