Direct Learning of DTRs, Part iii: Multiple Treatments and SDSS

Direct Learning of Dynamic Treatment Regimes: i. Doubly robust value search · ii. DTRESLO · iii. SDSS.

The binary rule in Part ii chooses between two treatments from the sign of one score. With three treatments, we instead assign a score to each treatment and compare their differences. We study a product-based surrogate from Simultaneous Direct Search with Surrogates (SDSS), developed by Laha, Chapagain, Cicherski, and Sonabend-W, and prove a regret bound for this specific construction. The paper's general characterization of Fisher consistency covers a broader class.

Recap: The probability identity from the binary case

Part ii uses an increasing sigmoid satisfying

Consequently, its two treatment weights form a probability distribution. We can therefore interpret the surrogate objective as the value of a regime that samples treatments from those probabilities. With multiple treatments, the products of pairwise weights need not sum to one. We first determine which of their properties can support a similar value comparison.

1. Treatment-specific scores

Begin with two stages and three treatments per stage. Observe independent trajectories $O=(X,A_1,L,A_2,R)$, with $H_1=X$, $H_2=(X,A_1,L)$, and $A_t\in\{1,2,3\}$. Assume consistency, no interference between individuals, sequential exchangeability, and known true propensities $e_t(a\mid h)\ge\kappa>0$. Assume standard Borel history spaces and a positive bounded terminal utility $0<r_0\le R\le M$.

Let $O_i\overset{\mathrm{iid}}\sim P$ be the logged records. Unless another subscript is shown, $\mathbb E$ and $\Pr$ use $P$, holding candidate scores and rules fixed; $\mathbb E_X$ averages only $X\sim P_X$. Conditional outcome means integrate $R\sim P_{R\mid H_2,A_2}$, while stage-one continuation means integrate $L\sim P_{L\mid X,A_1}$. A causal regime value instead averages the terminal outcome under the intervention distribution defined below; inverse weighting expresses that value using $O\sim P$.

A stage-specific score is a measurable map $f_t:\mathcal H_t\to\mathbb R^3$. Write $f=(f_1,f_2)$ and use the deterministic rule

The minimum fixes ties; it has no effect on the value of an optimal tied treatment. The inverse-weighting argument in Part i applies to any finite treatment set:

For a score vector $x\in\mathbb R^3$, treatment $a$ wins when its score exceeds every competing score. Away from ties,

This factorization suggests smoothing each pairwise comparison.

2. A product-based smooth surrogate

Use the same increasing sigmoid as Part ii:

Define the single-stage surrogate

For three treatments this is explicitly

The two-stage surrogate value is

This is the increasing pairwise-comparison convention for the product construction in the SDSS paper, Section 4.2.2 and Figure 1. We maximize $J$; equivalently, we minimize the inverse-weighted loss $1-\phi_{A_1}\phi_{A_2}$. No outcome-regression augmentation is present, so the criterion does not inherit Part i's double robustness.

Lemma 1: The surrogate weights form a subprobability vector

Put $a=g(x_1-x_2)$, $b=g(x_1-x_3)$, and $c=g(x_2-x_3)$. Symmetry gives

Thus $0<s(x)\le1$. Unlike the binary weights in DTRESLO, these weights generally do not sum to one. For instance, $x=(0,0,0)$ gives $\phi_j=1/4$ and $s=3/4$.

If $d=\min\arg\max_jx_j$, every margin $x_d-x_b$ is nonnegative. Monotonicity therefore gives

The regret proof uses both properties: total surrogate mass is at most one, and the treatment selected by the score retains a positive weight.

3. Relative scores and the learning algorithm

Adding the same constant to all scores changes neither the margins nor the selected treatment:

Because only score differences enter the criterion, we may subtract $f_{t,1}(h)$ and use only two free scores per stage:

The paper uses the opposite sign for its relative-score coordinates; both parameterizations represent the same score differences. For a deterministic parameterized class $\mathcal F_n$, define

For differentiable score functions $f_{t,a}(h;\theta_t)$, let $m_{tib}=f_{t,A_{ti}}(H_{ti})-f_{t,b}(H_{ti})$. Since $g'(u)/g(u)=2\{1-g(u)\}$, the product and chain rules give

Input: Trajectories $\{O_i\}_{i=1}^n$; true propensities $e_1,e_2$; differentiable relative-score maps $u_{t,2},u_{t,3}$ with joint parameter $\theta$; $B\ge1$ initializations; $K\ge1$ iterations; positive learning rates $\{\eta_r\}_{r=0}^{K-1}$.

Output: Three-treatment rules $\widehat d_1,\widehat d_2$.

1. Compute $w_i\gets R_i/\{e_1(A_{1i}\mid H_{1i})e_2(A_{2i}\mid H_{2i})\}$.
2. Represent each stage by $f_t(h;\theta)=(0,u_{t,2}(h;\theta_t),u_{t,3}(h;\theta_t))$.
3. for $b=1,\ldots,B$ do
4. Initialize all stage parameters jointly at $\theta^{(b,0)}$.
5. for $r=0,\ldots,K-1$ do
6. Evaluate the unnormalized product objective $\widehat J_n$ and its displayed stage gradients.
7. Update jointly: $\theta^{(b,r+1)}\gets\theta^{(b,r)}+\eta_r\nabla_\theta\widehat J_n(\theta^{(b,r)})$.
8. end for
9. end for
10. Select $\widehat\theta$ with the largest $\widehat J_n$ among all visited iterates, including initializations; break ties by visit order.
11. return $\widehat d_t(h)=\min\arg\max_{a\in\{1,2,3\}}f_{t,a}(h;\widehat\theta)$.

This pseudocode follows the structure of SDSS, Algorithm 1 and Section 5.1, using the full sample to compute each gradient. The paper also uses smaller batches of records and restart strategies; we do not reproduce all those details here. Neither smoothness nor restarts guarantee a global maximum. The algorithm does not require a separate pseudo-outcome regression at each stage.

4. Main theorem: A two-stage regret bound

Let $V^*=\sup_d V(d)$ over measurable deterministic regimes and $J^*=\sup_f J(f)$ over measurable finite-valued score tuples. For the product surrogate above,

This is a teaching result for our specific sigmoid product and positive bounded terminal utility. The proof below is self-contained; the constant is obtained from Lemma 1, not quoted as the constant of the paper's more general Proposition 1. That proposition covers a broader class of surrogates.

First, normalize the surrogate weights for the proof. Define

Lemma 1 implies that $\pi_t$ is a probability vector and

Under the stochastic regime $\pi$, treatment $A_t$ is sampled from $\pi_t(\cdot\mid H_t)$ and the subsequent history follows the corresponding intervention.

More explicitly, its distribution $P^\pi$ draws $X\sim P_X$, $A_1\mid X\sim\pi_1(\cdot\mid X)$, $L\mid X,A_1\sim P_{L\mid X,A_1}$, $A_2\mid H_2\sim\pi_2(\cdot\mid H_2)$, and $R\mid H_2,A_2\sim P_{R\mid H_2,A_2}$. Write $\mathbb E_\pi=\mathbb E_{O\sim P^\pi}$; the observed distribution $P$ instead uses $e_1,e_2$ for the treatment draws. Sequential probability cancellation gives

Because $R\ge0$ and $s_t\le1$, the discarded surrogate mass can only decrease this expected utility. We normalize the weights to compare values in the proof; the algorithm still optimizes the unnormalized objective $\widehat J_n$.

Next, decompose stochastic-regime regret. Using the Bellman recursion, define

Let $\mathbb E_{\pi_1}$ average over the first three draws $(X,A_1,L)$ above. The notation $\mathbb E_{d_1}$ uses those draws with $A_1=d_1(X)$ instead; it therefore need not use the logged marginal distribution of $H_2$. Adding and subtracting $\mathbb E_{\pi_1}M_2(H_2)$ yields

The second equality uses the tower property and $\sum_a\pi_t(a\mid h)=1$. In particular $V(\pi)\le V^*$; together with $J(f)\le V(\pi)$, this gives $J(f)\le V^*$.

To identify the surrogate supremum, choose a measurable optimal deterministic regime $d^*$ by backward induction and set

At the selected treatment both pairwise margins are $k$; at any other treatment a margin against the selected treatment is $-k$. Therefore

The inverse-weighted integrand is bounded by $M/\kappa^2$. Dominated convergence and causal identification give

Together with $J(f)\le V^*$, this proves $J^*=V^*$. A finite maximizing score need not exist.

Finally, compare the two regrets by retaining the deterministic treatment path. Write $d=d_f$. Applying the stochastic-regime regret decomposition to deterministic treatment choices gives

Nonnegative Bellman gaps allow us to drop the other first-stage treatments:

For $S_2$, condition on $X,A_1$, keep the first treatment $d_1(X)$, and then the second treatment $d_2(H_2)$:

Reaching the deterministic second-stage path requires both its first treatment and its second treatment. Each receives probability at least $1/4$, giving the two factors $1/4$. Combining these gap bounds with the value comparison,

Rearranging proves the theorem. In particular, $J(f_n)\to J^*$ implies $V(d_{f_n})\to V^*$. $\square$

5. Why one-stage consistency is insufficient

For a nonnegative vector of conditional outcomes $\mu=(\mu_1,\mu_2,\mu_3)$, put $M_\mu=\max_a\mu_a$, $d_x=\min\arg\max_a x_a$, and $\Psi(x;\mu)=\sum_a\mu_a\phi_a(x)$. Lemma 1 and scores concentrating on a maximizing coordinate imply

Moreover, subtracting the conditional surrogate value from this supremum gives

Nonnegativity justifies dropping the other terms. Thus a wrong treatment has a strictly positive conditional surrogate gap, while the optimized continuation value is exactly the largest conditional outcome.

Now shift every weight by $\varepsilon>0$: $\widetilde\phi_a=\phi_a+\varepsilon$. Its one-stage objective differs by a score-independent constant:

Thus the shift preserves the single-stage decision. However, in a two-stage problem its optimized continuation value becomes

not just $\max_a\mu_a$. The additional sum depends on the earlier treatment. Consider this positive outcome table with no other covariates:

The surrogate-optimal limiting first treatment is $2$, whereas the true optimum selects $1$. Indeed, after taking each second-stage surrogate supremum, first-stage maximization selects the largest shifted continuation; its own added constant cannot change that ordering. The limiting regime has value $3.5<4$ despite single-stage consistency.

This illustrates the distinction formalized by the SDSS paper's Theorems 4.1–4.2 and Lemma 4.3. For nonnegative separable surrogates, Theorem 4.1 requires single-stage calibration at every stage (Condition N1) and, at stages $t\ge2$, the continuation condition

The calculations above verify both properties for our three-treatment product, with $C_t=1$. The shifted surrogate retains calibration but violates this continuation condition. The paper's necessity theorem permits nonnegative outcomes; its sufficient theorem is stated under its positive-outcome assumptions. The general characterization is broader than the particular sigmoid-product proof given here.

6. More stages, and what the fitted learner needs

For $T$ stages and $K_t$ treatments at stage $t$, use

The subprobability property extends as follows. Independently compare each unordered pair, with probability $g(x_a-x_b)$ that $a$ beats $b$. The probability that $a$ beats every opponent is $\phi_{t,a}(x)$. Two different treatments cannot both beat every opponent, so the events are disjoint and their probabilities sum to at most one. At the largest-score treatment,

The same normalization and dominated-convergence arguments apply for fixed finite $T,K_t$, with positive bounded utility and uniformly positive true propensities. In the Bellman-gap decomposition, the stage-$t$ term under $\pi$ retains the deterministic prefix with probability at least $\prod_{s=1}^tc_s$. Since $c_s\le1$, summing the nonnegative gaps gives

This bound concerns unrestricted population scores. For a deterministic fitted score class $\mathcal F_n$, define measurable approximation, sampling, and optimization errors as in Part ii:

For the two-stage, three-treatment case, the same add-and-subtract proof yields

Here the training sample is $\mathcal T\sim P^{\otimes n}$. A fitted rule's value averages over a fresh individual under that rule's intervention distribution. To study statistical convergence, we vary $\mathcal T$ and any randomness in fitting, rather than replacing the intervention expectation by a training-sample average.

The learned value approaches the optimal value if all three errors vanish. A fixed linear class need not approximate the optimum arbitrarily well, and gradient optimization need not find the best fitted objective. Estimated propensities would add another source of error. The paper's quantitative rates require further conditions on smoothness, treatment contrasts, function-class complexity, and optimization. Fisher consistency alone does not establish those rates.

References

• Laha, N., Chapagain, N., Cicherski, V., and Sonabend-W, A. (2025). On Fisher Consistency of Surrogate Losses for Optimal Dynamic Treatment Regimes with Multiple Categorical Treatments per Stage. arXiv, paper and supplement. Sections 4–6 give the consistency conditions, surrogate constructions, SDSS algorithm, and regret analysis. Our displayed constant and proof concern the explicit sigmoid-product special case above.
• Laha, N., Sonabend-W, A., Mukherjee, R., and Cai, T. (2024). Finding the Optimal Dynamic Treatment Regimes Using Smooth Fisher Consistent Surrogate Loss. The Annals of Statistics, 52(2), 679–707. DOI.
• Part i: Doubly robust value search · Part ii: DTRESLO · Q-learning.