MARL, Part 3: Centralized Training, Decentralized Decisions

August 13, 2026

On this page

Part 1 introduced Markov games, and Part 2 studied shared-reward credit assignment under full observation. We now let agents receive different observations. Centralized training with decentralized execution (CTDE) uses joint information for training but restricts each deployed actor to its own information. We examine which centralized critics preserve the local actor's gradient and why a critic using only the physical state can discard relevant information from the actor's history.

Recap: The conditional-expectation identity behind a critic

The policy-gradient calculation weights a return by an actor's score. If the information $Z$ determines that score $\psi$ , the tower property gives

\mathbb E[\psi G] =\mathbb E[\psi\,\mathbb E[G\mid Z]].

Both expectations use the same fixed-policy trajectory distribution. The identity lets us replace the return by its conditional mean without changing the expected gradient. The critic's information $Z$ must retain everything needed to determine the score; otherwise this argument does not apply.

1. Specify what each agent knows

Consider $N$ cooperative agents and a finite horizon $T$ . States $S_t\in\mathcal S$ , actions $A_{i,t}\in\mathcal A_i$ , observations $O_{i,t}\in\mathcal O_i$ , and shared rewards $R_{t+1}\in\mathcal R$ take values in finite sets. Rewards are bounded and $0\le\gamma\le1$ . Write $\boldsymbol A_t=(A_{1,t},\ldots,A_{N,t})$ and similarly $\boldsymbol O_t$ . The environment has a latent controlled Markov state:

\begin{aligned} S_0&\sim\rho_0,\qquad \boldsymbol O_0\mid S_0\sim E_0(\cdot\mid S_0),\\ (S_{t+1},\boldsymbol O_{t+1},R_{t+1}) \mid\text{past},\boldsymbol A_t &\sim K(\cdot\mid S_t,\boldsymbol A_t). \end{aligned}

The joint observation kernel may correlate agents' observations. No conditional independence of observations is assumed. This is a finite-horizon decentralized partially observable Markov decision process (Dec-POMDP), using a joint kernel to include observed rewards. Compare Oliehoek, Spaan and Vlassis, Section 2.1.

Original Oliehoek, Spaan and Vlassis Figure 1 follows three decision times. A hidden environment state emits a joint observation; each agent receives only its observation component and selects its own action. Joint actions drive state transitions. The lower timeline marks the first, second and final decision stages. — **Figure 1. Local observations and joint actions in a Dec-POMDP.** Original figure from Oliehoek, Spaan and Vlassis (2008), Optimal and Approximate Q-value Functions for Decentralized POMDPs, Figure 1. The source writes the horizon as h and uses superscripts for time. Our formulation additionally allows a shared observed reward and a joint transition-observation-reward kernel. Full resolution ↗Figure excerpt for discussion; copyright remains with the authors/rightsholder.

In our convention every agent observes the shared reward after each action. Before choosing $A_{i,t}$ , agent $i$ has local history

H_{i,t}=(O_{i,0},A_{i,0},R_1,O_{i,1},\ldots,A_{i,t-1},R_t,O_{i,t}).

It does not automatically observe teammates' actions, observations, or the state. The joint history available to the trainer is

H_t=(\boldsymbol O_0,\boldsymbol A_0,R_1,\ldots, \boldsymbol A_{t-1},R_t,\boldsymbol O_t).

Each $H_{i,t}$ is a projection of $H_t$ . A decentralized policy with independent private randomization satisfies

\boxed{ \mathbb P_\theta(\boldsymbol A_t=\boldsymbol a\mid H_t=h,S_t=s) =\prod_i\pi_{i,\theta_i,t}(a_i\mid h_i). }

Here $h_i$ is the local projection of $h$ and $\theta_i$ is fixed during execution. Training may estimate $\theta_i$ using every agent's data; the runtime policy still receives only $h_i$ . Sharing trained parameters does not share current observations. Messages, if allowed, must be explicitly included in the receiving agent's history.

Let $P_\theta$ be the trajectory distribution generated by the initial law, these product policies, and $K$ . For a realized full trajectory $\tau$ , including latent states, its probability is

p_\theta(\tau)=\rho_0(s_0)E_0(\boldsymbol o_0\mid s_0) \prod_{t=0}^{T-1}\left[ \prod_i\pi_{i,\theta_i,t}(a_{i,t}\mid h_{i,t}) \right]K(s_{t+1},\boldsymbol o_{t+1},r_{t+1}\mid s_t,\boldsymbol a_t).

All on-policy expectations below use $P_\theta$ , keeping the environment and initial law fixed. Define

G_t=\sum_{k=t}^{T-1}\gamma^{k-t}R_{k+1},\qquad J(\theta)=\mathbb E_{\tau\sim P_\theta}[G_0].

CTDE specifies information access, not whether training uses a fixed dataset. Training can collect fresh simulator trajectories, as below. This differs from the offline-data restriction.

2. A centralized trainer is not a centralized controller

Let $\Pi_{\mathrm{dec}}$ contain the local-history product policies above. Let $\Pi_{\mathrm{joint}}$ allow a controller to observe $H_t$ and choose the whole joint action, and $\Pi_{\mathrm{full}}$ additionally reveal $S_t$ . Each larger controller can imitate the smaller one by ignoring extra information. Therefore

\Pi_{\mathrm{dec}}\subseteq\Pi_{\mathrm{joint}}\subseteq\Pi_{\mathrm{full}}, \qquad \boxed{J^*_{\mathrm{dec}}\le J^*_{\mathrm{joint}}\le J^*_{\mathrm{full}}},

where $J^*_{\mathcal C}=\sup_{\pi\in\Pi_{\mathcal C}}\mathbb E_{P_\pi}[G_0]$ . The inclusions compare complete history-based controllers; they do not assume that a current observation is a sufficient state. Centralized training searches within $\Pi_{\mathrm{dec}}$ . It cannot remove an execution-time information gap.

An exact one-decision information gap

Let $X,Y$ be independent $\operatorname{Bernoulli}(1/2)$ variables. Agent 1 observes only $X$ , agent 2 only $Y$ . Both choose binary actions, and

R=\mathbf 1\{A_1=A_2=X\mathbin\oplus Y\},

where $\oplus$ is exclusive-or. Conditional on $X=x$ , the parity $X\oplus Y$ is a fair bit independent of agent 1's private action draw. Thus, under every decentralized policy,

\begin{aligned} \mathbb E_{P_\pi}[R] &\le\mathbb P_{P_\pi}(A_1=X\oplus Y)\\ &=\sum_x\frac12\sum_a\pi_1(a\mid x) \underbrace{\mathbb P(X\oplus Y=a\mid X=x)}_{1/2} =\frac12. \end{aligned}

Both agents always choosing zero attain $1/2$ , so $J^*_{\mathrm{dec}}=1/2$ . A controller observing $(X,Y)$ chooses both actions equal to the parity and attains $J^*_{\mathrm{joint}}=1$ . Even a shared random seed independent of $(X,Y)$ cannot improve the decentralized bound: condition on the seed and repeat the same calculation.

If agents exchange their bits before acting over a noiseless, cost-free, bidirectional channel, both can compute the parity and attain one. The deployed information structure has changed. A centralized critic alone does not supply that channel.

Original Oliehoek, Spaan and Vlassis Figure 2 shows one Dec-Tiger agent's action-observation history tree on the left, with a deterministic policy highlighted in blue. On the right, the same policy is a smaller tree with actions at nodes and the agent's own hear-left or hear-right observations on branches. — **Figure 2. A local deterministic policy as a contingent plan.** Original figure from Oliehoek, Spaan and Vlassis (2008), Optimal and Approximate Q-value Functions for Decentralized POMDPs, Figure 2. These are one agent's local contingencies, not a controller observing everyone's history. The observation-only simplification on the right applies to the deterministic policy shown; stochastic policies generally retain their own past actions. Full resolution ↗Figure excerpt for discussion; copyright remains with the authors/rightsholder.

The policy tree is a plan whose branches use observations available to one agent. Training can construct that plan centrally without revealing other agents' observations at execution. For the deterministic policy shown, previous actions are inferable from its observation path; our stochastic-policy definition retains past actions explicitly.

3. Why centralized critics can train local actors

Assume separate parameter vectors $\theta_i$ and positive, continuously differentiable action probabilities. The score

\psi_{i,t}=\nabla_{\theta_i} \log\pi_{i,\theta_i,t}(A_{i,t}\mid H_{i,t})

depends only on the local history and sampled action. If the actor is recurrent, this derivative includes its dependence through the unrolled actor memory.

Derive the history-based policy gradient

First, differentiating the finite trajectory sum and using the likelihood-ratio identity gives

\begin{aligned} \nabla_{\theta_i}J &=\sum_\tau G_0(\tau)\nabla_{\theta_i}p_\theta(\tau)\\ &=\mathbb E_{\tau\sim P_\theta} \left[G_0\sum_{t=0}^{T-1}\psi_{i,t}\right]. \end{aligned}

Environment factors have no parameter derivative; the product policy leaves agent $i$ 's own scores. Next, the conditional score mean is zero:

\mathbb E_{P_\theta}[\psi_{i,t}\mid H_t,S_t] =\sum_{a_i}\pi_{i,\theta_i,t}(a_i\mid H_{i,t}) \nabla_{\theta_i}\log\pi_{i,\theta_i,t}(a_i\mid H_{i,t}) =\nabla_{\theta_i}1=0.

Rewards received before $t$ are measurable from $H_t$ . The tower property therefore removes their products with $\psi_{i,t}$ . The remaining return is $\gamma^tG_t$ , so

\boxed{ \nabla_{\theta_i}J =\sum_{t=0}^{T-1}\gamma^t \mathbb E_{\tau\sim P_\theta}[\psi_{i,t}G_t]. }

This is the finite-horizon counterpart of the policy-gradient theorem. Other agents' parameters are held fixed in this partial derivative.

Three critics, the same expected gradient

At fixed $t,\theta$ , define exact conditional team returns:

\begin{aligned} q_{i,t}^\theta(h_i,a_i) &=\mathbb E_{P_\theta}[G_t\mid H_{i,t}=h_i,A_{i,t}=a_i],\\ Q_t^{\theta,H}(h,\boldsymbol a) &=\mathbb E_{P_\theta}[G_t\mid H_t=h,\boldsymbol A_t=\boldsymbol a],\\ Q_t^{\theta,HS}(h,s,\boldsymbol a) &=\mathbb E_{P_\theta}[G_t\mid H_t=h,S_t=s,\boldsymbol A_t=\boldsymbol a]. \end{aligned}

Conditional averages use the policy-induced joint law, not independently sampled histories, states, or actions. Values on zero-probability conditioning events are immaterial to these identities. Each critic predicts the same team return, using different information.

Write $X_t=(H_t,\boldsymbol A_t)$ and $W_t=(H_t,S_t,\boldsymbol A_t)$ . Because the score is measurable from either tuple, conditioning gives

\begin{aligned} \mathbb E_{P_\theta}[\psi_{i,t}G_t] &=\mathbb E_{P_\theta}\!\left[ \mathbb E_{P_\theta}[\psi_{i,t}G_t\mid W_t]\right]\\ &=\mathbb E_{P_\theta}[\psi_{i,t}Q_t^{\theta,HS}(W_t)]\\ &=\mathbb E_{P_\theta}[\psi_{i,t}Q_t^{\theta,H}(X_t)]\\ &=\mathbb E_{P_\theta}[\psi_{i,t}q_{i,t}^\theta(H_{i,t},A_{i,t})]. \end{aligned}

The latter equalities apply the same tower argument to $X_t$ and then $(H_{i,t},A_{i,t})$ . A centralized critic can therefore weight a local actor's training gradient without becoming an input to its deployed decisions. Lyu et al. analyze the history-based relationship in Theorem 1 and the history-state relationship in Theorem 6; here the sums and conditionals are explicitly finite-horizon.

Original Lowe et al. Figure 1. The red execution region contains local policies with their own observations and actions. The larger green training region includes agent-specific critics that can use joint information to train the policies. These critics are outside the execution region. — **Figure 3. Centralized training with decentralized execution.** Original figure from Lowe et al. (2017), Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments, Figure 1. Unlike a single shared-critic sketch, the original explicitly shows an agent-specific critic for each agent. This is an information arrangement, not a convergence guarantee. Full resolution ↗Figure excerpt for discussion; copyright remains with the authors/rightsholder.

Lowe et al., Section 4.1 use decentralized actors with agent-specific centralized critics. Their method also permits different agent rewards; our derivation specializes to a shared team objective and retains histories when observations are insufficient. The figure's local observation inputs are not, by themselves, proof of Markov sufficiency.

The equality concerns exact on-policy conditional values. It does not imply that independently fitted critics produce identical finite-sample updates, or that concurrent gradient ascent reaches a global optimum. Part 2's counterfactual baseline can also be used with suitable history-aware critics; its cancellation still requires independence from the current own action conditional on the information held fixed.

4. More critic information need not reduce gradient variance

Fix one decision time $t$ and a policy $\theta$ . Let $Z_{i,t}=(H_{i,t},A_{i,t})$ and define the random gradient contributions

\begin{aligned} D&=\psi_{i,t}q_{i,t}^\theta(Z_{i,t}), &C&=\psi_{i,t}Q_t^{\theta,H}(X_t),\\ U&=\psi_{i,t}Q_t^{\theta,HS}(W_t), &M&=\psi_{i,t}G_t. \end{aligned}

All four use the same draw $\tau\sim P_\theta$ . The previous conditioning argument establishes

\mathbb E[C\mid Z_{i,t}]=D,\qquad \mathbb E[U\mid X_t]=C,\qquad \mathbb E[M\mid W_t]=U.

For example, the law of total covariance gives

\begin{aligned} \operatorname{Cov}(C) &=\operatorname{Cov}\!\left(\mathbb E[C\mid Z_{i,t}]\right) +\mathbb E[\operatorname{Cov}(C\mid Z_{i,t})]\\ &=\operatorname{Cov}(D) +\mathbb E[\operatorname{Cov}(C\mid Z_{i,t})] \succeq\operatorname{Cov}(D). \end{aligned}

Here expectations and covariances use $P_\theta$ , and $B\succeq A$ means $B-A$ is positive semidefinite. Applying this identity to each nested conditioning yields

\boxed{ \operatorname{Cov}(M)\succeq\operatorname{Cov}(U) \succeq\operatorname{Cov}(C)\succeq\operatorname{Cov}(D). }

Conditioning replaces the return by its mean while retaining the information needed for the actor's score. Among these nested exact critics, the local critic averages over the most remaining randomness. This is Rao–Blackwellization: replacing a random estimator by its conditional mean cannot increase its covariance. The ordering agrees with Lyu et al.'s Theorem 2 and Theorem 7.

Check the inequality in Part 2's example

Reuse independent $A_1\sim\operatorname{Bernoulli}(1/4)$ , $A_2\sim\operatorname{Bernoulli}(1/2)$ and $R=A_1+4A_2$ . For agent 1's logit score $\psi=A_1-1/4$ ,

Q^H(A_1,A_2)=A_1+4A_2,\qquad q_1(A_1)=\mathbb E[R\mid A_1]=A_1+2.

Enumerating the four joint outcomes gives

\begin{aligned} \mathbb E[\psi Q^H]=\mathbb E[\psi q_1]&=\frac3{16},\\ \operatorname{Var}(\psi Q^H)&=\frac{555}{256},\\ \operatorname{Var}(\psi q_1)&=\frac{363}{256},\\ \operatorname{Var}(\psi Q^H)-\operatorname{Var}(\psi q_1) &=\mathbb E[\psi^2\operatorname{Var}(R\mid A_1)] =\frac3{16}\cdot4=\frac34. \end{aligned}

These comparisons concern single-time contributions with exact critics and no added baselines. They do not rank fitted algorithms or estimators that use baselines. Nor do they compare sums whose contributions are correlated across time. In particular, they do not contradict Part 2's smaller COMA variance after subtracting a teammate-dependent baseline. A centralized critic may still be easier to fit accurately. Fitting error and sampling variance with exact critics are separate questions.

5. Why the physical state can be insufficient for a critic

A history-aware actor may behave differently after two histories that end in the same physical state. Its future behavior depends on what it remembers. Consequently $S_t$ alone need not determine the future distribution when that actor controls the environment.

To isolate the issue, consider even an exact state-only conditional predictor

\overline Q_t^\theta(s,\boldsymbol a) =\mathbb E_{P_\theta}[G_t\mid S_t=s,\boldsymbol A_t=\boldsymbol a].

The tuple $(S_t,\boldsymbol A_t)$ does not generally determine $\psi_{i,t}$ . The tower property gives an expectation containing $\psi_{i,t}G_t$ , not a product of two conditional means:

\begin{aligned} &\mathbb E_{P_\theta}[\psi_{i,t}G_t] -\mathbb E_{P_\theta}[\psi_{i,t}\overline Q_t^\theta(S_t,\boldsymbol A_t)]\\ &\qquad=\mathbb E_{P_\theta} \left[\operatorname{Cov}_{P_\theta} (\psi_{i,t},G_t\mid S_t,\boldsymbol A_t)\right]. \end{aligned}

For vector scores, we take the covariance with the scalar return one coordinate at a time. This covariance need not be zero. A state-only Bellman recursion also needs a separate justification: a fitted temporal-difference fixed point need not equal the exact conditional mean of the return. Lyu et al., Sections 3.3 and 5 examine this distinction between state and history, and the resulting state-critic bias.

A two-decision counterexample

Agent 2 has only a no-op. Agent 1 initially observes a fair private bit $C\sim\operatorname{Bernoulli}(1/2)$ , independent of the physical state. The state only records the decision time; it does not retain this irrelevant environmental signal. At $t=0$ ,

A\mid C=c\sim\operatorname{Bernoulli}(p_c), \qquad p_c=\operatorname{sigmoid}(\theta_c),\qquad R_1=0.

The physical transition does not depend on $C$ or $A$ . At $t=1$ , the current observation is constant. A fixed continuation actor remembers $(C,A)$ and draws

B\mid C,A\sim\operatorname{Bernoulli} \left(\varepsilon+(1-2\varepsilon)\mathbf 1\{A=C\}\right), \qquad 0<\varepsilon<\tfrac12.

It receives $R_2=B$ and the episode ends. This is a valid Markov environment: the final reward depends only on the current action $B$ . Memory enters through the actor, not the environment. All nontrivial action probabilities are positive; only the first decision's parameters are varied.

The exact objective and first-decision history critic are

\begin{aligned} J(\theta) &=\gamma\left[\varepsilon+ (1-2\varepsilon)\frac{1-p_0+p_1}{2}\right],\\ Q_0^H(c,a) &=\gamma\left[\varepsilon+(1-2\varepsilon)\mathbf 1\{a=c\}\right]. \end{aligned}

At $p_0=p_1=1/2$ , direct differentiation gives

\boxed{ \frac{\partial J}{\partial\theta_0} =-\frac{\gamma(1-2\varepsilon)}8, \qquad \frac{\partial J}{\partial\theta_1} =\frac{\gamma(1-2\varepsilon)}8. }

Yet $C$ is independent of $A$ at this parameter point. Averaging over $C\mid A=a$ makes both state-only action values equal:

\overline Q_0(S_0,a)=\frac12Q_0^H(0,a)+\frac12Q_0^H(1,a)=\frac\gamma2.

The score for parameter $\theta_c$ is $\mathbf 1\{C=c\}(A-p_c)$ . Thus the state-only critic gives

\mathbb E\left[\mathbf 1\{C=c\}(A-p_c)\overline Q_0(S_0,A)\right] =\frac\gamma2\mathbb E[\mathbf 1\{C=c\}(A-p_c)]=0,

which misses both nonzero derivatives. These expectations use the explicit $C,A,B$ sampling law above. The example concerns substituting a state-only action-value predictor for the return. A state-dependent baseline subtracted from an otherwise valid return estimator can still cancel correctly; it is a different use of state information.

6. Retain actor memory, then remove training-only components

Using $(H_t,S_t,\boldsymbol A_t)$ retains both the environment state and the information determining future actor behavior. It satisfies the exact conditional-gradient identity in Section 3. A compact alternative may retain the actors' actual controller memories instead of full histories.

For fixed recurrent controllers, suppose

M_{i,t}=f_i(H_{i,t}),\qquad A_{i,t}\mid M_{i,t}\sim\pi_{i,t}(\cdot\mid M_{i,t}), \qquad M_{i,t+1}=F_i(M_{i,t},A_{i,t},R_{t+1},O_{i,t+1}).

The augmented variable $(t,S_t,M_{1,t},\ldots,M_{N,t})$ has controlled Markov dynamics: $K$ generates the next state, observations and reward, and the displayed deterministic updates generate the next memories. It therefore suffices to evaluate these fixed controllers. A learned recurrent memory need not be an optimal belief state; this statement concerns evaluating the policy it actually implements.

Value-based CTDE makes a different separation. For example, QMIX trains a joint value of the form

Q_{\mathrm{tot}}(h,s,\boldsymbol a) =m_s\bigl(q_1(h_1,a_1),\ldots,q_N(h_N,a_N)\bigr).

When the mixer is nondecreasing in every utility, Part 2's coordinatewise argument applies at each fixed $(h,s)$ :

a_i^*\in\arg\max_{a_i}q_i(h_i,a_i) \quad\Longrightarrow\quad \boldsymbol a^*\in\arg\max_{\boldsymbol a}Q_{\mathrm{tot}}(h,s,\boldsymbol a).

Original Rashid et al. Figure 2 shows a mixing network with absolute-value constrained nonnegative weights supplied by state-conditioned hypernetworks, the combination of N agent utilities into Q-total, and a recurrent agent network that produces action utilities from local observations and previous actions. — **Figure 4. QMIX's state-conditioned monotonic mixing network.** Original figure from Rashid et al. (2018), QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning, Figure 2. All three architecture panels are retained. Monotonicity concerns the utility inputs at fixed state, not the state input itself; the source actors use local histories. Full resolution ↗Figure excerpt for discussion; copyright remains with the authors/rightsholder.

The mixer and its state-conditioned hypernetwork are training components. Each executing agent needs only its local utility network and memory. This guarantee maximizes the fitted joint value; it does not establish that the true optimal team value satisfies the monotonic restriction. See Rashid et al., Section 4.

7. A batch CTDE actor–critic procedure

We use independent fitting and actor-update episodes to separate critic estimation from gradient evaluation. This simplified on-policy procedure illustrates information access; it is not a reproduction of MADDPG, COMA, or QMIX.

For a frozen actor and fitting batch $\mathcal D_{\mathrm{fit}}$ , regress complete reward-to-go targets:

\widehat w\in\arg\min_w \frac{1}{2|\mathcal D_{\mathrm{fit}}|} \sum_{\tau\in\mathcal D_{\mathrm{fit}}}\sum_{t=0}^{T-1} \left\{Q_{w,t}^{HS}(H_t,S_t,\boldsymbol A_t)-G_t\right\}^2.

At population level, unrestricted squared-error regression has the conditional-mean target $Q_t^{\theta,HS}$ . A restricted fitted network need not attain that target.

Algorithm 1 History-aware on-policy CTDE with separate fitting and gradient batches

Input: Training simulator; local-history actors $\theta^{(0)}$ ; critic class; rounds $K$ ; batch sizes $n_{\mathrm{fit}},n_{\mathrm{grad}}\ge1$ ; learning rates $\alpha_k$ .

Output: Local actors $\{\pi_{i,\theta_i^{(K)},t}\}$ and their local memory updates.

for $k=0,\ldots,K-1$ do
Freeze $\theta\gets\theta^{(k)}$ .
Collect independent batches $\mathcal D_{\mathrm{fit}}\sim P_\theta^{\otimes n_{\mathrm{fit}}}$ and $\mathcal D_{\mathrm{grad}}\sim P_\theta^{\otimes n_{\mathrm{grad}}}$ , executing every actor from its local history only.
Compute $G_t$ on fitting episodes and fit $Q_{\widehat w,t}^{HS}$ by the displayed regression.
for each agent $i$ do
Set $\widehat g_i\gets n_{\mathrm{grad}}^{-1}\sum_{\tau\in\mathcal D_{\mathrm{grad}}}\sum_{t=0}^{T-1}\gamma^t\psi_{i,t}Q_{\widehat w,t}^{HS}(H_t,S_t,\boldsymbol A_t)$ , treating the critic as fixed.
Set $\theta_i^{(k+1)}\gets\theta_i^{(k)}+\alpha_k\widehat g_i$ .
end for
end for
Export only the actors, local preprocessing, and local memory updates; omit the critic and training-only state inputs.

At a fixed round, condition on $\theta$ and the fitted critic. Independence of the gradient batch gives

\begin{aligned} \mathbb E_{\mathcal D_{\mathrm{grad}}\sim P_\theta^{\otimes n_{\mathrm{grad}}}} [\widehat g_i\mid\mathcal D_{\mathrm{fit}}]-\nabla_{\theta_i}J =\sum_{t=0}^{T-1}\gamma^t\mathbb E_{P_\theta} \left[\psi_{i,t}\{Q_{\widehat w,t}^{HS}(W_t)-Q_t^{\theta,HS}(W_t)\} \mid\mathcal D_{\mathrm{fit}}\right]. \end{aligned}

An exact critic makes this gradient error zero. Splitting the episodes lets us evaluate the fitted critic conditionally, but its approximation and estimation errors remain. Data from a different behavior policy would additionally require an off-policy argument, and the simultaneous actor updates have no general global-convergence guarantee.

8. Check the deployed decision rule

For each actor, deployment must preserve the information restriction

\pi_{i,t}(\cdot\mid h_i,h_{-i},s) =\pi_{i,t}(\cdot\mid h_i).

Hold its local history fixed and change training-only state or teammates' histories: the actor's action distribution and local memory update must remain unchanged. Centralized features can still leak through preprocessing, shared runtime encoders, or a memory update that reads another agent's observations. Such a dependency violates the displayed restriction even if the final action head is labeled decentralized.

Methodological choice	Available during training	Needed for one deployed decision
Centralized actor–critic	Joint histories/actions; optionally state and actor memories	That actor's local history or controller memory
VDN/QMIX	Team reward and joint utility loss; QMIX may use state in its mixer	Local utility network, local memory, local argmax
Explicit communication	The declared messaging protocol and its timing	Local information plus messages actually received before acting

CTDE permits broader information for learning, not broader information for acting. History-aware conditional values preserve the expected gradient; access to the physical state alone does not establish that property. Part 4 uses this information distinction when developing multi-agent policy-gradient and actor–critic updates.

References

Oliehoek, F. A., Spaan, M. T. J., and Vlassis, N. (2008). Optimal and Approximate Q-value Functions for Decentralized POMDPs. Journal of Artificial Intelligence Research, 32, 289–353. Sections 2.1–2.4 define the model, histories and policies; Figures 1–2 distinguish joint dynamics from local contingent plans.
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017). Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. NeurIPS. Section 4.1 and Figure 1 describe decentralized actors and centralized critics; the present note does not reproduce its deterministic/off-policy optimizer.
Lyu, X., Baisero, A., Xiao, Y., Daley, B., and Amato, C. (2023). On Centralized Critics in Multi-Agent Reinforcement Learning. Journal of Artificial Intelligence Research, 77, 295–354. The journal article was published in 2023; this author-posted arXiv copy was uploaded in 2024. Sections 4–6 analyze history, state, and history-state critics.
Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. (2018). QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. ICML. Section 4 separates centralized value fitting from decentralized greedy utility selection.

The parity example, finite-horizon conditional-expectation derivations, and two-decision memory counterexample are worked constructions for this note. They are not new benchmark results or reproductions of the papers' experiments.