zbMATH — the first resource for mathematics

Measure-valued differentiation for stationary Markov chains. (English) Zbl 1278.90428
Summary: We study general state-space Markov chains that depend on a parameter, say,\( {\theta}\). Sufficient conditions are established for the stationary performance of such a Markov chain to be differentiable with respect to \({\theta}\). Specifically, we study the case of unbounded performance functions and thereby extend the result on weak differentiability of stationary distributions of Markov chains to unbounded mappings. First, a closed-form formula for the derivative of the stationary performance of a general state-space Markov chain is given using an operator-theoretic approach. In a second step, we translate the derivative formula into unbiased gradient estimators. Specifically, we establish phantom-type estimators and score function estimators. We illustrate our results with examples from queueing theory.

90C40 Markov and semi-Markov decision processes
Full Text: DOI