1 Introduction↩︎

In a classical Dynkin game, two players interact through their stopping strategies. The game ends when one of the players stops, and the payoffs depend on which player stops first. Since its introduction in [1], the Dynkin game has been studied in various directions. Works on zero-sum Dynkin games include [1], [2] in discrete time, [3]–[5] in continuous time, [6]–[8] with randomized strategies, [9] within non-Markovian settings, and [10] under model uncertainty. The nonzero-sum version of the game has been studied in [11]–[13] and more recently in [14]–[16], among others. It is worth noting that all the works mentioned above focus on Nash equilibrium in the sense that in the stopping games the two players choose their stopping strategies simultaneously. In many practical situations, however, strategic interactions are inherently asymmetric: one party may commit to a policy in advance, while the other responds optimally after observing this commitment, as in regulatory interventions, contract design, or oligopoly market with a dominant firm. Motivated by this, we study a variant of the Dynkin game in which the two players determine their stopping strategies sequentially, giving rise to a Stackelberg (leader–follower) stopping game. This framework provides a natural extension of standard Dynkin game problems, such as game options or convertible bonds, to settings with asymmetric timing.

Time inconsistency in Stackelberg games was first observed in [17], where the author studies a dynamic game with one dominant player and multiple nondominant players. Under open-loop and closed-loop policies, the original plan may no longer be optimal as time progresses, leading to a temptation for the dominant player to deviate from the initial plan. As the author notes, “only the feedback solution has the characteristic that the original plan will not be changed under replanning” [17]. As we point out in Section 2.2, such feedback solution corresponds to our equilibrium strategy defined in Definition 3. More recently, [18] provides an alternative perspective on time inconsistency in Stackelberg games by reformulating the leader’s problem as an optimal control problem for forward–backward stochastic differential equations. The key feature is that the follower’s optimal response, which depends on the entire strategy announced by the leader, enters the leader’s problem endogenously. As a result, the leader’s continuation optimization problem at the next step generally differs from the problem evaluated at the initial time. This leads to intrinsic time inconsistency even in the absence of non-exponential discounting or other exogenous sources of inconsistency. A detailed analysis of this phenomenon in a concrete Stackelberg game model is given in [19]. In the present paper, the same mechanism appears in the Stackelberg stopping framework; see Proposition 2 and Remark 13 for explicit illustrations.

In the literature of time inconsistency, the seminal work [20] proposes three approaches to deal with time inconsistency. The first approach is to consider a precommitment strategy, a strategy that is optimal with respect to (w.r.t.) the initial preference. The second approach is to use a naive strategy, under which the agent keeps solving the new problem based on her current preference and thus keeps updating her strategy. The third approach is to look for a time-consistent equilibrium strategy: given that all the future selves follow this strategy, the current self has no incentive to deviate. Recently there has been a lot of research on equilibrium strategies for time-inconsistent problems. See e.g., [21]–[26] and references therein for time-inconsistent control, and [27]–[32], among others, for time-inconsistent stopping. Time-consistent equilibrium strategies for Stackelberg games have received relatively less attention. We refer to [17]–[19], [33], [34] and the included references for the related literature.

In this paper, we consider a Stackelberg version of the two-player nonzero-sum Dynkin game in discrete time. In the game, Player 1 (the leader, she) announces her (randomized) stopping strategy first, and then Player 2 (the follower, he) chooses his stopping strategy correspondingly. We do not impose any restriction on the payoff processes for both players and the game is not necessarily of war-of-attrition type. The underlying process is modeled by a time-homogeneous Markov process. We do not include any exogenous source of time inconsistency (such as non-exponential discounting or non-linear functional of expectation) that commonly appears in a single-agent time-inconsistent problem. The absence of exogenous time inconsistency highlights that the time inconsistency in our framework arises intrinsically from the leader–follower structure of the stopping game.

We start with the finite-horizon setup for the purpose of motivation and concepts. We first consider pure stopping strategies and show in an deterministic example (Proposition 2) that the leader’s problem is time inconsistent. We then define and compare different notions of solutions, including leader’s (pure) precommitment and time-consistent equilibrium strategies for the Stackelberg game, as well as the Nash equilibrium for the classical Dynkin game. We demonstrate via an example (Proposition 4) that these notions of solutions are distinct from each other, by showing that leader’s initial strategies are different under different solution concepts. Furthermore, three classical solution concepts in dynamic Stackelberg games are contrasted with the precommitment and equilibrium strategies. Their connections and distinctions are emphasized in Remark 7. Next, we consider randomized precommitment and time-consistent equilibrium strategies. We observe that randomized strategies may yield strictly larger payoffs for the leader than any pure strategy does (Remark 10). We also present a deterministic two-period example (Proposition 9) indicating that an optimal precommitment strategy may not exist. The nonexistence of an optimal strategy is due to the discontinuity of the follower’s best response w.r.t. the leader’s strategy.

Motivated by the examples and results in the finite-horizon setup, we next consider an infinite-horizon time-homogeneous framework in which the payoff processes are exponentially discounted and the underlying Markov process takes values in a general Polish space. We first investigate the leader’s optimal value induced by randomized path-dependent (precommitment) strategies. The problem is formally defined in 10 and the leader’s problem is given by 11 . Since the leader’s problem is time inconsistent, the dynamic programming principle (DPP) cannot be directly applied. Inspired by [19], [35], we view the follower’s continuation utility as the state variable of the leader’s value function. Combined with a careful analysis of the admissible range of the follower’s utility, this allows us to recover a DPP for the leader’s value function (Theorem 14).

We then study randomized equilibrium strategies for the Stackelberg stopping game where the players adopt randomized Markov stopping policies. The problem is formulated in 16 , for which we introduce two versions of equilibrium, the feedback equilibrium (Definition 15) and exact equilibrium (Definition 16). Their connection and distinction are discussed in Remark 19. Roughly speaking, the exact equilibrium is a more restricted version of feedback equilibrium that requires that the follower always chooses the earliest optimal stopping time. We construct an example (Proposition 20) showing that exact equilibria may fail to exist even in the presence of randomized strategies. The nonexistence is caused by the discontinuity of the follower’s best-response map with respect to the leader’s strategy, since the follower’s optimal response is binary and non-randomized. In the finite-state setting, we establish the existence of feedback equilibria (Proposition 29), but it remains an open question for the general Polish space.

Motivated by this difficulty, we introduce an entropy-regularized Stackelberg stopping game in which the follower’s payoff is perturbed by an entropy term (see problem 17 ). The regularization induces a continuous randomized response rule of logit type and yields a tractable regularized equilibrium framework. Entropy regularization and logit-type responses are widely used in optimization, learning, and quantal-response models, where they can be interpreted in terms of imperfect optimization, exploration, or information-processing effects (see, for example [36] and [37]). In the present paper, the regularization primarily serves to smooth the follower’s best-response map and to obtain a well-posed equilibrium notion, regular equilibrium (Definition 19), in the general Polish-space setting. The notions of exact equilibrium, feedback equilibrium, regular equilibrium, and \(\varepsilon\)-equilibrium correspond respectively to equilibrium concepts for the original Stackelberg game, its relaxed feedback formulation, the entropy-regularized game, and approximate equilibria for the original game.

Under the entropy regularization, we establish the existence of regular equilibrium strategies (Theorem 26). We further prove that, when the regularization parameter is sufficiently small, the resulting regularized equilibria induce \(\varepsilon\)-equilibria (Definition 18) for the original Stackelberg stopping game (Proposition 27). In the special case where the Markov process has a finite state space, we show that as the entropy parameter converges to zero, regular equilibria admit limit points corresponding to feedback equilibria for the original game (Proposition 29). However, the compactness and continuity arguments used in the finite-state setting do not extend to general Polish spaces, and the existence of (exact) feedback equilibria in the general Polish-space framework remains open (see Remark 31). An application in the game option is provided in Section 3.4.2.

Our paper makes a very novel and conceptual contribution to the research of stopping games. Previous works on Dynkin games focus on the existence and construction of Nash equilibrium under which players make decisions simultaneously. To the best of our knowledge, this is the first work that investigates the Stackelberg version of Dynkin games. Our paper introduces a general framework for Stackelberg stopping games in discrete time. We show that the leader–follower structure generates intrinsic time inconsistency and leads to new phenomena that do not arise in classical Dynkin games with simultaneous actions. In particular, we characterize the leader’s value function under precommitment strategies, identify nonexistence phenomena for exact randomized equilibria, establish existence results for entropy-regularized equilibria in general Polish spaces, and derive approximation connections between regularized equilibria and the original stopping game.

Our paper also contributes to the literature of time-inconsistent problems. Most of the works on equilibrium strategies for time-inconsistent problems consider single-agent cases, and results for multi-agent settings are relatively limited. The investigation of equilibria for time-inconsistent games is conceptually richer, since players not only play against other agents but also their future selves, and thus the problem can be viewed as a two-layer game. Let us mention the works [38], [39] on time-inconsistent stopping games. [38] studies a Dynkin game and [39] analyzes a mean-field stopping game. In [38], [39] players take action simultaneously, the time inconsistency originates from non-exponential discounting, and time-consistent equilibria are investigated. In contrast, in our framework the time inconsistency comes from the leader-follower game structure, and both precommitment and equilibrium strategies are considered. The model and methods involved in our paper are also very different from those in [38], [39].

The rest of the paper is organized as follows. Section 2 provides motivation and introduces the main concepts via a finite-horizon setting. We illustrate the time inconsistency of the Stackelberg stopping game with a deterministic example and introduce the notions of precommitment and equilibrium strategies, comparing them with the Nash equilibrium in the classical Dynkin game (Section 2.1). Further comparison with classical solution notions is discussed in Section 2.2. We then extend the discussion to randomized strategies (Section 2.3). Section 3 focuses on randomized strategies in the infinite-horizon setup. In Section 3.1, we characterize the leader’s value induced by randomized precommitment (path-dependent) strategies and show that the optimal value may not be attained due to the discontinuity of the follower’s response. Section 3.2 discusses equilibrium strategies for the Stackelberg stopping game with randomized Markov policies. The definitions of feedback equilibrium, exact equilibrium and \(\varepsilon\)-equilibrium are given in this section, and a counterexample is presented to show that exact randomized equilibria may fail to exist. Section 3.3 studies the entropy-regularized Stackelberg stopping game. We define regular equilibrium, establish its existence in the general Polish-space setting, and prove that regular equilibria induce \(\varepsilon\)-equilibria for the original Stackelberg stopping game. The finite-state case and an application to game options are discussed in Section 3.4. All proofs for Section 3 are collected in Section 4.

2 The Finite-Horizon Stackelberg Stopping Game↩︎

2.1 Pure Strategies and Motivation↩︎

Denote \(\mathbb{N}:=\{1,2,\dotso\}\), \(\mathbb{N}_0:=\{0,1,2,\dotso\}\) and \([[s,t]]:=\{s,s+1,\dotso,t\}\) for \(s,t\in\mathbb{N}_0\) with \(s\leq t\). Take \(N,T\in\mathbb{N}\). Let \(X = (X_t)_{t\in \mathbb{N}_0}\) be a time-homogeneous discrete-time Markov chain with the state space \(\mathbb{X}:= \{1,2,\cdots,N\}\). The transition matrix of \(X\) is denoted by \(\pi = (\pi_{xy})_{x,y \in \mathbb{X}}\). For \(t \in [[0,T]]\) and \(x\in \mathbb{X}\), let \(X^{t,x} = (X^{t,x}_s)_{s\in [[t,T]]}\) be the Markov chain starting from time \(t\) with the initial state \(X_t = x\). Let \(\mathbb{P}_{t,x}\) and \(\mathbb{E}_{t,x}\) be the probability measure and expectation conditioned on \(X_t = x\). Let \(\mathbb{F}^{X} := \{\mathcal{F}^{X}_t\}_{t\in [[0,T]]}\) be the filtration generated by \(X\). For each \(t \in [[0,T]]\), we denote by \(\mathbb{T}_t\) the set of all \(\mathbb{F}^{X}\)-stopping times \(\tau\) with \(t\leq\tau\leq T\). For \(t = 0\), we simply write \(\mathbb{T}_0\) as \(\mathbb{T}\). Denote the path space of the Markov chain \(X\) over time \([[t, T]]\) by \[\Omega_{t} := \left\{\omega = (\omega_t, \cdots, \omega_T) \mid \omega_s \in \mathbb{X}, \forall s = t, \cdots, T\right\}.\] For \(t = 0\), we write \(\Omega_0\) as \(\Omega\) for simplicity. Let \(f_i,g_i,h_i:[[0,T]]\times\mathbb{X}\mapsto\mathbb{R}\) be payoff functions for \(i=1,2\).

Consider a discrete-time, finite-horizon, two-player stopping game in which both players observe the Markov chain \(X\) and choose stopping times to maximize their expected payoffs. The stopping game has a finite horizon \(T\). Let \(\tau\) and \(\rho\) be the stopping times chosen by Player 1 (leader, she) and Player 2 (follower, he), respectively. For each \(t\in [[0,T]]\) and \(x \in \mathbb{X}\), given a stopping time \(\tau \in \mathbb{T}_t\) selected by Player 1, Player 2 responds with a stopping time \(\rho \in \mathbb{T}_t\) that maximizes his expected payoff: \[J_2(t,x;\tau,\rho) : = \mathbb{E}_{t,x}[F_2(\tau,\rho)],\] where the payoff function \(F_2\) is given by \[F_2(\tau, \rho) = f_2(\rho,X_{\rho})\mathbb{1}_{\{\rho<\tau\}} + g_2(\tau,X_{\tau})\mathbb{1}_{\{\tau<\rho\}} +h_2(\rho,X_{\rho})\mathbb{1}_{\{\rho = \tau\}}.\] For simplicity of the analysis, we assume that Player 2 always chooses the earliest stopping time that achieves the maximum expected payoff. We denote this best response by \(\rho^*=\rho^*(\tau)\), which is uniquely given by \[\rho^*(\tau) = \rho^*_{t,x}(\tau) := \inf\left\{s \ge t \mid F_2(\tau, s) = \sup_{\rho \in \mathbb{T}_s} \mathbb{E}\left[F_2(\tau, \rho)|\mathcal{F}_s^X\right]\right\}.\] Given Player 2’s optimal response \(\rho^*\), Player 1 chooses \(\tau\) to maximize her own expected payoff: \[J_1(t,x;\tau,\rho^*) : = \mathbb{E}_{t,x}[F_1(\tau,\rho^*)],\] with payoff function \[F_1(\tau, \rho) = f_1(\tau, X_{\tau})\mathbb{1}_{\{\tau<\rho\}} + g_1(\rho, X_{\rho})\mathbb{1}_{\{\rho<\tau\}} +h_1(\tau, X_{\tau})\mathbb{1}_{\{\rho = \tau\}}.\] The Stackelberg stopping game reduces to an optimization problem for the leader: \[\label{eq:leader95problem} \sup_{\tau \in \mathbb{T}_t} V_{t,x}(\tau),\quad\text{with}\quad V_{t,x}(\tau) := J_1(t,x; \tau, \rho^*(\tau)).\tag{1}\]

We define the optimal solution in Problem 1 as follows.

The term precommitment strategy originates in the literature on time inconsistency. It refers to a strategy that is optimal when chosen at the initial time, but may cease to be optimal from the perspective of future selves. As we see in the following proposition, Problem  1 exhibits time inconsistency. For each fixed \(t\in [[0,T]]\) and \(x\in \mathbb{X}\), the stopping policy \(\tau^*_{t,x}\) is optimal in a static sense. That is, the decision is made at time \(t\) without taking into account the change of preferences in the future. Therefore, we refer to \(\tau^*_{t,x}\) as the precommitment strategy.

In the deterministic example above, time inconsistency arises inherently from the Stackelberg structure of the game. In contrast to the precommitment strategy, an equilibrium strategy accounts for the changing preferences of future selves. By interpreting the optimal stopping problem as a game among multiple temporal selves—each representing the decision maker at a different point in time. An equilibrium strategy is one in which the current self has no incentive to deviate, assuming all future selves adhere to it. We assume that each temporal self at \(t \in [[0,T]]\) adopts a pure Markov stopping policy that determines when to stop, defined as follows.

We formalize the notion of no-regret strategy as a subgame perfect Nash equilibrium in the leader’s intrapersonal game across time as follows.

In the finite-horizon setting, a pure equilibrium strategy can be constructed via backward induction. By definition, an equilibrium strategy is Markovian and, in general, differs from a precommitment strategy. In what follows, we compare these two notions with the concept of a Nash equilibrium in the classical Dynkin game, which has been extensively studied. For completeness, we first provide the definition of a Nash equilibrium in the Dynkin version of our game.

2.2 Comparison with notions in classical game theory↩︎

In classical dynamic Stackelberg games, three solution concepts are commonly considered: open-loop equilibrium, closed-loop equilibrium, and feedback equilibrium; see [17], [40]. In this subsection, we introduce stopping-time counterparts of these concepts and contrast them with the two solution notions discussed earlier, namely precommitment and equilibrium.

Open-loop equilibrium. We first describe the admissible policy sets of the two players. Define \[\mathcal{P}:= \left\{P: \omega \mapsto (P_0(\omega_0), P_1(\omega_0,\omega_1), \ldots, P_T(\omega)) \in \{0,1\}^{T+1} \;\middle|\; P_t \le P_{t+1}\;\forall t,\;P_T = 1 \right\}.\] There is a one-to-one correspondence between \(\mathcal{P}\) and the set \(\mathbb{T}\) of stopping times. Indeed, for any \(\tau \in \mathbb{T}\), the process \((\mathbf{1}_{\{\tau \le t\}})_{t\in [[0,T]]}\) belongs to \(\mathcal{P}\), and for any \(P \in \mathcal{P}\), the stopping time \[\tau^P := \inf\{t \ge 0 : P_t = 1\}\] belongs to \(\mathbb{T}\).

At time \(0\), the leader’s open-loop policy set is \(\mathcal{P}\). The follower’s policy depends on the leader’s choice and is therefore described by a mapping \(P \mapsto Q(P)\). Denote \[\mathcal{Q}:= \{ Q : \mathcal{P}\to \mathcal{P}\}.\]

If \(\tau_0^*\) is a precommitment strategy at time \(0\), define \[P_t^* = \mathbb{1}_{\{\tau_0^* \le t\}}, \qquad Q_t^*(P) = \mathbb{1}_{\{\rho^*(\tau^P) \le t\}}, \quad \forall t \in [[0,T]]\] where \(\rho^*(\tau^P)\) denotes the follower’s optimal response to \(\tau^P\). Then \((P^*,Q^*)\) forms an open-loop Stackelberg equilibrium. On the other hand, an open-loop Stackelberg equilibrium does not in general impose a specific tie-breaking rule (such as choosing the earliest optimal stopping time), and can therefore be viewed as a broader concept than precommitment strategy.

Closed-loop equilibrium. The set of pure Markov stopping policies corresponds to closed-loop policies for the leader: \[\mathcal{P}^{cl} := \{0,1\}^{[[0,T]] \times \mathbb{X}}:=\{p: [[0,T]] \times \mathbb{X}\ni (t,x) \mapsto p(t,x) \in \{0,1\}\}.\] The follower’s closed-loop policy set is \[\mathcal{Q}^{cl} := \{ q : \mathcal{P}^{cl} \to \mathcal{P}^{cl} \}.\]

In deterministic settings as in the previous subsection example, open-loop and closed-loop equilibria coincide. However, in general stochastic environments, we have \(\mathcal{P}^{cl} \times \mathcal{Q}^{cl} \subset \mathcal{P}\times \mathcal{Q}\).

Feedback equilibrium. Although the equilibrium strategy \(p^*\) in Definition 3 is Markovian, the pair \((p^*,q^*)\) does not form a closed-loop equilibrium in the above sense. Instead, it corresponds to a feedback Stackelberg equilibrium, where the follower’s action at time \(t\) depends on both the current state and the leader’s current action.

In the finite-horizon case, the feedback Stackelberg equilibrium is obtained via backward induction and coincides with the equilibrium notion introduced in Definition 3.

2.3 Randomized Strategies↩︎

In this subsection, we study relaxed/randomized strategies within the Stackelberg stopping framework. We formalize these notions in the finite-horizon setting and illustrate them using the deterministic example introduced earlier. A more detailed theoretical analysis is developed in the infinite-horizon setup in Section 3.

Let \(Y = (Y_t)_{t \in \mathbb{N}_0}\) be a sequence of independent uniformly distributed random variables on interval \([0,1]\), which are also independent of the Markov chain \(X\). For each sequence \(p = (p_0,p_1,p_2,\cdots) \in [0,1]^{\infty}\), define \[\tau^{p,Y} := \inf \{t \ge 0 \mid Y_t \le p_t\}.\] Then \(\tau^{p,Y}\) is a stopping time w.r.t. the filtration generated by \(Y\). Each \(p_t \in [0,1]\) represents the probability that the process will stop at \(t\), given that it has not stopped before time \(t\).

Now we consider randomized stopping policies that map the path \(\omega = (\omega_0, \cdots, \omega_T)\in \Omega\) to a sequence \(p = (p_0, \cdots, p_T) \in [0,1]^{T+1}\).

We now allow both players to adopt randomized stopping policies. To model this, let \(Y = (Y_t)_{t \in \mathbb{N}_0}\) and \(Y' = (Y'_t)_{t \in \mathbb{N}_0}\) be sequences of independent random variables, each uniformly distributed on \([0,1]\), and assume that both \(Y\) and \(Y'\) are independent of the underlying Markov chain \(X\) and of each other. We denote the stopping policies of the leader and the follower by \(P \in \mathcal{R}_T\) and \(Q \in \mathcal{R}_T\), respectively. Then stopping times induced by these policies are given by \[\tau^P = \tau^{P, Y}, \quad \tau^Q = \tau^{Q,Y'}.\] Intuitively, the sequences \(Y\) and \(Y'\) serve as internal randomization devices that allow each player to make probabilistic stopping decisions in a way that can depend on both the history of the process X and their own private source of randomness.

In the Stackelberg stopping game, the leader announces her policy \(P \in \mathcal{R}_T\) first, then the follower will respond after observing whether the leader stops at time \(0\). That is the follower’s response depends on both \(P\) and the realization of random variable \(Y_0\). If the leader stops at \(t = 0\), then the follower solves the following problem. \[W_S(0,x) := \sup_{q \in [0,1] } q h_2(0, x) + (1-q) g_2(0,x),\] and the optimal response is given by \[Q_S(0,x) = \mathbb{1}_{\{h_2(0,x) \ge g_2(0,x)\}}.\] Here the subscript ‘S’ stands for stop. In what follows, we also use subscript ‘C’ to indicate continue.

If the leader does not stop at \(t = 0\), then the follower solves the following problem. \[W_C(0,x; \theta_x \circ P) := \sup_{Q \in \mathcal{R}_T } \mathbb{E}_{t,x}[F_2(\tau^P, \tau^Q) \mid \tau^P > 0].\]

Note that the utility \(W_C\) is a function of \(\theta_x \circ P\), since the stopping time \(\tau^P\) does not depend on \(P_0(\cdot)\) given that \(\tau^P > 0\). Moreover, for this fixed stopping policy \(\theta \circ P \in \mathcal{R}_{1:T}\), the utility of Player 2 satisfies the following Bellman’s equation.

\[W_C(0,x; \theta_x \circ P) = \sup_{q \in [0,1]} q f_2(0,x) + (1-q) \mathbb{E}_{0,x}[W(1,X_1;\theta_x \circ P)],\] where \(W(t,x;\tilde{P})\) denotes the total utility of the follower at time \(t\), given \(X_t=x\) and the leader’s strategy \(\tilde{P}\in\mathcal{R}_{t:T}\). The optimal response is given by \[Q_C(0,x; \theta_x \circ P) = \mathbb{1}_{\{f_2(0,x) \ge \mathbb{E}_{0,x}[W(1,X_1; \theta_x \circ P)]\}}.\] The follower’s value at time \(0\) given \(X_0=x\) is given by \[W(0,x; P) = P_0(x)W_S(0,x) +(1-P_0(x))W_C(0,x,\theta_x \circ P).\]

Repeating the above process, we have that for each \(t \in [[0,T-1]]\), \(y \in \mathbb{X}\) and \(P\in\mathcal{R}_T\), \[\begin{align} &W(t,y; P^{t}) = P_t(y)W_S(t,y) +(1-P_t(y))W_C(t,y,P^{t+1});\\ &W_S(t,y) = \max\{h_2(t,y), g_2(t,y)\}; \quad \quad W_C(t,y; P^{t+1}) = \max\{f_2(t,y), \mathbb{E}_{t,y}[W(t+1,X_{t+1}; P^{t+1})] \};\\ &Q_S(t,y) = \mathbb{1}_{\{h_2(t,y) \ge g_2(t,y)\}}; \quad\quad Q_C(t,y; P^{t+1})= \mathbb{1}_{\{f_2(t,y) \ge \mathbb{E}_{t,y}[W(t+1,X_{t+1}; P^{t+1})]\}}. \end{align}\]

The above reasoning shows that the follower’s optimal response consists of two parts: \(Q_S\) and \(Q_C\). The stop-scenario response \(Q_S\) depends only on time \(t\) and current state \(X_t\), while the continue-scenario response \(Q_C\) depends on the leader’s remaining policy \(P^{t+1}\) in addition to time \(t\) and current state \(X_t\). For simplicity, we write \(Q^* = Q^*(P) = (Q_S, Q_C(P))\) as the total optimal response of the follower. Then the stopping time induced by \(Q^*\) is given by \[\tau^{Q^*} := \inf\left\{t \ge 0 \mid Y'_t \le Q_S(t,X_t) \mathbb{1}_{\{\tau^P \le t\}} + Q_C(t,X_t; P^{t+1})\mathbb{1}_{\{\tau^P > t\}}\right\},\]

For leader’s stopping policy \(P\) defined on time interval \([[t,T]]\), the optimal response \(Q^*\) by the follower can be defined in the same manner as before. To avoid cumbersome notation, we continue to use the symbol \(Q^*\) to denote this response. With a slight abuse of notation, we define the value function \[V_{t,x}(P) := \mathbb{E}_{t,x}[F_1(\tau^P, \tau^{Q^*(P)})], \quad t\in [[0,T]], \, x \in \mathbb{X}, \, P\in \mathcal{R}_{t:T}.\]

As in the pure strategy case, the Stackelberg game can be formulated as an optimal control problem from the perspective of the leader: \[\label{eq:optimal95control95problm} \sup_{P \in \mathcal{R}_T} V_{0,x}(P).\tag{2}\]

Since the set of randomized stopping policies strictly contains the set of pure stopping policies, it is natural to expect that a randomized policy may yield a strictly higher utility for the leader than any pure policy does. On the other hand, as we will see in the proof of Proposition 9, a randomized precommitment strategy may not exist. Intuitively, this nonexistence arises from the discontinuity of the follower’s best response mapping \(Q^*(P)\) w.r.t. the leader’s policy \(P\).

In contrast to the randomized precommitment strategy, the existence of a randomized equilibrium strategy—a generalization of the pure equilibrium strategy, as defined in the following paragraph—is guaranteed in the finite-horizon setting.

3 The Infinite-Horizon Stackelberg Stopping Game↩︎

Motivated by the results and examples in the finite-horizon setting, we undertake a comprehensive analysis of both the precommitment and equilibrium strategies with randomization to address time inconsistency in the Stackelberg stopping game under an infinite-horizon framework. Moreover, we generalize the Markov chain by extending its state space from a finite set to an uncountable one. There are three main reasons for focusing on the infinite-horizon case.

First, the time-homogeneous structure of the Markov chain and the use of exponential discounting simplify the notation and allow for a clearer investigation of the time-inconsistent nature of the Stackelberg game.

Second, the analysis of precommitment strategies in the infinite-horizon setting closely parallels that of the finite-horizon case, making it a natural extension.

Third, the study of equilibrium strategies in the infinite-horizon case leads to novel and nontrivial challenges, particularly due to the need for fixed-point arguments. In contrast, equilibrium strategies in the finite-horizon setting can be constructed via backward induction, rendering the analysis relatively straightforward.

Let us first state the infinite-horizon set-up. Consider a Polish space \(\mathbb{X}\) and denote by \(\mathcal{B}(\mathbb{X})\) the family of Borel sets in \(\mathbb{X}\). Let \(X = (X_t)_{t\in \mathbb{N}_0}\) be a time-homogeneous Markov chain taking values in \(\mathbb{X}\). The transition kernel of \(X\) is denoted by \(\Pi\), i.e., \[\mathbb{P}(X_{t+1} \in A | X_t = x) = \int_A \Pi(x, dy), \quad \forall A \in \mathcal{B}(\mathbb{X}), x \in \mathbb{X}, t \in \mathbb{N}_0.\] Denote by \(\mathbb{P}_x\) and \(\mathbb{E}_x\) the measure under which \(X_0 = x \in \mathbb{X}\) a.s. and the associated expectation, respectively. The canonical space of the Markov chain on infinite horizon is denoted by \[\Omega := \{\omega = (\omega_0, \omega_1, \cdots) \mid \omega_t \in \mathbb{X}, \forall t \in \mathbb{N}_0\},\] and the filtration generated by \(\{X_t\}_{t\in \mathbb{N}_0}\) is denoted by \(\mathcal{F}^{X}\).

In the Stackelberg stopping game, the leader (Player 1) and the follower (Player 2) choose their stopping strategies to maximize their expected payoff, which depends on the stopping times of both players. Denote the stopping time of the leader and the follower by \(\tau\) and \(\rho\), respectively. Denote \[\label{eq:F1F2} \begin{align} &F_1(\tau, \rho) := f_1(X_{\tau})\mathbb{1}_{\{\tau<\rho\}} + g_1(X_{\rho})\mathbb{1}_{\{\rho<\tau\}} + h_1(X_\tau)\mathbb{1}_{\{\rho=\tau\}},\\ &F_2(\tau, \rho) := f_2(X_{\rho})\mathbb{1}_{\{\tau>\rho\}} + g_2(X_{\tau})\mathbb{1}_{\{\rho>\tau\}} + h_2(X_\rho)\mathbb{1}_{\{\rho=\tau\}}, \end{align}\tag{4}\] where \(f_i, g_i, h_i, i \in \{1,2\}\) are Borel measurable real-valued functions on \(\mathbb{X}\).

For the remainder of the paper, we impose the following two assumptions.

As shown in the previous section, the leader-follower structure in the Stackelberg stopping game introduces time inconsistency into the leader’s optimization problem. In the infinite-horizon setting, despite time-homogeneity, it becomes necessary to consider path-dependent stopping policies. We revisit the concept of randomized stopping policy and randomized Markov stopping policy in Definition 8 and Definition 9, and formulate them in the infinite-horizon framework as follows.

We endow the set of randomized stopping policies \(\mathcal{R}\) with the topology induced by the distance function \[\label{eq:distance} d(P,P'):= \sum_{t=0}^\infty\frac{1}{2^t} \int_{X^{t+1}}|P_t(y_0,y_1,\dotso,y_t)-P'_t(y_0,y_1,\dotso,y_t)| \mu(d y_0) \cdots \mu(d y_t),\quad P,P'\in\mathcal{R}.\tag{5}\] Under such topology, \(\mathcal{R}\) is a Polish (complete separable metric) space.

As in the finite-horizon case, we assume that \(Y = (Y_t)_{t \in \mathbb{N}_0}\) and \(Y' = (Y'_t)_{t \in \mathbb{N}_0}\) are the private sources of randomness for the leader and follower, respectively. They are two independent sequences of independent uniformly distributed random variables on \([0,1]\), which are also independent of \(X\). The stopping time induced by the stopping policy \(P \in \mathcal{R}\) and the private randomness source \(Y\) is given by \[\label{eq:tau95bigP} \tau^P = \tau^{P,Y} := \inf\{t\ge 0 \mid Y_t \le P_t(X_0, \cdots, X_t)\},\tag{6}\] which is a stopping time w.r.t. the filtration generated by \(X\) and \(Y\).

Since the follower makes his move based on the information of both the leader’s strategy and the leader’s current state, stopping or continuing. the follower’s strategy \(Q\) consists of two parts, \(Q_S \in \mathcal{R}\) and \(Q_C \in \mathcal{R}\), which represent the path-dependent randomized stopping policies when the leader stops and when the leader continues at the current step, respectively. Thus we denote the set of the follower’s policies by \(\mathcal{R}^2\). Specifically, the stopping time corresponding to the follower’s policy \(Q = (Q_S, Q_C) \in \mathcal{R}^2\) is given by \[\label{eq:tau95bigQ} \tau^{Q_S, Q_C} = \tau^{Q,Y'} :=\inf\{t\ge 0 \mid Y'_t \le Q_{S,t}(X_0, \cdots, X_t) \mathbb{1}_{\{\tau^P \le t\}} + Q_{C,t}(X_0, \cdots, X_t) \mathbb{1}_{\{\tau^P > t\}} \}.\tag{7}\] Note that \(\tau^Q\) is a stopping time with respect to the filtration generated by \(X,Y'\) and the process \(\left(\mathbb{1}_{\{\tau^P \le t\}}\right)_{t \in \mathbb{N}_0}\).

Similar to , the induced stopping time of the leader’s and the follower’s Markov policy \(p \in \mathcal{R}_0\) and \((r,q) \in \mathcal{R}_0^2\) are given by \[\begin{align} &\tau^p = \tau^{p,Y} := \inf\{t \ge 0 \mid Y_t \le p_{X_t}\}; \tag{8}\\ &\tau^{r,q} = \tau^{r,q,Y'} :=\inf\{t\ge 0 \mid Y'_t \le r_{X_t} \mathbb{1}_{\{\tau^p \le t\}} + q_{X_t} \mathbb{1}_{\{\tau^p > t\}} \}.\tag{9} \end{align}\]

We continue to use the notation of the one-step shift operator \(\theta_x\) and the more general \(t\)-step shift operator \(\theta_{\omega_0, \cdots,\omega_{t-1}}\) as introduced in Remark 8.

For convenience, we introduce the notation \[\mathcal{R}^{\mathbb{X}} := \{ \boldsymbol{P}: \mathbb{X}\to \mathcal{R}\text{ is Borel measurable.}\}\] Throughout the remainder of the paper, we use boldface \(\mathbf{P}\) to denote elements of \(\mathcal{R}^{\mathbb{X}}\), \(P\) to denote a stopping policy in \(\mathcal{R}\), \(P_0\) (or \(p\)) to denote a Markov stopping policy in \(\mathcal{R}_0\), and \(P_0 \oplus \mathbf{P}\) to denote the pasting of \(P_0 \in \mathcal{R}_0\) and \(\mathbf{P} \in \mathcal{R}^{\mathbb{X}}\).

The proofs of the results in this section are deferred to Section 4.

3.1 Precommitment Strategy↩︎

We begin by formulating the Stackelberg stopping game in which both players adopt randomized stopping policies from the set \(\mathcal{R}\). Although the formulation of the Stackelberg stopping game in the infinite-horizon setting closely parallels that of the finite-horizon case, we find it useful to restate the model here to emphasize the time-homogeneous structure and to introduce notation that will be used in the subsequent analysis. With a slight abuse of notation, the objective functions \(J_1\) and \(J_2\) in Section 3.1 and Section 3.2 are functions of stopping policies \((P,(Q_S, Q_C))\) and \((p, (r,q))\), respectively, while \(J_1\) and \(J_2\) are functions of stopping times \((\tau, \rho)\) in Section 2.

The infinite-horizon Stackelberg stopping game is characterized by the following objective function pair \((J_1, J_2)\): \[\label{eq:game95precomit} \begin{align} \text{Follower: } & \sup_{(Q_S, Q_C) \in \mathcal{R}^2} J_2(x, P, (Q_S, Q_C));\\ \text{Leader: }& \quad\,\,\, \sup_{P \in \mathcal{R}} \quad J_1(x, P, (Q_S^*, Q_C^*)), \end{align}\tag{10}\] with \[\begin{align} &J_i(x, P, (Q_S, Q_C)) := \mathbb{E}_x\left[ \delta_i^{\tau^{P} \wedge \tau^{(Q_S, Q_C)}} F_i(\tau^{P}, \tau^{(Q_S, Q_C)})\right], \quad i \in \{1,2\}, x\in \mathbb{X}, ;\\ & (Q_S^*, Q_C^*) = \arg\max_{(Q_S, Q_C) \in \mathcal{R}^2} J_2(x, P, (Q_S, Q_C)), \end{align}\] where \(\delta_1, \delta_2 \in (0,1)\) are the discount factors of the leader and the follower, respectively, and \(\tau^P\), \(\tau^{Q_S,Q_C}\), and \(F_i\) are defined in 6 , 7 , and 4 , respectively.

If we further assume that the follower always chooses the earliest optimal stopping time, then the follower’s optimal response \(Q^* = (Q^*_S, Q_C^*)\) is a function of the leader’s strategy \(P\). Denote this response function by \(Q^*(P)\), the above Stackelberg stopping game 10 can be reduced to the following optimization problem for the leader, \[\label{eq:leader95problem95infT} \sup_{P \in \mathcal{R}} V(x,P), \text{ with } V(x, P) := J_1(x, P, Q^*(P)).\tag{11}\]

As discussed in Section 2, the problem 11 is time inconsistent. The solution to this problem is defined as precommitment strategy (see Definition 14) and characterized in Proposition 14.

3.1.1 The follower’s problem↩︎

Denote by \(W: = W(x,P)\) the follower’s maximal expected payoff function before observing the leader’s state (i.e., stopping or continuing) at the current time, which depends on current state \(X_0 = x\) and the leader’s policy \(P \in \mathcal{R}\). Then we have \[W(x,P) = P_0(x) W_S+ (1-P_0(x)) W_C,\] where \(W_S\) and \(W_C\) are the follower’s maximal expected payoff functions when the leader stops and when the leader continues at the current step, respectively. Specifically, \[\begin{align} &W_S := \sup_{Q \in \mathcal{R}^2} \mathbb{E}_x\left[\delta_2^{\tau^P\wedge\tau^{Q}}F_2(\tau^P, \tau^{Q}) \mid \tau^P = 0\right]; \quad \quad W_C := \sup_{Q \in \mathcal{R}^2} \mathbb{E}_x\left[\delta_2^{\tau^P\wedge\tau^{Q}}F_2(\tau^P, \tau^{Q}) \mid \tau^P > 0\right]. \end{align}\]

Again we assume that the follower always chooses the earliest stopping time that attains the maximal expected payoffs \(W_S\) and \(W_C\). By the definition of \(F_2\), we have that \(W_S\) is a function of current state \(x\): \[\label{eq:W95S} W_S = W_S(x) := \sup_{Q_{S,0}(x) \in [0,1]} Q_{S,0}(x)h_2(x) + (1- Q_{S,0}(x))g_2(x) = \max \{h_2(x), g_2(x)\},\tag{12}\] of which the optimal response is given by \(Q^*_S(x) = \mathbb{1}_{\{h_2(x) \ge g_2(x)\}}.\) Consequently, the follower only needs to solve\[W_C := \sup_{Q_C \in \mathcal{R}} \mathbb{E}_x\left[\delta_2^{\tau^P\wedge\tau^{Q^*_S, Q_C}}F_2(\tau^P, \tau^{Q^*_S, Q_C}) \mid \tau^P > 0\right].\] Given \(\tau^P > 0\), the quantity \(P_0\) will not affect the follower’s utility, and \(W_C\) depends on the leader’s strategy starting from the next step \(t = 1\), namely \(\theta_x \circ P\), in the following way: \[W_C = W_C(x, \theta_{x}\circ P) := \sup_{Q_C\in \mathcal{R}}Q_{C,0}(x) f_2(x) + (1-Q_{C,0}(x) )\delta_2\mathbb{E}_x\left[ J_2 \left(X_1, {\theta_x \circ P},({Q^*_S, \theta_x \circ Q_C})\right)\right].\] For the remainder of this subsection, we omit the superscript \(Q^*_S\) and denote \(Q_C\) by \(Q\) whenever no confusion arises. With a slight abuse of notation, denote\[\label{eq:J2new} J_2(x,P,Q): = J_2 \left(x, P,{Q^*_S, Q_C}\right), \quad \forall x \in \mathbb{X},\,\, \forall P, Q \in \mathcal{R}.\tag{13}\]

Thanks to the time-homogeneity of the Markov process \(X\) and the exponential discounting, we derive in Lemma 2 recursive equations satisfied by \(W_C\), which will be used in the subsequent analysis. Before that, let us establish a continuity property in Lemma 1 under the following assumption.

Since each Markov policy \(p\in \mathcal{R}_0\) can be regarded as a degenerate path-dependent policy \(P \in \mathcal{R}\) with \(\theta_x \circ P = P = p\) for any \(x \in \mathbb{X}\), the following corollary follows directly from Lemma 2.

Before studying the leader’s optimization problem, we prove some properties of \(W_C\) as a function of \(P\). For each \(x \in \mathbb{X}\), define the set of all feasible continuation values \(W_C(x,P)\) by \[D_x := \{W_C(x,P) \mid P \in \mathcal{R}\}.\]

Lemma 3 shows that for each \(x \in \mathbb{X}\), the function \(P \mapsto W_C(x,P)\) is continuous. This results is used in the proof of Lemma 4, which shows that for each \(x \in \mathbb{X}\), the set \(D_x\) is a closed interval, and its endpoints can be characterized using the DPP.

Define \[D:=\{(x,w)\in\mathbb{X}\times\mathbb{R}:\;x\in\mathbb{X},w\in D_x\}.\] From the above lemma we know that \(D\) is Borel measurable and \(D_x\) is closed for each \(x\in\mathbb{X}\). Then by the Kuratowski-Ryll-Nardzewski measurable selection theorem [41], there exists a Borel measurable function \(w(\cdot):\mathbb{X}\mapsto\mathbb{R}\) such that \(w(x)\in D_x\) for any \(x\in\mathbb{X}\). Denote the set of such Borel measurable selectors by \(D^{\mathbb{X}}\), i.e., \[D^{\mathbb{X}}: = \{w: (\mathbb{X}, \mathcal{B}(\mathbb{X})) \to (\mathbb{R}, \mathcal{B}(\mathbb{R})) \mid w(x) \in D_x, \,\, \forall x \in \mathbb{X}\}.\]

3.1.2 The leader’s problem↩︎

From the preceding section, the follower adopts the following optimal response functions\[Q^*_S(x) = \mathbb{1}_{\{h_2(x)\ge g_2(x)\}}, \quad Q^*_C(x,P) = \mathbb{1}_{\{f_2(x)=W_C(x,P)\}},\,\, \forall x\in \mathbb{X}, P \in \mathcal{R}.\] Given the resulting optimal response function \(Q^*(P) = (Q^*_S, Q^*_C(P))\), we define the leader’s utility level at stopping policy \(P \in \mathcal{R}\) and initial state \(x\) by \(V(x,P)\). This utility admits the decomposition \[\label{eq:v95def} V(x,P) := J_1\left(x, \tau^P, \tau^{Q^*(P)}\right) = P_0(x) V_S(x) + (1-P_0(x)) V_C(x, \theta_x \circ P),\tag{14}\] where \[V_S(x) = \mathbb{E}_x\left[F_1(\tau^P, \tau^{Q^*}) \mid \tau^P = 0 \right]\] denotes the utility conditional on the leader stopping at the current step, and \[V_C(x, \theta_x \circ P) = \mathbb{E}_x \left[\delta_1^{\tau^P\wedge\tau^{Q^*}}F_1(\tau^P, \tau^{Q^*}) \mid \tau^P > 0 \right]\] denotes the utility conditional on the leader continuing at the current step, and it depends on \(P\) only through \(\theta_x \circ P\).

It is straightforward that\[V_S(x) = h_1(x) \mathbb{1}_{\{h_2(x) \ge g_2(x)\}} + f_1(x) \mathbb{1}_{\{h_2(x) < g_2(x)\}}.\] By taking conditional expectation, we obtain the following recursive equation satisfied by the leader’s continuation utility \(V_C\): for each \(P \in \mathcal{R}, x \in \mathbb{X}\), \[\begin{align} V_C(x,P) =&g_1(x) \mathbb{1}_{\{W_C(x,P) = f_2(x)\}} + \delta_1\mathbb{E}_x[V(X_1,P)] \mathbb{1}_{\{W_C(x,P) > f_2(x)\}}. \end{align}\]

A similar argument to that used in the proof of Lemma 1 yields the following representation for the utility function \(V(x,P)\): \[\begin{align} V(x,P) =& \sum_{t = 0}^{\infty} \delta_1^t \mathbb{E}_x\left[\prod_{k =0}^{t-1} \left( (1-P_k)(1- Q^*_k \right)\left[V_S(X_t) P_t + g_1(X_t) (1-P_t) Q^*_t\right]\right], \end{align}\] where \(Q^*_k = Q^*_C(x_k, \theta_{x_0, \cdots, x_{k-1}}\circ P) = \mathbb{1}_{\{f_2(x_k) = W_C(x_k, \theta_{x_0, \cdots, x_{k-1}}\circ P)\}}, \forall k \in \mathbb{N}_0\) is Borel measurable.

It is easy to see that when \(P \mapsto Q^*(P)\) is continuous, the map \(P \mapsto V(x,P)\) is continuous, and hence the map \((x,P) \mapsto V(x,P)\) is Borel measurable. By a monotone class argument, the same mapping is Borel measurable if \(P \mapsto Q^*(P)\) is Borel measurable. Consequently, \((x,P) \mapsto V_C(x,P)\) is also Borel measurable, and the expectation \(\mathbb{E}_x[V(X_1,P)]\) admits the following integral representation: \[\mathbb{E}_x[V(X_1,P)] = \int_{\mathbb{X}}[P_0(y) V_S(y) + (1-P_0(y)) V_C(y, \theta_y \circ P)] \Pi(x, dy).\] Similarly, for Markov stopping policy \(p \in \mathcal{R}_0\), the continuation value \(V(x,p)\) satisfies recursive equation

\[\label{eq:v95markov95bellman} V_C(x,p) = g_1(x) \mathbb{1}_{\{W_C(x,p) = f_2(x)\}} + \delta_1 \left(\int_{\mathbb{X}}[p_y V_S(y) + (1-p_y)V_C(y, p)]\Pi(x,dy) \right) \mathbb{1}_{\{W_C(x,p) > f_2(x)\}}.\tag{15}\]

To study the precommitment strategy, we draw inspiration from [19], [35]. Thanks to equation 14 , the leader’s optimal control problem ?? can be simplified as \[\sup_{P \in \mathcal{R}} V(x,P) = \max\left\{V_S(x), \sup_{P\in \mathcal{R}}V_C(x, P)\right\}.\] In the remainder of this section, we focus on characterizing \(\sup_{P\in \mathcal{R}}V_C(x, P)\). The key idea is to view the leader’s continuation value \(V_C\) as a function of the follower’s continuation value \(W_C\). The entire infinite sequence \(P\) is effectively summarized by a scalar value \(W_C(x,P)\) at each state \(x\), capturing the follower’s continuation value. This perspective allows us to formulate a Bellman equation on the space of all feasible values of \(W_C(x, P)\), leading to a tractable characterization of the leader’s utility function \(v(w)\) as given in Proposition 14.

For each \(x \in \mathbb{X}, w \in D_x\), we define the leader’s utility function \(v(x,w)\) by \[v(x,w) := \sup_{P\in \mathcal{L}(x,w)} V_C(x,P),\] where the set \(\mathcal{L}(x,w)\) is given by \[\mathcal{L}(x,w) := \{P \in \mathcal{R}\mid W_C(x, P) = w\}.\] Since \(P \to W_C(x, P)\) is continuous, for each feasible \(w \in D_x\), the set \(\mathcal{L}(x,w) \subset \mathcal{R}\) is non-empty and closed. Intuitively, \(v(x,w)\) represents the maximum expected utility that the leader can achieve at state \(x\), under the constraint that the follower’s continuation value is exactly \(w\). It is easy to check that the optimization problem \(\sup_{P \in \mathcal{R}} V_C(x,P)\) can be reduced to \(\sup_{w \in D_x} v(x,w)\), as indicated by the following result.

In the following theorem, we show that \(v(x,w)\) can be characterized by a Bellman’s equation ?? .

3.2 Exact Equilibrium Strategy↩︎

In addition to the precommitment strategy in the preceding subsection, a common approach to addressing time inconsistency is to consider an equilibrium strategy, from which the player has no incentive to deviate, assuming all their future selves adopt the same strategy. In our infinite horizon setting, we assume that all future selves adopt a randomized Markov stopping policy \(p \in \mathcal{R}_0\) as defined in Definition 13. When restricted to the Markov stopping policies, the infinite-horizon Stackelberg stopping game is characterized by the following objective function pair \((J_1, J_2)\): \[\label{eq:game95exact95equilibrium} \begin{align} \text{Follower: } & \sup_{(r, q) \in \mathcal{R}_0 ^2} J_2(x, p, (r, q));\\ \text{Leader: }& \,\,\,\sup_{p \in \mathcal{R}_0 } \quad J_1(x, p, (r^*, q^*)), \end{align}\tag{16}\] with \[\begin{align} &J_i(x, p, (r, q)) := \mathbb{E}_x\left[ \delta_i^{\tau^{p} \wedge \tau^{(r, q)}} F_i(\tau^{p}, \tau^{(r, q)})\right], \quad i \in \{1,2\}, x\in \mathbb{X},\\ & (r^*, q^*) = \arg\max_{(r, q) \in \mathcal{R}^2} J_2(x, p, (r, q)),\footnotemark \end{align}\] where \(\delta_1, \delta_2 \in (0,1)\) are the discount factors of the leader and the follower, respectively, and \(\tau^p\), \(\tau^{r,q}\), and \(F_i\) are defined in 8 , 9 , and 4 , respectively.

As discussed in subsection 2.2, the notion of equilibrium strategy in the Stackelberg stopping game corresponds to a feedback Stackelberg equilibrium where the follower’s strategy is formulated as a function of the leader’s strategy. We revisit these concepts in the infinite horizon setting and discuss their connection and distinction as follows.

Here the first condition guarantees that \((r, \boldsymbol{q})\) is a best response function for the follower without assuming that the follower always chooses the earliest optimal stopping time, and the second condition means that the leader has no incentive to deviate from strategy \(p\) at the current step, given that her future selves all stick to strategy \(p\). By 14 , the follower’s response \(\boldsymbol{q}\) essentially depends only on the leader’s future strategy since the follower can observe that the leader will continue at the current state. We thus conclude that \[J_1(x, {p' \oplus_1 p}, (r,\boldsymbol{q}(p' \oplus_1 p))) = J_1(x, p' \oplus_1 p, (r,\boldsymbol{q}(p))), \quad p'\in \mathcal{R}_0, (r,\boldsymbol{q}) \in \mathcal{R}_0 \times \mathcal{Q}.\] Therefore, the above definition can be reduced to the following equivalent characterization.

Note that a feedback equilibrium does not impose any tie-breaking rule for the follower, which implies that the follower optimal response strategy \((r, \boldsymbol{q})\) may not be unique. However, in the preceding setting, we assume that the follower always chooses the earliest stopping time that attains the maximal expected payoffs. This assumption is reasonable, as choosing the earliest optimal stopping time avoids additional risk and simplifies the definition of the follower’s best response function as follows.

In contrast to the finite-horizon case, we cannot rely on backward induction to construct an equilibrium strategy in the infinite-horizon setting. The following proposition shows that the existence of a randomized exact equilibrium strategy is not guaranteed.

Since an exact equilibrium strategy may not exist, and the existence of feedback equilibria is difficulty to establish due to the multiplicity of the follower’s best responses w.r.t. the leader’s strategy (see Remark 31), we turn to the notion of an \(\varepsilon\)-equilibrium.

Similar to the characterization of Definition 15 in Proposition 18, we obtain an \(\varepsilon\)-feedback equilibrium for free if we have an \(\varepsilon\)-equilibrium defined as follows.

Comparing conditions ?? and ?? with conditions ?? and ?? (which are satisfied by an exact equilibrium, see Remark 19), we see that the \(\varepsilon\)-equilibrium from Definition 18 can be interpreted as an approximation of the exact (feedback) equilibrium strategy in the original (unregularized) game. In the next subsection, the existence of \(\varepsilon\)-equilibrium is established via considering an entropy-regularized Stackelberg stopping game, in which an entropy term is added to the follower’s optimization problem. This regularization yields a continuous best-response function, allowing us to establish the existence of a randomized equilibrium in the entropy-regularized game.

3.3 Regular Equilibrium Strategy↩︎

An entropy-regularized Stackelberg game with parameter \(\lambda >0\) is characterized by the objective function pair \((J_1,J_2^{\lambda})\):

\[\label{eq:game95regular95equilibrium} \begin{align} \text{Follower: } & \sup_{(r, q) \in \mathcal{R}_0 ^2} J^{\lambda}_2(x, p, (r, q));\\ \text{Leader: }& \,\,\,\sup_{p \in \mathcal{R}_0 } \quad J_1(x, p, (r^{*,\lambda}, q^{*,\lambda})), \end{align}\tag{17}\] with \[\begin{align} &J^{\lambda}_2(x, p, (r, q)) := \mathbb{E}_x\left[ \delta_2^{\tau^{p} \wedge \tau^{(r, q)}} F_2(\tau^{p}, \tau^{(r, q)})\right] + \lambda \mathbb{E}_x\left[ \sum_{t = 0}^{\tau^{{p}} \wedge \tau^{{q},{r}}} \delta_2^t \mathcal{H}(q_{X_t} \mathbb{1}_{\{\tau^{{p}} >t \}} + r_{X_t} \mathbb{1}_{\{\tau^{{p}} = t \}} )\right],\\ &J_1(x, p, (r, q)) := \mathbb{E}_x\left[ \delta_1^{\tau^{p} \wedge \tau^{(r, q)}} F_1(\tau^{p}, \tau^{(r, q)})\right], \quad x\in \mathbb{X}, p\in \mathcal{R}_0, (r,q) \in \mathcal{R}_0^2;\\ & (r^{*,\lambda}, q^{*,\lambda}) := \arg\max_{(r, q) \in \mathcal{R}^2} J^{\lambda}_2(x, p, (r, q)), \end{align}\] where \(\delta_1, \delta_2 \in (0,1)\) are the discount factors of the leader and the follower, respectively, \(\tau^p\), \(\tau^{r,q}\), and \(F_i\) are defined in 8 , 9 , and 4 , respectively, and \(\mathcal{H}\) is Shannon’s entropy defined by \[\mathcal{H}(q)= -q \log(q)-(1-q) \log(1-q),\quad q \in [0,1].\]

3.3.1 The follower’s problem↩︎

For a fixed constant \(\lambda > 0\), denote the follower’s value function by \[W^{\lambda}(x,{p}): = \sup_{(r,q) \in \mathcal{R}_0^2} J_2^{\lambda}(x, {p},(r,q) ),\] which can be decomposed as \[W^{\lambda}(x, {p}) = p_x W^{\lambda}_S(x) + (1-p_x) W^{\lambda}_C(x,{p}),\] with \[\label{eq:w95lambda95s} W^{\lambda}_S(x) := \sup_{r_x \in [0,1]} r_x h_2(x) + (1-r_x) g_2(x) + \lambda \mathcal{H}(r_x),\tag{18}\] and \[W^{\lambda}_C(x,{p}) := \sup_{{q}, {r} \in \mathcal{R}_0} q_x f_2(x)+(1-q_x)\delta_2\mathbb{E}_x[J^{\lambda}_2(X_1, p, (r,q))] + \lambda \mathcal{H}(q_x).\]

Thanks to the concavity of \(\mathcal{H}\), we obtain the following result by straightforward computation.

Recall that the set of randomized Markov (stationary) stopping policies is defined by \[\mathcal{R}_0 := \{p: (\mathbb{X}, \mathcal{B}(\mathbb{X})) \to ([0,1], \mathcal{B}([0,1])) \}.\]

Under Assumption 1, we have that \(\mathcal{R}_0 \subset L^{\infty}(\mathbb{X}, \mu)\), which is compact in weak-* topology \(\sigma(L^{\infty}(\mathbb{X},\mu), L^1(\mathbb{X}, \mu))\). We say a sequence of function \(\{p^n\}_{n\in \mathbb{N}} \in \mathcal{R}_0\) converges to \(p \in \mathcal{R}_0\), i.e., \[p^n \xrightarrow{w*} p,\] if and only if for any test function \(\phi \in L^1(\mathbb{X}, \mu)\),\[\int_{\mathbb{X}} p^n(y)\phi(y)\mu(dy) \to \int_{\mathbb{X}} p(y)\phi(y)\mu(dy).\]

The following lemma will be used repeatedly in establishing continuity properties.

The next corollary follows directly from Proposition 22.

3.3.2 The leader’s problem↩︎

Given the follower’s optimal response \((r^{*,\lambda}, q^{*,\lambda}(p))\) to the leader’s strategy \(p\), in equations ?? and ?? , the entropy-regularized game 17 now can be reduced to the leader’s problem: \[\label{eq:leader95problem95regular} \sup_{p \in \mathcal{R}_0}V^{\lambda}(x, p), \text{ with } V^{\lambda}(x, p) := J_1(x, p, (r^{*,\lambda},q^{*,\lambda}(p))) \quad \forall x \in \mathbb{X}, p\in \mathcal{R}_0.\tag{19}\]

Following similar reasoning as in Corollary 1, the value function \(V^{\lambda}\) in 19 can be decomposed as \[V^{\lambda}(x, p) = p_x V^{\lambda}_S(x) + (1-p_x) V^{\lambda}_C(x, p),\] where the stopping value \(V^{\lambda}_S(x)\) is given explicitly by \[\label{eq:vs95lambda} V^{\lambda}_S(x) := \mathbb{E}_x[F_1(\tau^{p}, \tau^{r^{*,\lambda},q^{*,\lambda}}) \mid \tau^{p} = 0] = r^{*,\lambda}_x h_1(x) + (1-r^{*,\lambda}_x) f_1(x);\tag{20}\] and the continuation value \(V^{\lambda}_C(x, p)\) satisfies iteration equation \[V^{\lambda}_C(x, p) := \mathbb{E}_x[\delta_1^{\tau^{p} \wedge \tau^{r^{*,\lambda},q^{*,\lambda}}}F_1(\tau^{p}, \tau^{r^{*,\lambda},q^{*,\lambda}}) \mid \tau^{p} > 0] = q^{*,\lambda}_x(p) g_1(x) + (1-q^{*,\lambda}_x(p)) \delta_1 \mathbb{E}_x[V^{\lambda}(X_1, p)].\]

When the leader deviates from the strategy \(p\) to \(p'\oplus_1 p\), the follower’s response remains unchanged, as it depends only on the leader’s current state and future strategy. The corresponding value function of the leader is therefore given by \[\label{eq:v95lambda95decompose} V^{\lambda}(x,p'\oplus_1 p) = p'_x V^{\lambda}_S(x) + (1-p'_x)V^{\lambda}_C(x, p).\tag{21}\]

Analogously to the exact equilibrium defined in Definition 16, we introduce the notion of a regular equilibrium in the entropy-regularized Stackelberg stopping game.

For any fixed \(x\), the value \(V^{\lambda}(x, p'\oplus_1 p)\) depends on \(p'\) only through the single component \(p'_x\); see 21 . This observation leads to the following proposition, which provides a sufficient condition for the above inequality to hold for almost all \(x \in \mathbb{X}\).

The following proposition is a key step to establish the existence of a regular equilibrium.

Thanks to Theorem 26, although the existence of an exact equilibrium for the original problem 16 is not guaranteed, the existence of a regular equilibrium ensures that an approximation to the exact equilibrium (i.e., an \(\varepsilon\)-equilibrium) always exists.

3.4 Special Cases and Application Example↩︎

3.4.1 Convergence of regular equilibrium in the finite state space↩︎

In this subsection, we consider the special case in which the state space \(\mathbb{X}\) is finite. Without loss of generality, let \(\mathbb{X}= \{1, \cdots ,N\}\) for some \(N \in \mathbb{N}\). When the state space is finite, as the entropy parameter \(\lambda \to 0\), the family of regular randomized equilibria \(\{p^{\lambda}\}_{\lambda>0}\) admits at least one limit point \(p^*\). Moreover, any such limit point is an exact feedback equilibrium as defined in Definition 15 for the unregularized problem 16 . To illustrate this, we explicitly construct an exact feedback equilibrium for the counterexample in the proof of Proposition 20. In contrast, these observations do not extend to the case of an uncountable state space. This limitation highlights the necessity of introducing the entropy regularization term in the general setting considered in the previous section.

Recall that the follower essentially only need to solve for his optimal response conditional on the leader continuing at the current step. When the leader stops at the current step, the follower’s optimal response is independent of the leader’s strategy \(p\) and can be determined directly. Accordingly, throughout the remainder of this subsection we define the follower’s response in this case by \[\label{eq:r95star95new} r^*_x = \begin{cases} 1, & \text{if } g_2(x) < h_2(x),\\[0.3em] \frac{1}{2}, & \text{if } g_2(x) = h_2(x),\\[0.3em] 0, & \text{if } g_2(x) > h_2(x), \end{cases} \qquad \forall x \in \mathbb{X}.\tag{22}\] In contrast to the earliest optimal stopping time \(\mathbb{1}_{\{g_2(x) \le h_2(x)\}}\), the policy \(r^*\) defined in 22 is a natural choice when the earliest-stopping restriction is lifted, as it coincides with the limit of \(r^{*,\lambda}\) in ?? as \(\lambda \to 0\), and also satisfies the requirement in Proposition 18. Moreover, for any sequence \(\{\lambda_n\}_{n \in \mathbb{N}}\) satisfying \(\lambda_n \to 0\), we have \[\label{eq:vs95limit} \lim_{n \to \infty} V_S^{\lambda_n}(x) = \lim_{n \to \infty}r^{*,\lambda_n}_x h_1(x) + \bigl(1 - r^{*,\lambda_n}_x\bigr) f_1(x) = r^{*}_x h_1(x) + \bigl(1 - r^{*}_x\bigr) f_1(x) = V_S(x).\tag{23}\]

The following proposition shows that the family of regular randomized equilibria \(\{(p^{\lambda}, (r^{*, \lambda},q^{*, \lambda})\}_{\lambda>0}\) admits at least one limit point \((p^*, (r^*,q^*))\) that satisfies ?? and ?? , and hence prove the existence of exact feedback equilibrium.

3.4.2 Application to game option with a dominant player↩︎

Let us consider a Stackelberg variant of the game option example in [14]. In the classical setting of game options, not only does the holder have the right to exercise the option at a chosen time, but the issuer is also allowed to recall or cancel the contract. In the Stackelberg variant, we assume that the issuer is the dominant player and announces her strategy in advance.

Let \(X_t\) denote the price of the underlying asset. Suppose that the issuer stops at time \(\tau\) and the holder stops at time \(\rho\). The payoff received by the holder from the issuer is given by \[\Gamma(\tau, \rho) := L(X_{\rho}) \mathbb{1}_{\rho<\tau} + U(X_{\tau}) \mathbb{1}_{\rho>\tau} + M(X_{\tau}) \mathbb{1}_{\tau = \rho}.\] Here, \(L(X_{\rho})\) (resp. \(U(X_{\tau})\), \(M(X_{\tau})\)) denotes the amount received by the holder when he stops before (resp. after, simultaneously with) the issuer.

As discussed in [14], we introduce utility functions \(\varphi_1\) and \(\varphi_2\) for the issuer and the holder, respectively. The Stackelberg stopping game can then be formulated as problem 16 , with payoff functions given by \[\begin{align} &f_1(x) = \varphi_1(-U(x)), \quad g_1(x) = \varphi_1(-L(x)), \quad h_1(x) = \varphi_1(-M(x));\\ &f_2(x) = \varphi_2(U(x)), \quad g_2(x) = \varphi_2(L(x)), \quad h_2(x) = \varphi_2(M(x)). \end{align}\]

The following theorem shows that, under suitable conditions, this Stackelberg stopping game admits an exact equilibrium.

The following example satisfies the condition in the theorem above.

4 Proof of Main Results↩︎

4.1 Proofs in Section 3.1↩︎

4.2 Proof in Section 3.2↩︎

4.3 Proofs in Section 3.3↩︎

4.4 Proof of Section 3.4↩︎

References↩︎

---

1. For each \(p \in \mathcal{R}_0,x\in \mathbb{X}\), the quantity \(\boldsymbol{q}_x(p)\) represents the conditional probability of stopping given that the current state is \(x\), the leader’s strategy is p, and the leader will continue at current step; and the quantity \(r_x\) represents the conditional probability of stopping given that the current state is \(x\), and the leader stops at the current step. (In this case, the follower’s decision no longer depends on the leader’s strategy.)↩︎

2. In this context, the term ‘exact’ is introduced to distinguish this solution from the ‘regular’ randomized equilibrium strategy obtained via entropy regularization (see Definition 19). When necessary, we also refer to the feedback equilibrium as the ‘exact’ feedback equilibrium.↩︎

3. Assumption 1 holds when the state space is finite. We can simply choose the reference distribution to be \(\mu(x) = 1/N, \forall x \in \mathbb{X}\) and the kernel to be \(\pi(x,y) = N\mathbb{P}(X_{t+1}= y | X_t = x ), \forall x,y \in \mathbb{X}\).↩︎