Beyond Smoothed Analysis: Analyzing the Simplex Method by-the-book
October 24, 2025
Abstract
Narrowing the gap between theory and practice is a longstanding goal of the algorithm analysis community. To further progress our understanding of how algorithms work in practice, we propose a new algorithm analysis framework that we call by-the-book analysis. In contrast to earlier frameworks, by-the-book analysis not only models an algorithm’s input data, but also the algorithm itself. Results from by-the-book analysis are meant to correspond well with established knowledge of an algorithm’s practical behavior, as they are meant to be grounded in observations from implementations, input modeling best practices, and measurements on practical benchmark instances. We apply our framework to the simplex method, an algorithm which is beloved for its excellent performance in practice and notorious for its high running time under worst-case analysis. The simplex method similarly showcased the previous state of the art framework smoothed analysis (Spielman and Teng, STOC’01). We explain how our framework overcomes several weaknesses of smoothed analysis and we prove that under input scaling assumptions, feasibility tolerances and other design principles used by simplex method implementations, the simplex method indeed attains a polynomial running time. Our results provide analytical justification for these features which are common to all high-quality simplex method implementations.
1 Introduction↩︎
There is a longstanding need for frameworks explaining the performance of algorithms whose users—whether they are engineers, operations researchers or mathematicians—know that they perform very well for their applications, but for which traditional worst-case analysis is too pessimistic. The simplex method for linear programming (LP) problems is an example of such an algorithm: it was a driving force in the development of electronic computers [1] as it was observed to perform efficiently; requiring a linear number of pivot steps in practice [2]–[7]. A long line of research attempts to explain this behavior theoretically by a suitable algorithm analysis framework. The project of explaining the performance of the simplex method is not only interesting from a mathematical point of view. There is also interest from practical users who are constantly facing new areas of application. They might fear that the algorithm will have worse performance in their new application domains. As such, there is a need for theoretical models showing that the simplex method is guaranteed to be efficient.
The simplex algorithm serves well as a showcase for the developments in the algorithm analysis community. Following the traditional line of algorithm analysis research, in 1972, the first exponential worst-case analysis lower bounds were shown for the number of pivot steps the simplex method may take with certain pivot rules, and many analogous results followed [8]–[24]. This discovery opened up the search for algorithm analysis frameworks which could more properly capture the behavior of the simplex algorithm and algorithms which showed a similar behavior.
For algorithms like these, average-case analysis is the theoretician’s next attempt to provide a better understanding of the running time. In the case of the simplex method, average-case analysis saw much activity during the 1970s and 1980s, and as a result, we know tight polynomial upper and lower bounds in the average case setting [25]–[27], [27]–[34]. However, linear programs seen in the average case setting greatly differ on a structural level from linear programs seen in practice. As such, average case analysis offered an incomplete explanation. Despite the development of algorithms for linear programming which run in polynomial time under a worst-case analysis, the simplex method is still an essential part of all linear programming software. As such, there remains a need to understand its performance in practice.
In their seminal work [35], Spielman and Teng proposed the smoothed analysis framework as an explanation as to “why the simplex method usually takes polynomial time," and which sought to incorporate the benefits of randomness in average case analysis while retaining some of the structure found in practical linear programs. Their framework was subsequently applied to many more algorithms (see [36] for an overview). In , we review smoothed analysis of the simplex method. In , we discuss the limitations of the smoothed analysis framework as a model for the simplex method. These limitations undermine the ability of smoothed analysis to explain why the simplex method usually runs in polynomial time.
In , we describe our new algorithm analysis framework and how our framework is overcoming these limitations.
Using our by-the-book-analysis framework, we explain how a simplex method with bound perturbations is guaranteed to solve LPs in expected polynomial time.
1.1 Smoothed Analysis and the Simplex Method↩︎
We first recall the basic principles of the simplex method. The simplex method is best thought of as a class of algorithms. A simplex method first determines a basic feasible solution using a Phase I procedure and then iteratively moves to new basic feasible solutions until an optimal solution has been found. The individual moves are called pivot steps, and the number of pivot steps is a proxy for the running time. The choice for which basic feasible solution to move to is governed by a pivot rule. Some popular examples of pivot rules are the most negative reduced cost rule [37], the steepest edge rule and its approximations [38]–[40], and the shadow vertex rule, also known as the parametric rule [25], [41]. It is this last pivot rule that is the foundational tool for most probabilistic analyses, including all results in smoothed analysis.
In the smoothed analysis of linear programming, we assume that an adversary specifies an LP \[\begin{align} \operatorname{maximize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ \operatorname{subject~to} \quad & \bar A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \bar {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}, \end{align}\] and that subsequently a perturbation \(\hat{A} \in \mathbb{R}^{n \times d}, \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) is sampled. We assume that the entries of \((\hat{A}, \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}})\) are independently sampled from the Gaussian distribution with mean \(0\) and standard deviation \(\sigma > 0\), and we assume that the rows of the extended matrix \((\bar A, \bar {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}})\) each have Euclidean norm at most \(1\). Under these assumptions, smoothed analysis aims to develop simplex methods for solving \[\begin{align} \operatorname{maximize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ \operatorname{subject~to} \quad & (\bar A + \hat{A}){\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \bar {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}+ \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}, \end{align}\] for which the expected running time can be bounded by a polynomial function in \(\sigma^{-1}, d\) and \(n\). After a line of work [35], [42]–[46], Bach and Huiberts [47] found a simplex method that requires no more than \[O(\sigma^{-1/2} d^{11/4} \log(n)^{7/4})\] pivot steps. They also described a lower bound which applies to all pivot rules, stating that \[\Omega(\sigma^{-1/2} d^{1/2}\ln(1/\sigma)^{-1/4})\] pivot steps are necessary when \(n = (4/\sigma)^d\).
Despite these upper and lower bounds agreeing on the exponent of \(\sigma^{-1/2}\), smoothed analysis can hardly be said to be a complete explanation of the simplex method’s running time. The LPs in smoothed analysis are unlike LPs in practice in a number of important ways, of which we describe three in the next section. These three differences undermine the three leading interpretations given for the smoothed analysis of the simplex method.
1.2 Limitations of Smoothed Analysis↩︎
1.2.0.1 Sparse perturbations
Most linear programs seen in practice are very sparse; less than 0.1% of their entries are non-zero (see, e.g., [1], [48]–[50]). The same holds true for known worst-case inputs. This stands in stark contrast with smoothed analysis, where 100% of the entries are non-zero since they are randomly perturbed according to a continuous probability distribution. As such, in the smoothed probability model, the set of sparse linear programs has measure \(0\).
Since both practical linear programs and theoretical worst-case inputs are sparse, it remains unclear whether smoothed analysis provides a viable synthetic model for describing the “brittleness” of worst-case inputs and whether smoothed analysis indeed helps to explain why the simplex method is usually fast on inputs observed in practice.
In their foundational work introducing smoothed analysis [35], Spielman and Teng highlighted the lack of sparsity as a weakness of their model. As a possible fix for this problem, they proposed zero-preserving and multiplicative perturbations. However, we note that exponential running times for simplex methods still occur in this zero-preserving model, even for constant-size perturbations. Consider the Klee-Minty instance [8]: \[\begin{align} \operatorname{maximize} \quad &{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_d \\ \operatorname{subject~to} \quad & 0 \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_1 \leq 1 \\ & \varepsilon{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{j-1} \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_j \leq 1-\varepsilon{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{j_1} \qquad \forall j \in \{2,\dots,d\}. \end{align}\] This construction gives exponential-length simplex paths for Bland’s least-index rule for any value of \(0 < \varepsilon< 1/2\). The different occurrences of \(\varepsilon\) need not be equal for their proof to work. As such, if we take an instance of the KM-cube with \(\varepsilon=1/4\) and perturb all non-zero entries with small but constant-size noise, the same simplex path will remain a simplex path for Bland’s rule. This demonstrates that the simplex method with Bland’s rule has exponential smoothed running time for both zero-preserving and multiplicative perturbations. No other sparse smoothed analysis model has been proposed.
1.2.0.2 Matrix entry precision
When LP constraint data is noisy or is rounded to low precision, this affects LP solver behavior negatively. For example, a nominally singular set of constraints might become a basis (potentially an ill-conditioned one) due to improper rounding, and this can lead to performance degradation. For this reason it is recommended to specify all constraints with maximum numerical precision (see, e.g., [51], [52]): when possible use all-integer inputs, otherwise prefer double-precision over single-precision floating point numbers. Higher-precision matrix entries tend to give better performance.
In contrast, smoothed analysis produces stronger performance guarantees if more noise is added to the constraint matrix, thereby seeming to predict that lowering the precision of constraint data yields performance improvement. This appears to be a disagreement between practical advice and smoothed analysis’ theoretical prediction. This disagreement indicates that smoothed analysis might not be an appropriate model for the effects of round-off error or floating-point imprecision.
1.2.0.3 Independence of matrix entries
A small perturbation to the constraint matrix can lead to a large change in the feasible region. This fact has long been observed on practical LPs in the robust optimization literature. For example, [53] describes that the instance PILOT4 has constraints whose entries are seemingly random with 7 digits significant. One example they give is
constraint 372 which reads \[\begin{align} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{a}}}
{\boldsymbol{\textstyle\boldsymbol{a}}}
{\boldsymbol{\scriptstyle\boldsymbol{a}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{a}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\equiv &- 15.79081{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{826} - 8.598819{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{827} - 1.88789{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{828} - 1.362417{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{829} - 1.526049{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{830} \\
&-0.031883{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{849} - 28.725555{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{850} - 10.792065{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{851} - 0.19004{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{852} - 2.757176{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{853} \\
&-12.290832{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{854} + 717.562256{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{855} - 0.057865{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{856} - 3.785417{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{857} - 78.30661{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{858} \\
&-122.163055{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{859} - 6.46609{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{860} - 0.48371{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{861} - 0.615264{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{862} - 1.353783{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{863} \\
&-84.644257{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{864} - 122.459045{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{865} - 43.15593{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{866} - 1.712592{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{870} - 0.401597{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{871} \\
&+{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{880} - 0.946049{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{898} - 0.946049{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_{916} \geq b \equiv 23.387405.
\end{align}\] When the “ugly reals”1 entries are each independently randomly changed by as little as \(0.1\%\), then the original LP
solution has a mean violation \(\max(0,\frac{b - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{a}}}
{\boldsymbol{\textstyle\boldsymbol{a}}}
{\boldsymbol{\scriptstyle\boldsymbol{a}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{a}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*}{b})\) of \(150\%\). Surely we must assume that even these LPs, which are sensitive to small independent random perturbations, were formulated appropriately
by the modeler for their purpose. Despite looking random, these entries may have non-trivial relations between them which are destroyed when perturbing independently.
We interpret this to imply that the constraint data is highly accurate, in the sense that the constraint matrix entries cannot be independently perturbed by small amounts. In this manner, we argue that the independence assumption makes smoothed analysis an inappropriate model for measurement error or other forms of limited precision input.
1.2.0.4 What is modelled
The variable nature of \(\sigma\) leaves smoothed analysis agnostic to what it models. Although different interpretations have been given in the literature, they largely fall within one of three categories: smoothed analysis models the “brittleness” or “isolation” of worst-case instances, smoothed analysis models the effects of numerical imprecision in floating-point arithmetic, or smoothed analysis models the effects of measurement error or other arbitrary influences. As we have seen above, each of these three interpretations has notable weaknesses. We must note that, when considered as a scientific theory, the flexibility of interpretation in regards to the meaning of \(\sigma\) is not a strength of the smoothed analysis model.
We conclude that smoothed analysis is a breakthrough framework that led to sound mathematical theorems. As a scientific theory, there are several interpretations about what smoothed analysis purports to model. This ambiguity results from the fact that its inherent assumptions are made for mathematical simplicity, without being traceable to direct observations. In this work we will propose an alternative analytical framework designed toamong other thingsaddress precisely this weakness of smoothed analysis. We offer what we argue is a much improved explanation for the practical behavior of the simplex method and we expect this new framework to offer improved explanations on the behavior of other algorithms as well. Although we seek specifically to deviate from those aspects of smoothed analysis which we have identified above as weaknesses, the proof techniques which were developed in the smoothed analysis literature will play a crucial role in the proposed by-the-book analysis presented in this paper; a framework we detail in the next section.
1.3 Introducing By-The-Book Analysis↩︎
In this paper we propose a new algorithm analysis framework that we call by-the-book analysis. Unlike smoothed analysis, by-the-book analysis is explicitly empirical by being intentional about what effect it models. As part of this, we open up the possibility of modeling not only the input data but also aspects of an algorithm’s implementation. It is well-known that theoretical descriptions of algorithms differ from their software implementations. These differences are often thought to be incidental and to be theoretically uninteresting, even though they have repeatedly proven to be beneficial in practice. We posit that these differences, if they are found in state of the art implementations, can be essential to a full theoretical understanding. In doing so, by-the-book analysis answers the call of [54] for an empirically-based explanatory theory capturing “the all-important tricks that are engineered into commercial codes.”
With by-the-book analysis, we strive to prove performance guarantees while using assumptions that are maximally grounded in computational experience. For this reason, our research process has occurred in three phases as depicted in . We will first describe what we propose as the general framework of by-the-book analysis, and then we will exemplify this framework by outlining the specifics of our own by-the-book analysis of the simplex method.
In general, by-the-book analysis proceeds in three phases. In the first phase, one establishes an understanding of true and relevant features of real-world implementations of an algorithm and features of real-world inputs. This process can involve reading of the scientific literature, open-source code, user manuals, running experiments etc. in order to establish how the algorithm is implemented in practice, how users of the algorithm are instructed to formulate their inputs, and what features are common among real problem instances.
The second phase is a process which converts parts of this understanding to a mathematical model of the algorithm on which some form of traditional theoretical analysis can be performed. Based on the observations made in the first phase, one chooses an algorithm description and a collection of assumptions satisfied by the algorithm and/or its inputs. The chosen algorithm description may deviate from traditional theoretical descriptions of the algorithm if that deviation is warranted by observations. One might assume, for example, that certain relevant parameters of the input are bounded. Although one must inevitably make assumptions that make the upcoming mathematical analysis tractable, one should, to the greatest extent possible, choose a mathematical model and mathematical assumptions that comport with the observations made in the first phase. This is what gives a by-the-book analysis its empirical basis.
The third phase is a traditional mathematical analysis of the chosen model under the chosen assumptions. In this phase, by-the-book analysis is agnostic to the mathematical techniques used provided that any further assumptions inherent to those techniques are likewise grounded (to the greatest extent possible) in the observations of the first phase. For example, in this phase the mathematical analysis could resemble parts of a worst-case, average-case, smoothed analysis, etc. Ultimately, what makes something a by-the-book analysis is not the choice of mathematical tools used, but that the model, assumptions, and tools are grounded in observation.
Note that the order of the phases is important. In particular, the determination of mathematical assumptions occurs before the production of theoretical running time bounds. It is contrary to by-the-book analysis to produce a theoretical bound parameterizing by whichever parameters turn out to be mathematically fruitful and to only then search for explanations as to why these parameters may be bounded in practice. By basing the theoretical approach on observations, we argue that this methodology is better positioned to produce accurate explanations. This is the ultimate objective of performing a by-the-book analysis. In this spirit, we emphasize again that what makes something a by-the-book analysis is not which type of mathematical tools or proof techniques are used, but whether or not the research methodology follows the general framework outlined above. To call something a by-the-book analysis (or not) is not a mathematical distinction, but a methodological one; it is not a question of what mathematical assumptions are made, but of how they are produced in the first place.
We will now briefly outline how we applied the above procedure to the simplex method in the present analysis. In the initial phase we studied the scientific literature, source-code, and user manuals of simplex implementations, and we interviewed developers working on multiple different LP software packages. A primary role in this phase was played by the titular textbooks [1], [55] which describe how to implement a simplex method. After observing the prominent role that numerical tolerances are given in both books (on which we will elaborate later), we verified their importance by way of inspecting both the source code of open-source LP solvers and the user manuals of closed-source LP solvers. Because the simplex method is a popular and well-documented algorithm, we were able to draw on a diverse array of sources and identify points of broad agreement. The outcomes of this phase are described in detail in .
In the second phase, we balanced verisimilitude with our observations against the tractability of the upcoming mathematical analysis. We aimed for a traditional, theoretical performance guarantee which, when all parameters are filled in with plausible values, gives the strongest possible conclusion in terms of how well this bound is reflecting an algorithm’s behavior in practice (subject, of course, to adherence to our observations in the first phase). We observed that every major modern simplex solver perturbs its right-hand side vector with random exponentially distributed noise (whose magnitude is grounded in our observations). For this analysis, we hence chose a model of the simplex method that perturbs the right-hand side vector and which solves the input LP up to optimality and feasibility tolerances (whose magnitudes are grounded in our observations) using the shadow vertex pivot rule with an auxiliary objective function chosen uniformly at random. To maintain mathematical tractability, we assumed the algorithm to be implemented using exact real valued arithmetic, Dantzig’s ratio test, and the shadow vertex pivot rule. Relevant instance-specific parameters are assumed to be bounded in magnitude (again, grounded in observation). Detailed descriptions can be found in and .
In the third and final phase, we used our mathematical assumptions on the data and the algorithm to prove running time bounds using a mathematical analysis similar to those found in smoothed analysis, using the randomness in our perturbations and our auxiliary objective function.
As is evident from the specificity of some choices detailed above, different by-the-book analyses for the same algorithm are possible. They can vary in how the algorithm and the data are modeled, i.e., which assumptions and parameters are chosen, and they can vary in the types of mathematical techniques used to prove running time bounds. We can distinguish at least two paths for improvements on a particular by-the-book analysis. The first occurs in the third phase and is one that we are used to in theoretical computer science: for the same set of mathematical assumptions, stronger mathematical analysis techniques can be developed, leading to stronger theorems. The second is to formulate different mathematical assumptions to model the same or different observations performed in the first phase.
One is unlikely to produce a singular, holistic explanation for the performance of an algorithm, parameterizing by all possible factors and perfectly distilling the impact of these parameters into a bound. Theoretical analysis requires simplified models on which theoretical analyses can be performed. The goal is not to produce a perfect model of reality, but instead to produce a mathematical model which is better informed by, and more reflective of, what we can observe. As such, a by-the-book analysis can be improved in the first phase by making more or better observations, and second phases by better modelling. Some axes on which such improvement can be evaluated include, but are not limited to, evaluability and verisimilitude. It is preferable to express bounds in terms of parameters which can be effectively observed. It is also preferable to use mathematical assumptions which more closely resemble computational practice, such as to have a theoretical description of an algorithm which comports with the way in which that algorithm is implemented in code.
To exemplify this last point, we wield it against our present analysis in . We evaluate how well the chosen model, assumptions, and mathematical techniques comport with our observations, and evaluate how well our research process adhered to the framework of by-the-book analysis. For the sake of good scientific practice, such a final evaluation phase is crucial. We recommend the evaluation process to be not only conducted as an isolated final last step. Instead, we recommend to evaluate the observations and decisions made as a steady part of the process throughout all phases.
1.4 Observations of Practical LP↩︎
Having described the framework of by-the-book analysis in general, we can now present a by-the-book analysis of the simplex method. We begin here with an overview of our findings about the properties held by LPs in practice. As described in , these findings will inform our mathematical assumptions. As noted already, two main contributing factors to our findings are the software implementations of the simplex method and the observed characteristics of (well-formulated) inputs. A given problem can be modeled in various ways, and a given LP has a wide array of equivalent formulations. When LPs are solved in practice, they are typically modeled by domain experts who are constructing their formulations according to best practice recommendations explicitly detailed in software user manuals.
We review a number of software packages and describe our observations. For the open-source solvers Glop and HiGHS, we report our observations from reading their respective source codes. For the closed-source solvers Gurobi and MOSEK, we read their user manuals in detail and spoke extensively with developers. We ran an experiment on the MIPLIB 2017 benchmark set to support our assumption that feasible sets of practical LPs have small mean width. The features of solvers and data described here are mostly limited to those which are directly relevant to our models of scaling, tolerances, and bound/cost perturbations. We first describe the general principles of what we observe, and in , we describe what observations we have with regard to specific software packages. There are many other features we do not remark upon, although we believe that some can be potentially of interest to theoretical computer scientists and may lead to improved by-the-book analyses of the simplex method.
1.4.0.1 Condition number
In an IEEE 754 double precision floating point number, there is 1 bit for the sign, 11 bits for the exponent, and 53 bits for the significand (of which 52 are explicitly stored). This means that any such number can have a relative precision of at most \(2^{-53} \approx 10^{-16}\). Any arithmetic performed with numbers whose relative magnitudes differ by many orders of magnitude will yield imprecise results. This effect is particularly visible when solving a system of linear equations. Here, the backward error up to which it can be solved increases when the linear system has a high condition number. The condition number \(\kappa\) of a matrix \(A\) is defined as \(\kappa \mathrel{\vcenter{:}}= \|A\|\|A^{-1}\|\). The relation to the backward error is the following. When solving the linear system \(Ax = b\), a condition number \(\kappa = 10^k\) indicates that one may loose up to \(k\) digits of accuracy in \(x\) from the accuracy in \(b\). Typically, in linear programming software, the condition number of linear systems is assumed to be no greater than \(10^{12}\), see the discussion pertaining to Gurobi in for example.
1.4.0.2 Tolerances
With \(53\) bits of accuracy and a condition number of order \(10^{10}\), solving a linear system \(A_B {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B\) can be done up to backward error around \(10^{-6}\). That is, the software is able to find \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\) such that \(\|A_B^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B - \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\| \leq 10^{-6}\cdot \|A_B^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B\|\). Because of this inevitable inaccuracy, every LP solver based on floating-point arithmetic includes a number of different tolerances and thresholds. Here we discuss only the user-facing absolute feasibility tolerances. There are two of these, a primal feasibility tolerance and a dual feasibility tolerance, also called the optimality tolerance. These two numbers, both commonly of order \(10^{-6}\) (e.g. the default value in Glop and Gurobi, as noted in ), indicate how large the violations of primal or dual constraints are allowed to be for the solver to still report a solution as being ‘feasible and optimal’. In contrast to the relative error described in the preceding paragraph, solver tolerances are absolute.
In this work we model the primal feasibility and complementary slackness conditions as follows. For more information on the deduction of this statement of complementary slackness, see . There, we solve linear programs of the form \[\begin{align} \operatorname{maximize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ \operatorname{subject~to} \quad & A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\\ &{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}, \end{align}\] where we assume that every row of \(A\) has norm \(1\). The solver may return (as optimal and feasible up to tolerances) any primal-dual solution pair \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^* \in \mathbb{R}^d, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^* \in \mathbb{R}^n_{\geq 0}\) which satisfies primal feasibility up to primal feasibility tolerance \(\mathrm{feasTol}>0\) and complementary slackness up to dual feasibility tolerance \(\mathrm{optTol}>0\). Formally, they satisfy the primal feasibility conditions \[\begin{align} A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^* &\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}+ \mathrm{feasTol}\cdot \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_n \label{eq:apx-feas}\\ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}&\geq -\mathrm{feasTol}\cdot\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d, \nonumber \end{align}\tag{1}\] and dual feasibility conditions \[\begin{align} A^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^* &\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}- \mathrm{optTol}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d \\ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^* &\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}. \end{align}\] To ensure approximate optimality, we have a complementary slackness condition \[\begin{align} \text{~for all~} i \in [n] &\text{,~if~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*_i > 0 \text{~then~} (A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*)_i \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_i. \label{eq:apx-opt} \\ \text{~for all~} j \in [d] &\text{,~if~} (A^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*)_j > c_j + \mathrm{optTol}\text{~then~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_j^* \leq 0. \nonumber \end{align}\tag{2}\]
In our analysis we will take \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^* \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\), but solvers may assume a bit more flexibility and take \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^* \geq -\mathrm{optTol}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) due to their need to obtain \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*\) as the solution to a system of linear equations.
1.4.0.3 Scaling
A well-formed LP should be scaled appropriately: the input data, the output solution, and intermediate values should be not too large and not too small. This is important to the functioning of real-world software.
The primary way in which this scaling requirement has been explained is through the effects of floating-point arithmetic. A poorly scaled matrix is likely to have a high condition number. The numerical concerns described above are an important reason for why proper scaling of the input is explicitly recommended.
The tolerances provide a second important anchor point for scaling. When some of the optimal values are small relative to the tolerances, this can negatively affect the quality of the solution for its intended purpose. When some of the solution values are too large relative to the tolerance, the solver may be unable to satisfy all feasibility and optimality conditions within the small (absolute) tolerance.
It is best practice to choose the measurement units for the rows and columns such that the non-zero values of variables across basic feasible solutions lie in some common, limited range of magnitudes. Most LP solvers also include automatic scaling algorithms in case the user’s scaling is deemed insufficient. This practice speaks directly to the geometry of the feasible set.
Overall, the role of scaling for the performance of the simplex method is poorly understood [1].
1.4.0.4 Perturbations
When feasibility tolerances are permitted, solvers can be made faster by intentionally changing the right-hand side \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\), variable bounds, or the objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\). A number of mechanisms make use of this allowance. We focus on one specific such mechanism, which is random perturbation before starting the simplex method. Generally these perturbations are done in such a manner as to make the feasible region larger. The perturbations are of a similar order of magnitude to the feasibility tolerance as discussed in . The primary use of a priori perturbation in practice is in perturbing the objective vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\) before starting the dual simplex method. During both the primal and dual simplex method, more perturbations and shifts can be performed at later iterations if needed. One consequence of bound perturbations is that degeneracy is avoided and consequently the simplex method cannot cycle. We furthermore analyze their implications on the running time of the simplex method.
As noted by [56], perturbations are ubiquitous among LP solvers but “despite its wide application, the discussion of why perturbation works is relatively rare." He highlights that in the survey by Bixby [57] it is noted that more”aggressive” perturbations are experimentally viewed as more effective and that, in personal communication, it became clear there was no theoretical understanding as to why. Our results provide the first theoretical explanation of this phenomenon.
1.5 Overview↩︎
The body of the paper is split into three parts. In we describe in detail our observations from the user manuals of Gurobi and MOSEK and the source code of HiGHS and Glop, as well as the results of experiments on the LP relaxations of instances in MIPLIB 2017. The observations described are related to the three properties described above: scaling, tolerances, and perturbations, as well as the condition number of the constraint matrix. It is common for a simplex method in state of the art implementations to add perturbations. We choose these perturbations proportional to the feasibility tolerances, mirroring their use in practice as found in .
After mathematical preliminaries in , in we describe our full two-phase simplex method whose running time we analyze, and connect the required parameters with the observations from . Its analysis relies on , which we devote to proving. On the perturbed linear program, we assume our algorithm stays exactly feasible at all times without considering the tolerances, i.e., without accounting for floating point arithmetic errors. These observations will yield an argument that certain parameters should be assumed to be bounded, show that bound perturbations can be part of a complete simplex method, and provide estimates for the order of magnitude that different quantities have in practice. We end with a discussion of the results in . In the remainder of this overview, we summarize and , which contain all proofs.
1.5.0.1 Two-Phase Simplex Method
We describe our algorithm in detail in . In phase 1, roughly speaking, we start from the basis of non-negativity constraints and add an auxiliary variable \(0 \leq s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\) to make the corresponding point \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\) feasible as \(A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}+ s\frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty} \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) with \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\) and \(s=\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\). We pivot to find a basic feasible solution with \(s=0\), which yields a basic feasible solution to the original LP. Then Phase II works as expected, pivoting to the optimal solution. Assuming non-degeneracy, this strategy can be executed with the semi-random shadow vertex method with no unexpected complications: we move from an initial objective to a random objective on all the coordinates \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\), then we minimize \(s\) to find a basic feasible solution which is optimal for a random objective on \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\), and finally we move from the random objective to the input objective.
The main challenge in our phase 1 is to make this strategy work when the right-hand side and bound vectors are perturbed: the basis of non-negativity constraints should remain feasible. To achieve this, we slightly change the coefficients on \(s\) to obtain \(A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}+ s\frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}-\mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty} \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\). Now the point \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}, \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty)\) has slack \(\mathrm{feasTol}\sqrt{\ln n}\) on each constraint. Using Hoeffding’s inequality, we can show that the solution \((\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}- \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}, \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty)\) is feasible with good probability. Note that this requires that solution to have mean \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}},\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty)\), which we achieve by adding a coefficient on the auxiliary variables into the perturbed non-negativity bounds \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} s\). Perturbing the right-hand side \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) to make \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) does not change this feasibility because \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\). The bounds \(0 \leq s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\) remain unperturbed. The analysis can accommodate for a single variable with fixed bounds, incurring a constant factor increase in the expected number of pivot steps.
We end with a proof that the resulting solution is primal feasible up to primal feasibility tolerance, dual feasible up to the dual feasibility tolerance, and that it satisfies the complementary slackness up to optimality tolerance, indicating that the returned solution satisfies all requirements.
1.5.0.2 Running time bound
folds the variable bounds into the constraint matrix and considers linear programs of the form \[\begin{align} \operatorname{maximize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ \operatorname{subject~to} \quad & A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}, \end{align}\] where again every row of the matrix \(A\) is assumed to have norm equal to \(1\). Assuming that the right-hand side vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) is perturbed with two-sided exponentially distributed perturbations with rate \(\eta > 0\), the expected number of pivot steps taken by the shadow vertex method to travel from the vertex maximizing a uniformly random objective vector \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^{d-1}\) to the vertex maximizing an objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) is, by , at most \[O\left(d \sqrt{\frac{d \ln(n) M}{\eta} \ln\left(\frac{dN\ln (n)}{\eta \cdot \varepsilon} \right)}\right),\] where \(M > 0\) indicates the expected mean width of the feasible set and \(N\) is a proxy for the maximum absolute objective value. The ratio \(N/\varepsilon\) is essentially scale-invariant in \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|\), but our analysis does require the linear program to be “well-scaled” in two ways: the feasible set must be scaled such that the mean width \(M\) is bounded, and the constraint matrix must be scaled such that the rows of \(A\) each have norm at most \(1\). These assumptions are to be expected. Scaling assumptions are required for the feasibility tolerances to be well-defined, and are common for state of the art software implementations as well, as described in . The perturbation size is chosen such that \(\eta \cdot \ln(n) \approx 10^{-6}\) is the feasibility tolerance.
In order to learn the mean width of typical linear programming problems, we sampled the widths for LP relaxations of the MIPLIB 2017 benchmark set [49] and for LP problems found in NETLIB [50]. For the MIPLIB instances, we found that the majority of LP relaxations has a feasible set with mean width smaller than \(\sqrt{\frac{2d}{\pi}}\), the limit mean width of the unit cube. On NETLIB, the mean widths were often larger, but only seldom above \(10^6\). We expect that if the columns of the NETLIB instances are rescaled properly, that the mean widths will be smaller.
All parameters which appear in our running time bound can be effectively estimated. That is, after treating our chosen parameters (as identified during the first and second phases of by-the-book analysis) as having realistic and constant values, we obtain a polynomially bounded running time. Importantly, we have produced a theoretical treatment of the simplex method’s running time that is principally grounded in observable traits of real-world codes and problem instances.
We note that the explanatory power of this result strongly exceeds that of smoothed analysis, since the result presented here follows from modeling practical implementations of the simplex method applied to real-world data. In particular, the constraint matrix is not perturbed, so our analysis applies to sparse matrices, and all parameters involved are based on measures computable in practice. Our analysis is built to know ahead of time exactly what real-world properties our mathematical assumptions are modeling. In contrast (as discussed in ), there is not agreement on what, if anything, is actually modeled in smoothed analysis by perturbing the constraint matrix. We argue that the result presented here narrows the gap between what we can explain using theoretical frameworks and what is observed in practice.
1.5.0.3 Proof overview
The proof of our main technical result proceeds in three phases. First, in we describe the effect of the right-hand side perturbations. We show that, conditional on a basis being feasible, the corresponding basic solution will have all its non-zero slack values on the inequalities be large. Specifically, the smallest non-zero slack will be at least \(\frac{\eta}{1240 d \ln(n)}\) with probability at least \(0.9\). As in the previous paragraph, \(\eta\) is a measure of the magnitude of the perturbations. The lower bound on the slacks is derived using a Chernoff bound. This argument is derived from a similar one used in smoothed analyses of the simplex method [46], [47].
Second, in we show a dual analog to the above, based on the shadow vertex simplex path from a random auxiliary objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to a largely fixed objective \(\varepsilon^{-1} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\). We show that, conditional on a basis lying on this shadow path, with probability at least \(0.9\) there exists an intermediate objective \(t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) with \(t \in [0,\varepsilon^{-1}]\) for which this basis is optimized and for which the reduced costs on the tight constraints are large. Note that the existence of \(t \in [0,\varepsilon^{-1}]\) for which the basis is optimal for \(t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) is guaranteed by virtue of being on the shadow path. The novelty is that we can guarantee a choice of \(t\) yielding large reduced costs on tight constraints. In order to simplify this part of the analysis, we will throughout assume that the random objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) is sampled from an \(L\)-log-Lipschitz probability distribution. This will not affect the final conclusion.
Finally, in we show our upper bound on the length of the semi-random shadow vertex path. As a consequence of our results in and , we show that a large fraction of the feasible bases on this path must have both (in the above explained senses) large non-zero slacks and admit an intermediate objective for which all \(d\) reduced costs are large. Afterwards, we argue that whenever two subsequent bases on this path both have both properties, then the pivot step leading from the first to the second of them must make large progress. This progress is measured through four different potential functions. Two of these potential functions measure the progress of the intermediate objective vector \(t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) on the line segment from \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to \(\varepsilon^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\). The third and fourth potentials encode progress in the objective value and auxiliary objective value respectively. A majority of pivot steps will consume a certain amount of at least one of these four potential functions. The total amount of potential available is bounded, which leads to our upper bound on the expected number of pivot steps.
1.6 Related Work↩︎
The simplex method has been studied in the worst case complexity framework under various additional assumptions. One line of work has assumed that the constraint matrix \(A \in \mathbb{Z}^{n \times d}\) is integer and has every square submatrix have a determinant which is bounded from above in absolute value. Under this assumption, upper bounds on the running time of the simplex method have been proven [58]–[62]. The semi-random shadow vertex pivot rule of [60] strongly informed our own. Approximating the maximum subdeterminant is known to be NP-hard [63]. Similar results are available for a closely related condition measure, the circuit imbalance [64]. A study estimating the circuit imbalance on the instances in MIPLIB 2017 [65] found it to require much higher numerical precision than is available with IEEE 754 double precision floating point arithmetic.
A parallel line of work investigates the worst-case performance of the simplex method when the slack values and reduced costs are bounded away from zero [66]–[68]. These papers extend an earlier result about the policy iteration algorithm for Markov Decision Processes with bounded discount rate [69], and are similar to a calculation done by Dantzig [70]. The condition number underlying this line of work is NP-hard to compute as shown in [71]. The same paper also estimated this condition number on the LP instances in the NETLIB test problem set [50], though the measurement approach they used severely limits what conclusions can be drawn based on these estimates. The use of maximal and minimal non-zero slacks was a direct inspiration for our analysis, although we have the minimal non-zero slack be probabilistically bounded using perturbations instead of deterministically by assuming structure in the input data.
In our by-the-book analysis, we use a simplex method with bound perturbations to obtain provable running time guarantees. Bound perturbations are often incorporated in codes with the stated purpose of preventing issues of degeneracy and stalling [1], [72]. Other methods to prevent stalling are possible, including the use of expanding tolerances or minimum step sizes [73]. In a theoretical regime where bounds may not be changed, other approaches are available [74]–[76].
A technique akin to, but developed independently of, bound perturbations was previously used in [77] to find a weakly polynomial-time algorithm for linear programming. Although aspects of what they do are similar to what we do here, their arguments adapted to our setting would produce exponentially worse running time dependencies on the parameters we consider.
2 Observations↩︎
In the following we discuss our findings in the several open source codes and user manuals which provide evidence and specificity in support of our claims in . The order of the solvers is alphabetical.
2.0.0.1 Glop
As part of Google’s software suite OR-Tools [78], Glop is an open-source LP solver. For any finite values in the input LP, Glop by default2 allows numbers with absolute values as low as \(10^{-30}\) and as high as \(10^{30}\). However, in order to increase sparsity, by default any floating-point numbers with absolute value below \(10^{-14}\) are set to zero3. Thus, some restrictions on the scaling of the LP are inherent in the solver.
Glop has configurable primal and dual feasibility tolerances, which have a value of \(10^{-6}\) by default4. In the dual simplex method, Glop can perturb costs before starting. When this is done, the default perturbation on the \(i\)’th element of the cost vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\) is given by5 \[\begin{align} r_i \cdot (10^{-5} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}_i + 10^{-7} \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|_\infty), \end{align}\] where \(r_i\) is sampled uniformly from the interval \([1,2]\). This cost perturbation is turned off by default6.
2.0.0.2 Gurobi
The Gurobi user manual recommends that any right-hand side coefficients are scaled such that their absolute value is \(10^4\) or less [51]. The same recommendation is given with regards to the objective coefficients and variable bounds. Similarly, they recommend that every LP has an optimal value less than \(10^4\). The latest version of Gurobi includes 3 different automatic scaling modes, but the documentation does not specify what algorithms are used or when they are used.
In Gurobi, any constraint coefficient whose absolute value is smaller than \(10^{-13}\) is treated as zero [51]. The user manual recommends that the user scales the input problem such that all non-zero matrix coefficients have their absolute value between \(10^{-3}\) and \(10^{6}\), and that they span no more than \(6\) orders of magnitude. The default big-\(M\) value (essentially, the finite number that stands in for ‘infinity’ in ILP formulations) is \(10^6\), further indicating that coefficients larger than this are undesirable.
The primal feasibility tolerance defaults to \(10^{-6}\) but is configurable to any value between \(10^{-2}\) and \(10^{-9}\). The same holds for the optimality tolerance. The primal tolerance is described as follows [51]: “\({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{a}}} {\boldsymbol{\textstyle\boldsymbol{a}}} {\boldsymbol{\scriptstyle\boldsymbol{a}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{a}}}}\cdot {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq b\) will be considered to hold if \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{a}}} {\boldsymbol{\textstyle\boldsymbol{a}}} {\boldsymbol{\scriptstyle\boldsymbol{a}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{a}}}}\ast {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}) - b \leq\) FeasibilityTol.” Here, ‘\(\cdot\)’ is the exact inner product, whereas ‘\(\ast\)’ is the computed product (subject to errors).
The Gurobi user manual further indicates that a condition number of \(10^{12}\) for a basic submatrix is considered large [51]. The Gurobi solver warns the user of large bounds when a variable has a bound with absolute value that is \(10^{10}\) or larger. It warns the user of large objective coefficients when there is a coefficient with absolute value \(10^{10}\) or larger.
Conversation with the developers of Gurobi confirmed that perturbations are actively used in all their simplex codes. Initial perturbations have magnitude proportional to the feasibility tolerance and perturbation size may grow when the simplex method makes insufficient progress.
2.0.0.3 HiGHS
In the open-source MIP solver HiGHS [79], the simplex method incorporates two scaling methods (one using equilibration and one based on maximum value7). If all non-zero elements of the constraint matrix are between \(0.2\) and \(5.0\) then no automatic scaling will take place8. Notably, this automatic scaling is not performed for the interior-point method.
HiGHS has configurable parameters for the smallest non-zero and largest finite entries that may be present in the constraint matrix9. By default, any matrix entry whose absolute value is smaller than \(10^{-9}\) is treated as \(0\), and any matrix entry whose absolute value is larger than \(10^{15}\) is treated as infinity.
There are configurable primal and dual feasibility tolerances which equal \(10^{-7}\) by default and which can be tuned to be as low as \(10^{-10}\). In the primal simplex method, when a bound is found to be infeasible, it is shifted to make it feasible by at least the feasibility tolerance, plus a uniformly random quantity between \(0\) and the feasibility tolerance10. In the dual simplex method, the solver perturbs the cost vector already during initialization11. These perturbations are uniformly distributed in an interval.
The PhD thesis of Qi Huangfu [56] documents the development of the HiGHS simplex method solver. In his work, he suggests that feasibility and optimality tolerances are typically on the order of \(10^{-6}\) or \(10^{-7}\). For HiGHS he describes12 an explicit choice of value for cost perturbation of \[(10^{-5}|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}_{j}| + 10^{-7}\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|_\infty +10 \cdot \mathrm{optTol}) (r+1),\] where \(\mathrm{optTol}> 0\) is the dual feasibility tolerance and \(r\) is a uniformly random number in \((0,1]\). In particular, the above cost perturbation incorporates a term whose magnitude is proportional to the objective and a term with absolute magnitude. In our analysis, we model these terms as being the same by assuming our LP is sufficiently well-scaled.
The HiGHS documentation recommends users to scale their primal variables such that the optimal solution has its entries be of order unity. The documentation further states that “this typically occurs if all objective and constraint matrix coefficients, as well as finite variables and constraint bounds, are of order unity.”
2.0.0.4 MOSEK
The MOSEK instruction manual [80] mentions that problems containing large or small coefficients, outside the range \([10^{-7}, 10^9]\), are often hard to solve13. Although it includes a method to automatically scale variables, no specifics are reported about how this scaling is performed or under what circumstances. The solver will print an error14 to warn the user if any matrix element is larger than \(10^{10}\), if a cost entry is larger than \(10^8\) or if a bound is larger than \(10^8\). If an upper and a lower bound for a variable differ by less than \(10^{-8}\), then MOSEK will consider the two numbers to be equal15. The MOSEK Modeling Cookbook [81] considers “a model to be badly scaled if variables are measured in very different scales, or constraints or bounds are measured on very different scales.”
The absolute tolerance16 for the primal and dual infeasibility both default to \(10^{-6}\). A technical report [82] mentions that, when troubleshooting infeasible problems, the most important constraints are those whose relative values in the Farkas certificate are at least \(10^{-8}\).
Interviews with the developers of MOSEK confirm that perturbations are actively used in all their simplex codes, and that adding larger perturbations are an effective tool when the simplex method stalls. Initial perturbations have magnitude proportional to the feasibility tolerance.
2.0.0.5 Experiments on MIPLIB 2017
Our main technical result is stated in terms of the mean width of the feasible set of the linear program (after bounds are perturbed). The mean width of a polyhedron \(P\) is defined as the expected value of \[\begin{align} \operatorname{maximize} \quad & \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}- {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}')\\ \operatorname{subject~to} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}' \in P, \end{align}\] where \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^{d-1}\) is uniformly random distributed on the unit sphere.
In order to empirically ground this quantity, we performed experiments to measure it on two benchmark libraries: the NETLIB library of LPs [50] and the LP relaxations of the MIPLIB 2017 benchmark set of Mixed Integer Linear Programs [49]. The experiment is rudimentary in design, foregoing any considerations of automatic scaling or presolve.
For every instance, we sampled the width in 100 random directions as follows. Due to Milman’s concentration of measure phenomenon (see, e.g., [83]), the width is quite sharply concentrated around the mean.
We sampled an objective vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) with entries sampled independently at random from a Gaussian distribution with mean \(0\) and standard deviation \(1\). We loaded
the MPS files using Gurobi 13.0.1 and used it to obtain the LP relaxation. Over this LP feasible set we maximized and minimized the objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\). The difference between the two values was divided by \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|\) to obtain the width in direction \(\frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|}\), which is a uniformly random unit direction. The code for this experiment can be found at https://github.com/sophiehuiberts/miplib-meanwidth, and it was run on an Intel i5-14600KF CPU.
Besides neos-5114902-kasavu, every individual minimization and maximization completed within the deterministic work limit of 300. Any instance finding an unbounded or infeasible status code (INFEASIBLE, INF_OR_UNBD
or UNBOUNDED) was dropped. On NETLIB, 58 out of 114 instances remained after this phase. On MIPLIB, 186 out of 240 instances remained.
All sample means obtained were between \(10^{-1}\) and \(10^{9}\). In MIPLIB, the sample means were relatively small. 46 instances have sample means in \([10^{-1},10^1]\), 65 have sample means in \([10^1,10^2]\), 54 in \([10^2,10^4]\), 10 in \([10^4,10^6]\) and the remaining two instances with sample means between \(10^6\) and \(10^9\). The mean widths on NETLIB were larger on average, with 7 instances each having sample mean widths in \([10^{-1},10^1]\) and \([10^1,10^2]\), 26 instances in \([10^2,10^4]\), 12 in \([10^4,10^6]\) and the remaining 6 in \([10^6,10^9]\).
The full results are drawn in , with the sample mean width on the vertical axis and the total variable count of the instance on the horizontal axis. Besides the results for NETLIB and MIPLIB benchmark instances, the figure includes sample mean widths for unit cubes \([0,1]^d\) in various dimensions. The mean width of the unit cube converges to \(\sqrt{\frac{2d}{\pi}}\), and most MIPLIB instances are below that line.
We see that many instances have sample mean widths that are small. We take these results as preliminary evidence that the mean width of the feasible set is bounded from above for well-scaled linear programs.
Although many instances have small mean widths, there is a sizable fraction with large mean width. This is especially pronounced for the NETLIB set. We suspect that the instances the instances with high mean width are poorly scaled. More work is needed to verify whether indeed the better scaled instances tend to have lower mean widths.
In production software, the simplex method is run on linear programs after they have been presolved and automatically rescaled. In our experiment we measured the mean width of the feasible set as given by the user, instead of the mean width of the presolved and scaled feasible set. This may affect the observed results. An in-depth computational study of the interaction of mean width and presolve/scaling is justified.
3 Preliminaries↩︎
We write \([n] = \{1,\dots,n\}\) and \(\binom{[n]}{d} = \{B \subseteq [n] : \abs{B}=d\}\). We take \([0]=\emptyset\) to be the empty set. The \(d \times d\) identity matrix is denoted \(I_{d\times d}\). Given a matrix \(A \in \mathbb{R}^{n \times d}\) and a set \(S \subseteq [n]\) we write \(A_S\) to indicate the submatrix consisting of rows indicated by indexes in \(S\), and its associated map \(A_S : \mathbb{R}^d \to \mathbb{R}^S\). A set \(B \in \binom{[n]}{d}\) is called a basis if the square matrix \(A_B\) has full rank.
Vectors are bolded. The all-ones vector is \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d\), where the dimension of the vector may be omitted and denoted as just \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) if clear from context. We denote the \(i\)’th vector in the standard coordinate basis is \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}^{i}\). For a vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) we write \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_S \in \mathbb{R}^S\) to indicate the restriction of \(b\) to the entries indexed in \(S\). When \(A, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) are clear from context, we define the basic solution for a basis \(B\) as \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B = A_B^{-1} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B\). Scalars are denoted by lowercase letters, vectors by bold lowercase letters, and sets and matrices by uppercase letters. For two vectors \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}' \in \mathbb{R}^d\) we denote by \([{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}'] \subset \mathbb{R}^d\) the straight line segment connecting them. The unit sphere is denoted by \(\mathbb{S}^{d-1}= \{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\in \mathbb{R}^d : \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\|=1\}\).
3.1 The Shadow Vertex Simplex Method↩︎
Our analysis makes use of a particular simplex method called the shadow vertex simplex method originally introduced by Gass and Saaty in [41] in the context of parametric linear programming. A textbook introduction that proves the standard properties we rely on may be found in [84].
The shadow vertex simplex method works as follows. We fix a linear program \[\begin{align} \operatorname{maximize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ \operatorname{subject~to} \quad & A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}, \end{align}\] with \(A \in \mathbb{R}^{n \times d}\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) as input and start with an initial feasible basis \(B\). We obtain an auxiliary objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\in \mathbb{R}^{d}\) as follows. Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) be generically chosen such that \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}A_{B}^{-1} > {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\) holds. If \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}A_{B}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\), already the current basis is optimal and we are done. Otherwise, there exists a coordinate \(i\) such that \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}A_{B}^{-1})_{i} < 0\). Over all those indices \(i\) for which \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}A_{B}^{-1})_{i} < 0\), we apply the following ratio test. We define \[\lambda = \min_i\left(- \frac{({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}A_{B}^{-1})_{i}}{({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}A_{B}^{-1})_{i}}: ({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}A_{B}^{-1})_{i} < 0 \right),\] and let \(p\) be the argument minimizing the same function. Then, by the minimality of \(\lambda\) and positivity of \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}A_{B}^{-1}\), \((({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}+ \lambda {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}})A_{B}^{-1})_{i} > 0\) for all \(i \neq p\) and \((({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}+ \lambda {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}})A_{B}^{-1})_{p} = 0\). By non-degeneracy, there is a unique basis \(B^{1} \neq B\) such that \(B^{1} \supseteq B \setminus \{p\}\). We let \(B^{1}\) be the new basis and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}+ \lambda{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\) as the new auxiliary objective and continue the process from \(B^{1}\) following a sequence of adjacent bases \(B^{1}, B^{2},\dots, B^{k}\) until reaching an optimal basis. We note that the genericity of \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) ensures that for each basis \(B^{i}\) on the path, there exists \(t > 0\) such that \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}+t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}})A_{B^{i}}^{-1} > 0\). The existence of such a \(t\) is what allows the loop to continue even though \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}+ \lambda {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}})A_{B^{1}}^{-1}\) is only positive in the coordinates of \(B^{1} \cap B\) and \(0\) otherwise.
There are two standard geometric interpretations of the shadow vertex simplex method. The first is in terms of a two dimensional linear projection. Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}= A_{B}^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B}\) be the vertex for the initial basis \(B\). Then \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) is a generic vector optimized uniquely at \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}\). Then orthogonally projecting onto the \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}})\)-plane gives a polygon. By construction, \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}\) and the \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\)-maximizer are both projected onto the boundary of that polygon and are the vertices maximizing the first and second coordinate respectively. A path in the polygon between the projection of \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}\) and the projection of the \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\)-maximizer corresponds uniquely to a path on the polytope of the same length. This path is precisely the path chosen by the shadow vertex simplex method with auxiliary vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\).
The second geometric interpretation relies on a dual perspective on a polytope. Given any feasible basis \(B\), the cone \(\mathbb{R}^{d}_{\geq 0} A_{B}\) is called the normal cone of \(B\) or equivalently the normal cone of the vertex \(A_B^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B}\). It is precisely the cone of all objectives optimized at that vertex. Then the shadow vertex simplex method functions by choosing a generic vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) in the normal cone of the initial vertex \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}\). Then the path traversed consists of all vertices with normal cones that intersect the line segment \([{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}]\). Equivalently, that is precisely the set of normal cones of feasible bases that intersect the ray \(\{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}+\lambda {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}: \lambda \geq 0\}\). We call the points on \([{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}]\) or the ray \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}+ \lambda {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\) intermediate objectives.
When a linear program has no feasible solutions, any shadow path on it will be considered to have length \(0\).
These geometric interpretations are useful for understanding the algorithm and explaining its efficiency in average case, smoothed analysis, and in the by-the-book analysis we undertake here.
4 A Simplex Method with Bound Perturbations↩︎
In this section, we overview and describe how to form a complete two-phase simplex method with running time that can be analyzed using . For notational simplicity we assume that we are given a linear program of the form \[\begin{align} \operatorname{maximize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}} \label{eq:input-LP}\\ \operatorname{subject~to} \quad & A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\nonumber \\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\nonumber \end{align}\tag{3}\] with \(A \in \mathbb{R}^{n \times d}\). We assume that every row of \(A\) has \(\ell_2\) norm at most \(1\). Any LP can easily be transformed to this format. This specific form of LP will aid in a simpler presentation of our Phase I method. We will solve this LP up to tolerances \(\mathrm{feasTol}> 0\) and \(\mathrm{optTol}> 0\). See 1 and 2 for definitions of solving up to tolerances.
To start, we perturb our bound vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\) and the right-hand side vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\). Unlike the uniform distribution used in solvers, we perturb according to the following distribution:
Definition 1. Let \(\eta, \gamma \in \mathbb{R}_{>0}\), and let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}\in \mathbb{R}^{k}\). Then define for each \(i \in [k]\), \[f_{i}(t) = \frac{1}{2\eta} e^{-\abs{t - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i - \gamma\eta}/\eta}.\] Define a random vector \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}\) to be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}, \eta,\gamma)\)-exponentially distributed if its entries \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i}\) are independently distributed with each entry’s probability density function given by \(f_{i}\).
We sample \((-\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}, \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}) \in \mathbb{R}^{d+n}\) to be \(\left(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}), \frac{\mathrm{feasTol}}{4\ln(d+n)}, 2\ln(d+n)\right)\)-exponentially distributed. Strong tail bounds will ensure that these perturbed bounds are suitable:
Lemma 1. Let \(\eta, \gamma \in \mathbb{R}_{>0}\), and let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}\in \mathbb{R}^k\). Let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}\) be a \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}, \eta,\gamma)\)-exponentially distributed random vector. Then \[\Pr\big[\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i} \in [{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i} + 2\gamma \eta] \text{ for all } i \in[k]\big] \geq 1 - ke^{-\gamma}.\]
Proof. By applying a union bound we know that \[1 - \Pr\big[\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i} \in [{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i} + 2\gamma \eta] \text{ for all } i \in [k]\big] \leq \sum_{i = 1}^{k} \Pr[ \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i < {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i] + \Pr[ \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i > {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i+2\gamma\eta].\] Note that by construction the latter probabilities are symmetrically of the same size, i.e., \(\Pr[\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i < {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i] = \Pr[\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i > {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i+2\gamma\eta]\) for every \(i \in [k]\). Writing down their probability densities, we find \[\begin{align} \Pr[\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i < {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i] &= \frac{1}{2\eta} \int_{-\infty}^{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i} e^{-\abs{t-{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_i - \gamma\eta}/\eta}\operatorname{d}t \\ &= \frac{1}{2\eta} \int_{-\infty}^{-\gamma\eta} e^{-\abs{t}/\eta}\operatorname{d}t \\ &= \frac{1}{2} \int_{-\infty}^{-\gamma} e^{-\abs{t}}\operatorname{d}t = \frac{1}{2}e^{-\gamma}. \end{align}\]
Summing over all \(i \in [k]\), we see that \(1 - \Pr\big[\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i} \in [{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i} + 2\gamma \eta] \text{ for all } i \in [k]\big] \leq ke^{-\gamma}\), and hence \(\Pr\big[\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i} \in [{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{v}}} {\boldsymbol{\textstyle\boldsymbol{v}}} {\boldsymbol{\scriptstyle\boldsymbol{v}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{v}}}}_{i} + 2\gamma \eta ]\text{ for all } i\in [k]\big] \geq 1 - ke^{-\gamma}.\) ◻
By plugging in our chosen parameter values, it follows from that with probability greater than \(1-\frac{1}{d+n}\) we have \[\begin{align} -\mathrm{feasTol}\cdot \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d &\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\label{eq:perturbationswithintols}\\ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}&\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}+ \mathrm{feasTol}\cdot \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_n. \nonumber \end{align}\tag{4}\] In order to ensure that the above inequalities 4 are guaranteed to hold for our algorithm’s output, we reject any vector \((\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}, \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}})\) which violates these conditions and run the entire algorithm anew with fresh samples \((\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}, \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}})\). The expected number of samples needed is \((1-\frac{1}{n+d})^{-1}\). Wald’s equation tells us that the expected number of pivot steps when this rejection sampling is applied grows by at most a factor \((1-\frac{1}{n+d})^{-1}\) compared to when rejection sampling is inactive.
The simplex method described will, in Phase II, visit vertices of the perturbed LP \[\begin{align} \operatorname{maximize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}} \label{eq:perturbed-LP}\\ \operatorname{subject~to} \quad & A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\nonumber \\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\nonumber \end{align}\tag{5}\]
4.0.0.1 Sampling Perturbations
The space of possible perturbation distributions has not been explored systematically. At the moment of writing it is not clear what options are available when designing a perturbation distribution, nor is it clear how to evaluate the quality of any such distribution. Nevertheless, we describe here how to sample from the distribution used in this paper as an aid for readers looking to try the above perturbation distribution in their own simplex code.
The general sampling mechanism has four parameters: an offset, a lower bound for rejection sampling, an upper bound for rejection sampling, and a decay rate. The probability density is then the unique function that has its maximum at the offset, which continuously decays at the prescribed rate when going up or down from the offset to either bound, and which is zero outside the bounds. Code to sample is given in .
In the main mathematical analysis of , we assume that no rejection sampling is performed, i.e., that lowerReject\(= -\infty\) and upperReject\(= \infty\). In
order to bound the expected running time with finite rejection bounds, we choose lowerReject\(=\)offset\(- 2\ln(d+n)\)logLipschitz and
upperReject\(=\)offset\(+ 2\ln(d+n)\)logLipschitz. Using , we see that the total probability of any sample out of \(n+d\) being rejected is less than \(1-\frac{1}{n+d}\).
In our analysis we assume that we can solve systems of linear equations exactly, and we search for a solution that is feasible up to the feasibility tolerance. As such, for our analysis we choose \(\texttt{lowerReject} = 0\) and \(\texttt{upperReject} = \mathrm{feasTol}\) as per 4 . Solving for the other parameters gives \(\texttt{offset} = \mathrm{feasTol}/2\) and \(\texttt{logLipschitz} = \frac{\mathrm{feasTol}}{4\ln(n+d)}\).
4.1 Initial Basis↩︎
Traditionally, Phase I LPs have a trivial basic feasible solution from which to start. However, our tools only allow us to move from the optimum of a random objective to the optimum of a fixed objective, or from the optimum of a fixed objective to the optimum of a random objective. In order to conclude Phase I, we need to ensure that the introduced artificial variable becomes zero, which is to say, our Phase I procedure concludes at the optimum of a fixed objective. As such, in order for us to bound the time necessary to solve the Phase I LP, our Phase I procedure must start from the optimum of a random objective. In this section, we show how to do this by solving the LP (6 ) below.
Let \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^{d-1}\) be uniformly distributed. Consider the linear program \[\begin{align} \operatorname{maximize} \quad &\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\nonumber \\ \operatorname{subject~to} \quad & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} + (\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}) \nonumber \\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}.\label{eq:phase0} \end{align}\tag{6}\] This linear program will be solved in order to find the initial basis from which Phase I will be started.
Lemma 2. The point \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}= \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ (\mathrm{feasTol}/2)\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) is a basic solution to 6 . Conditional on 4 , with probability at least \(1-1/n\) this basic solution is feasible.
Proof. We can observe that \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) is part of the system defining 7 . The point satisfies these at equality, and they are clearly linearly independent. Thus, the point is a basic solution.
The point is feasible if and only if \(A(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}) \leq (\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}) + (\mathrm{feasTol}\sqrt{\ln n})\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\). Since we know \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) satisfies \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\), it suffices to show that \(A(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}) \leq (\mathrm{feasTol}\sqrt{\ln n})\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) holds with probability at least \(1- 1/n\).
Let \(f\) denote the probability density function of \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\). Recall that we chose parameters for our distribution function \(\gamma = \frac{\mathrm{feasTol}}{4 \ln(d+n)}\) and \(\eta = 2 \ln(d+n)\). By definition, the distribution function we chose for \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\) is the Laplace distribution centered at \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}- \eta \gamma\) and so \[\mathbb{E}[\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}] = {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}- \eta \gamma \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} + \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} =-\frac{\mathrm{feasTol}}{4 \ln(d+n)}(2 \ln(d+n))\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} + \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} = \mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}.\] Then, since linear transformations commute with expectations, \(\mathbb{E}[A(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}})] = {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\).
Our chosen distribution of the right-hand side perturbations is symmetric around the mean. In particular, the vector \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) follows a \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}, \eta,\gamma)\) distribution. Even after conditioning on 4 , its distribution is invariant under arbitrary sign flips. Hence the probability density function \(f\) is invariant under flipping any signs: for every \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\in \mathbb{R}^d\) we have \[f({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}_1,\dots,{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}_d) = f(\abs{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}_1},\dots,\abs{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}_d}) = f(\pm{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}_1,\dots,\pm{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}_d).\] Thus we get strong concentration according to Hoeffding’s inequality.
Let \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\varepsilon}}} {\boldsymbol{\textstyle\boldsymbol{\varepsilon}}} {\boldsymbol{\scriptstyle\boldsymbol{\varepsilon}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\varepsilon}}}\in \{+1,-1\}^d\) be sampled uniformly at random. For any \(i \in [n]\) we find \[\begin{align} \Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}}[A_i^\top(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}) \geq (\mathrm{feasTol}\sqrt{\ln n})\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}] &= \Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}}\left[\sum_{j=1}^d A_{ij} (\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}_j + \frac{\mathrm{feasTol}}{2}) \geq \mathrm{feasTol}\sqrt{\ln n}\right] \\ &= \mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}}\left[ \Pr_{\varepsilon}\left[\sum_{j=1}^d \varepsilon_j A_{ij} (\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}_j + \frac{\mathrm{feasTol}}{2}) \geq \mathrm{feasTol}\sqrt{\ln n}\right] \right]\\ &\leq \mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}}\left[ \exp\left(-\frac{2(\mathrm{feasTol}\sqrt{\ln n})^2}{\sum_{j=1}^d (2 A_{ij} \abs{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}_j + \frac{\mathrm{feasTol}}{2}} )^2 }\right) \right] \\ &\leq \mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}}\left[ \exp\left(-\frac{2(\mathrm{feasTol}\sqrt{\ln n})^2}{\sum_{j=1}^d (A_{ij} \mathrm{feasTol})^2 }\right) \right] \\ &\leq \mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}}\left[ \exp\left(-2(\sqrt{\ln n})^2\right) \right] \\ &= 1/n^2. \end{align}\] The first inequality is Hoeffding’s, the second uses that \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\geq -\mathrm{feasTol}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) holds by 4 , and the third inequality used that the rows of \(A\) each have Euclidean norm at most \(1\). Hence, any particular inequality constraint has probability at most \(1/n^2\) of being violated by \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\). Taking the union bound over all choices of \(i \in [n]\), we find that \(\Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}}[A(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}) \leq (\mathrm{feasTol}\sqrt{\ln n})\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}] \geq 1-1/n\) as required. ◻
Once again, since the failure probability is small enough we may use rejection sampling in order to assume that feasibility holds when necessary. The increase in expected number of pivot steps is no greater than a factor \((1-\frac{1}{n})^{-1}\) by another application of Wald’s equation.
Before Phase I can start, we must pivot from \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}= \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+(\mathrm{feasTol}/2)\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) to the basis optimizing 6 . Notice that \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}=\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+(\mathrm{feasTol}/2)\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) maximizes the objective \(-\sum_{i=1}^d {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_i\) over the same constraints.
Lemma 3. The optimal basic feasible solution to 6 can be found in \[O\left(\sqrt{\frac{M d^3 \ln(n)}{\eta} \ln\big(d \ln(n) \cdot \|\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d T^{-1}\|_1 \big)}\right)\] pivot steps, where \(M\) is the expected mean with of 6 and \(T\) is any invertible submatrix of \(\begin{pmatrix} A \\ - I_{d\times d} \end{pmatrix}\) for which \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} T^{-1} \geq 0\).
Proof. We will apply . When we consider 6 in standard inequality form, we observe that all rows of the extended constraint matrix \(\begin{pmatrix}A\\ -I_{d \times d}\end{pmatrix}\) have Euclidean norm at most \(1\). The right-hand side vector consists of \(\mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} + (\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}})\) and \(- \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}- \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\), which are respectively \((\mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}},\eta,\gamma)\)-exponentially distributed and \((-\frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}},\eta,\gamma)\)-exponentially distributed with \(\eta = \frac{\mathrm{feasTol}}{4\ln(d+n)}\) and \(\gamma=2\ln(d+n)\).
Choose \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}= -\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} = -\sum_{i=1}^d {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}_i\) and observe that since \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\leq{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\), \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) is still maximized by \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}=\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+(\mathrm{feasTol}/2)\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\). Thus we can choose \(\varepsilon= 1\) for our initial objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\), observing that \(\varepsilon^{-1} = 1 \geq \frac{1}{1000d^3\gamma\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|_2}\) as required. Thus we apply to find an upper bound on the expected number of pivot steps from \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) to \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) of \[102 + 316\sqrt{\frac{M d^3 \ln(n+d)}{\eta} \ln\left(\frac{6742 d^{3} N \ln^2(n+d)}{\eta} \right)},\] where \(M\) is the expected mean width of the feasible set 6 , defined in , and where \(N\) is defined in . To upper bound \(N\), since \(N\) is defined as a minimizer across all pairs of bases, we must in particular have \(N \leq N_{B^0,B^1}\) where for \(B^1\) we take the constraints \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\), i.e. the basis which maximizes the objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\), and for \(B^0\) we take any basis whose basic submatrix \(T\) of \(\begin{pmatrix} A \\ - I_{d\times d} \end{pmatrix}\) satisfies \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} T^{-1} \geq \mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}\). This basis will be dual feasible for minimizing objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\). Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}\) abbreviate the (unperturbed) right-hand side vector of 6 , such that the two sets of basic inequalities are equivalently expressed as \(A_{B^0}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^0}\) and \(A_{B^1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^1}\).
By definition, \[N_{B^0,B^1} = {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top(A_{B^1}^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^1} - A_{B^0}^{-1} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^0}) + \gamma\eta(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^{1}}^{-1}\|_{1} + \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^{0}}^{-1}\|_1).\] Once again, by our choice of \(\gamma\) and \(\eta\), we have \(\gamma \eta = \frac{\mathrm{feasTol}}{2}\). The basis \(B^1\) indicates the constraints \(-{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq -\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}- \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\). When computing \(N\) we disregard the perturbation and get \(A_{B^{1}}^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^{1}} = \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\). Furthermore, \(A_{B^{0}} = T\) by definition, and \(A_{B^{1}} = -I_{d \times d}\) also by definition. \[\begin{align} N_{B^0,B^1} &= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top(A_{B^1}^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^1} - A_{B^0}^{-1} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^0}) + \gamma\eta(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^{1}}^{-1}\|_{1} + \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^{0}}^{-1}\|_1) \\ &= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top}(\frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} -T^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^{0}}) + \frac{\mathrm{feasTol}}{2}(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|_{1} + \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top} T^{-1}\|_{1}). \\ &= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top}(\frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} - T^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^{0}}) + \frac{\mathrm{feasTol}}{2}(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top} T^{-1}\|_{1}+d). \\ &\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top(\frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}) + \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top}T^{-1}\|_1\cdot\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^{0}}\|_\infty + \frac{\mathrm{feasTol}}{2}(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top} T^{-1}\|_{1}+d) \\ &= \frac{\mathrm{feasTol}}{2}({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top} \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} +d) +\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top}T^{-1}\|_1\cdot\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^{0}}\|_\infty + \frac{\mathrm{feasTol}}{2}\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top} T^{-1}\|_{1} \\ &= \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top}T^{-1}\|_1\cdot\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^{0}}\|_\infty + \frac{\mathrm{feasTol}}{2}\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top} T^{-1}\|_{1}, \end{align}\] where to go from the second to last line we use that \(\mathbf{c} = -\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\), so \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^{\top} \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} = -d\). Finally, note that \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^0}\) is a subvector of \(\begin{pmatrix}\mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} \\ - \frac{\mathrm{feasTol}}{2}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} \end{pmatrix}\). Hence \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{f}}} {\boldsymbol{\textstyle\boldsymbol{f}}} {\boldsymbol{\scriptstyle\boldsymbol{f}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{f}}}}_{B^{0}}\|_\infty \leq \mathrm{feasTol}\sqrt{\ln n}\).
Putting this together, we find \(N \leq N_{B^0,B^1} \leq \mathrm{feasTol}(\sqrt{\ln n}+\frac{1}{2}) d \|\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d T^{-1}\|\), which we fill in to the bound. Finally, we use \(n \geq d\) to obtain \(\ln(n+d) = O(\ln(n))\) and our choices of \(\gamma\) and \(\eta\) to finish the proof. ◻
Remark 1. With \(\gamma = 2\ln(d+n)\), \(\eta = \frac{\mathrm{feasTol}}{4\ln(d+n)}\), and \(M\) the expected mean width of 6 , the ratio \(\frac{M}{\eta}\) is a measure of \(A\), independent of \(\mathrm{feasTol}\) or \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) or \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\).
4.2 Phase I↩︎
The Phase I auxiliary LP we use is stated below. We extend the objective vector \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in\mathbb{S}^{d-1}\) of the LP (6 ) to uniformly random objective \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^d\) by sampling \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_d\), appending it to \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\), and re-normalizing to ensure \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) has unit length.
\[\begin{align} \operatorname{maximize} \quad &\bar \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s) \nonumber \\ \operatorname{subject~to} \quad & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}+ \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- \mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty} s \leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\nonumber \\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} s \nonumber \\ &0 \leq s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty.\label{eq:phaseI} \end{align}\tag{7}\]
Note that the bounds/right-hand sides of the first two sets of constraints are perturbed, but not those of the constraints \(0 \leq s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\). By construction, the face with \(s=\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\) has the same solutions as 6 . The face with \(s=0\) is the set of points which are exactly feasible for 5 . Hence we will follow two shadow paths: the first from \(s + \varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^ \top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\) to \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^ \top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\) and the second from \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^ \top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\) to \(-s + \varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^ \top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\). We now determine our choice of \(\varepsilon> 0\) to make sure this path starts and ends where we need it to.
For the next lemma, we add an assumption that \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty \geq \mathrm{feasTol}\sqrt{\ln n}\). In all reasonable circumstances this should be satisfied, since \(\mathrm{feasTol}\) is a very small number and since \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty \geq 1\) should hold for all reasonable scalings of the LP data whenever \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\neq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\).
Lemma 4. For a generic \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^{d}\), let \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*, s^*)\) denote an optimal solution to \[\begin{align} \operatorname{maximize} \quad &\varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s) + s \nonumber \\ \operatorname{subject~to} \quad & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}- \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- \mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty} s \leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\nonumber \\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} s \nonumber \\ & 0 \leq s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty. \nonumber \end{align}\] Assume \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty \geq \mathrm{feasTol}\sqrt{\ln n}\). Let \(s_\mathrm{min}\) denote the minimum singular value among all feasible bases of 6 . If \(0 \leq \varepsilon\leq \frac{1}{1 + 2 \sqrt{d} s_\mathrm{min}^{-1}}\) then \(s^* = \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\).
Proof. Let \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\) be the basic solution obtained at the end of the path, where \(\varepsilon\to 0\). This solution must satisfy \(s' \geq 0\) by feasibility. We assume that 5 is feasible, and hence by this value is achieved. Since \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\) maximizes the objective \(s\), this means that \(s' = \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*={\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}'\). If \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\) is maximal for every objective \(\varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s) + s\) with every possible value \(\varepsilon> 0\) then the conclusion holds directly. Hence, we may assume for the remainder of this proof that at least one basic constraint has negative reduced cost for the objective \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\).
When \(\varepsilon= 0\), the corresponding dual solution for this basis has weight \(1\) on the constraint \(s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\) and weight \(0\) on all other constraints. Thus if \(\varepsilon\) increases infinitesimally, by genericity of \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) and optimality of \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\), the zero weights must go up and the weight on \(s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\) must go down. In order to prove the lemma’s conclusion, we require an upper bound on \(\varepsilon\) which guarantees that the weight on \(s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\) remains non-negative.
Let \(M\) be the \(d \times d\) submatrix of the \(n \times d\) matrix \(\begin{pmatrix} A \\ I_{d\times d} \end{pmatrix}\), \(M_{\cdot,d+1}\) be the corresponding subvector of \(\begin{pmatrix}-\frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- \mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_n}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty} \\ \frac{\mathrm{feasTol}}{2\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d\end{pmatrix}\), and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\) be the subvector of \(\begin{pmatrix} \hat{b} \\ \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\end{pmatrix}\) such that the basic constraints for \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\) are equivalently stated as \(M {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}+ M_{\cdot,d+1} s\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\) and \(s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\). The corresponding dual solution for an objective \(\varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)+s\) is obtained as \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\lambda}}} {\boldsymbol{\textstyle\boldsymbol{\lambda}}} {\boldsymbol{\scriptstyle\boldsymbol{\lambda}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\lambda}}}^\top= \varepsilon(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_1,\dots,\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_d) M^{-1}\) on the constraints \(M {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}+ M_{\cdot,d+1}s \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\) and a multiplier of \[1 + \varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_{d+1} - M_{\cdot,d+1}^\top\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\lambda}}} {\boldsymbol{\textstyle\boldsymbol{\lambda}}} {\boldsymbol{\scriptstyle\boldsymbol{\lambda}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\lambda}}}\label{eq:multiplierinphase0opt}\tag{8}\] on the constraint \(s\leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\). When \(\varepsilon=0\), these multipliers are respectively \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}_d\) and \(1\). Since there exist values \(\varepsilon> 0\) for which the multipliers are all non-negative, the genericity of \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) mandates that \((\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_1,\dots,\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_d) M^{-1}\) is strictly positive. As such, the vector of multipliers is non-negative so long as 8 is non-negative. Hence the basic solution \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\) is optimal for any value \(\varepsilon\geq 0\) for which \(\varepsilon\leq \frac{1}{\abs{\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_{d+1} + (\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_1,\dots,\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_d) M^{-1} M_{\cdot,d+1}}}\). Using Cauchy-Schwarz and \(\|\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\| = 1\), we find a sufficient condition that \[\varepsilon\leq \frac{1}{1 + \|M_{\cdot,d+1}\|\cdot\|M^{-1}\|}.\] By definition we have \(s_\mathrm{min} \leq \|M^{-1}\|\). Since \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty \geq 2\mathrm{feasTol}\sqrt{\ln n}\) by assumption, we find that \(\|M_{\cdot,d+1}\| \leq 2\sqrt{d}\).
Thus, by hypothesis, \[\varepsilon \leq \frac{1}{1+2\sqrt{d} s_{\min}^{-1}} \leq \frac{1}{1 + \|M_{\cdot,d+1}\|\cdot\|M^{-1}\|},\] so \(\varepsilon \overline{\theta}(\mathbf{x},s) + s\) is optimized at a vertex in the facet maximizing \(\mathbf{s}\). It follows that \(\mathbf{s}^{\ast}\) is maximal and therefore must be equal to \(\|\mathbf{b}\|_{\infty}\) as desired. ◻
Assuming that 5 is feasible, we will find a feasible solution when maximizing \(-s\).
Lemma 5. The Phase I linear program 7 has a feasible solution \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\) with \(s=0\) if and only if 5 has a feasible solution.
Proof. Directly by construction. ◻
Moreover, the solution that we find will maximize a uniformly randomly sampled objective direction.
Lemma 6. Let \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^{d}\) be arbitrary. Assume that \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*,s^*)\) is a feasible solution to 7 and that for some \(\varepsilon> 0\), \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*,s^*)\) maximizes the objective \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s) \mapsto -s + \varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\). If \(s^*=0\) then \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*\) is a feasible solution to 5 . Moreover, if \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*,s^*)\) is basic for 7 then \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*\) is basic for 5 and maximizes \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\mapsto \bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},0)\).
Proof. The feasibility of \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*\) for 5 when \(s^*=0\) is direct from the construction. When \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*,s^*)\) is basic for 7 then, since \(s^*=0\) and all basic solutions are non-degenerate with probability \(1\), the basis describing this solution includes the constraint \(s \geq 0\) and exactly \(d\) constraints out of \(A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}+ \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- (\mathrm{feasTol}\sqrt{\ln n})\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty} s \leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} s\). Filling in \(s=0\) in these constraints yields a set of \(d\) constraints which all appear in the description of 5 and which form a basis describing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*\).
The same dual solution which certifies optimality of \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*,s^*)\) for the objective \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s) \mapsto -s + \varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\) over 7 also certifies, using the same multipliers but leaving out the constraint \(0 \leq s\), the optimality of \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*\) for the objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\mapsto \varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},0)\) over 5 . ◻
Similar to , we can estimate what value of \(\varepsilon> 0\) we need to reach.
Lemma 7. For a generic \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^{d}\), let \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*, s^*)\) denote an optimal solution to \[\begin{align} \operatorname{maximize} \quad &\varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s) - s \nonumber \\ \operatorname{subject~to} \quad & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}- \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- \mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty} s \leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\nonumber \\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} s \nonumber \\ & 0 \leq s \leq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty. \nonumber \end{align}\] Assume \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty \geq \mathrm{feasTol}\sqrt{\ln n}\). Let \(s_\mathrm{min}\) denote the minimum singular value among all feasible bases of 5 . Suppose that 5 is feasible. If \(0 \leq \varepsilon\leq \frac{1}{1 + 2 \sqrt{d} s_\mathrm{min}^{-1}}\) then \(s^* = 0\).
Proof. Let \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\) be the basic solution obtained at the end of the path, where \(\varepsilon\to 0\). This solution must satisfy \(s' \geq 0\) by feasibility. We assume that 5 is feasible, and hence by this value is achieved. Since \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\) maximizes the objective \(-s\), this means that \(s' = 0\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*={\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}'\).
When \(\varepsilon= 0\), the corresponding dual solution for this basis has weight \(1\) on the constraint \(-s \leq 0\) and weight \(0\) on all other constraints. Thus if \(\varepsilon\) increases infinitesimally, by genericity of \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) and optimality of \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\), the weight on \(-s \leq 0\) must go down while all other weights go up. In order to prove the lemma’s conclusion, we require an upper bound on \(\varepsilon\) which guarantees that the weight on \(-s \leq 0\) remains non-negative.
Let \(M\) be the \(d \times d\) submatrix of the \(n \times (d+1)\) matrix \(\begin{pmatrix} A \\ -I_{d\times d} \end{pmatrix}\), \(M_{\cdot,d+1}\) be the corresponding subvector of \(\begin{pmatrix}-\frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- \mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_n}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty} \\ \frac{\mathrm{feasTol}}{2\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d\end{pmatrix}\), and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\) be the subvector of \(\begin{pmatrix} \hat{b} \\ \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\end{pmatrix}\) such that the basic constraints for \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\) are equivalently stated as \(M {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}+ M_{\cdot,d+1} s\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\) and \(-s \leq 0\). The corresponding dual solution for an objective \(\varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)-s\) is obtained as \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\lambda}}} {\boldsymbol{\textstyle\boldsymbol{\lambda}}} {\boldsymbol{\scriptstyle\boldsymbol{\lambda}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\lambda}}}^\top= \varepsilon(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_1,\dots,\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_d) M^{-1}\) on the constraints \(M {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\) and a multiplier of \[1 - \varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_{d+1} - M_{\cdot,d+1}^\top\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\lambda}}} {\boldsymbol{\textstyle\boldsymbol{\lambda}}} {\boldsymbol{\scriptstyle\boldsymbol{\lambda}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\lambda}}}\label{eq:multiplierinphase1opt}\tag{9}\] on the constraint \(-s\leq 0\). When \(\varepsilon=0\), these multipliers are respectively \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\) and \(1\). Since there exist values \(\varepsilon> 0\) for which the multipliers are all non-negative, the genericity of \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) mandates that \((\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_1,\dots,\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_d) M^{-1}\) is strictly positive. As such, the vector of multipliers is non-negative so long as 9 is non-negative. Hence the basic solution \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}',s')\) is optimal for any value \(\varepsilon\geq 0\) for which \(\varepsilon\leq \frac{1}{\abs{\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_{d+1} + (\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_1,\dots,\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_d) M^{-1} M_{\cdot,d+1}}}\). Using Cauchy-Schwarz and \(\|\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\| = 1\), we find a sufficient condition that \[\varepsilon\leq \frac{1}{1 + \|M_{\cdot,d+1}\|\cdot\|M^{-1}\|}.\] By definition we have \(s_\mathrm{min} \leq \|M^{-1}\|\). Since \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty \geq \mathrm{feasTol}\sqrt{\ln n}\) by assumption, we find that \(\|M_{\cdot,d+1}\| \leq 2\sqrt{d}\). This proves the lemma. ◻
We will bound our total Phase I running time in terms of the expected mean width of a slightly larger set than our Phase I LP. The difference in size is minor.
Definition 2. Let \((-\hat{L}, \hat{U})\) be \(((0,U),\eta,\gamma)\)-exponentially distributed, independent of \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\). Let \(\mathcal{M}\) denote the expected mean width of the following set \[\begin{align} \{({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}, s) \in \mathbb{R}^{d+1} : & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}+ \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- \mathrm{feasTol}\sqrt{\ln n}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty} s \leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}, \nonumber \\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+ \frac{\mathrm{feasTol}}{2\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} s \nonumber, \\ &\hat{L} \leq s \leq \hat{U} \} \label{eq:phaseIextraperturb} \end{align}\qquad{(1)}\]
Theorem 2. Let \(\kappa\) denote the maximum condition number among all \(d \times d\) submatrices of \(\begin{pmatrix} A \\ -I_{d\times d} \end{pmatrix}\). Assume \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty \geq \mathrm{feasTol}\sqrt{\ln n}\). The proposed simplex method satisfies the following properties:
\(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^0,s^0)=(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}+(\mathrm{feasTol}/2)\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}},\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty)\) is a basic feasible solution to 6 .
The initial basis from which to start Phase I is found after an expected number of pivot steps no greater than \[O\left(d^{1.5} \ln(n) \sqrt{\frac{\mathcal{M}}{\mathrm{feasTol}} \ln\left(\frac{d \ln(n) \cdot \kappa}{\mathrm{feasTol}} \right)}\right)\]
Phase I uses an expected number of pivot steps no greater than \[O\left(d^{1.5} \ln(n) \sqrt{\frac{\mathcal{M}}{\mathrm{feasTol}}\ln\left(\frac{d \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty \ln(n) \kappa}{\mathrm{feasTol}}\right)}\right)\] to find a feasible basis for 5 .
The basis of 5 returned by Phase I maximizes a uniformly random objective direction \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^{d-1}\).
Proof. The first point holds by construction, because we used rejection sampling to ensure that it was satisfied. The second point is established using , which gives an upper bound of \[O\left(\sqrt{\frac{M^0 d^3 \ln(n)}{\eta} \ln\left(\frac{d \ln(n) \cdot \gamma \cdot \|\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d T^{-1}\|_1}{\eta} \right)}\right),\] where \(M^0\) denotes the expected mean width of 6 . We now fill in all the quantities used in this bound. We have \(\eta = \frac{\mathrm{feasTol}}{4\ln(d+n)}\) and \(\gamma=2\ln(d+n)\). The rows of \(A\) each have norm at most \(1\), which means that any basis matrix from \(\begin{pmatrix} A \\ -I_{d\times d} \end{pmatrix}\) has operator norm at most \(d\). Since the condition number of a matrix \(T\) is \(\|T\|\cdot\|T^{-1}\|\), we know that the vector \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d T^{-1}\) satisfies \[\|\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d T^{-1}\|_1 \leq d \|T^{-1}\| \leq d^2 \kappa.\] Finally, to obtain the desired statement, we must bound the expected mean width \(M^0\) of 6 . To achieve that, notice that the feasible set of 6 is a lower-dimensional slice of ?? , namely the slice where \(s=0\). As such, the expected mean width \(M^0\) is at most the mean width of ?? divided by a factor \(1 - O(\frac{1}{\sqrt{d}}) \leq \mathbb{E}[\|(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_1,\dots,\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_d)\|] \leq 1\). This scaling factor is to account for the fact that the mean width of the slice is being measured in a lower dimension.
For the third point, we traverse shadow paths on the feasible set 7 . First we follow the shadow path from fixed objective \(s\) to random objective \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\). As stated in , the solution we found by solving 6 gives a starting basis for this path which is optimal for \(s+\varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\) with \(\varepsilon=\frac{1}{1+2\sqrt{d}s_\mathrm{min}^{-1}}\). Observe that, when we write the constraints of 7 as a standard inequality form LP, then the rows of the constraint matrix each have norm at most \(3\). We can still apply by scaling down all inequalities by a factor \(3\), which reduces the effective value of \(\eta\) by the same factor \(3\). This part of the shadow path has expected length at most \[113 + 344d \sqrt{\frac{d \ln(n+d+2) \mathcal{M}}{\eta/3} \ln\left(\frac{6742d^{3}(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty + 2\eta\gamma/3)\ln^2 (n+d+2)}{\eta \varepsilon/3} \right)}.\] We further fill in the quantities in the bound. We bound \(s_\mathrm{min}^{-1} \leq d^2\kappa\) as before and fill in \(\varepsilon=\frac{1}{1+2\sqrt{d}s_\mathrm{min}^{-1}}\) and \(\eta = \frac{\mathrm{feasTol}}{4\ln(d+n)}\) and \(\gamma=2\ln(d+n)\). This gets an asymptotic running time of \[O\left(d^{1.5} \ln(n) \sqrt{\frac{\mathcal{M}}{\mathrm{feasTol}} \ln\left(\frac{d\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\|_\infty\ln (n)\kappa}{\mathrm{feasTol}} \right)}\right).\]
The next shadow path traverses the feasible set 7 , now from the random objective \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\) to the fixed objective \(-s\). We stop at the intermediate objective \(-s+\varepsilon\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},s)\) with \(\varepsilon=\frac{1}{1+2\sqrt{d}s_\mathrm{min}^{-1}}\). As shown in , this yields a solution with \(s=0\). We see in that this solution satisfies the properties claimed in the final point of this theorem. The running time bound for this final path of Phase I is identically obtained as the previous path. ◻
4.3 Phase II↩︎
After the end of Phase I, the current basis of the algorithm is optimal for \[\begin{align} \operatorname{maximize} \quad & \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ \operatorname{subject~to} \quad & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}. \end{align}\] As such, we can follow the shadow path starting at the auxiliary objective \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) in order to optimize the linear program \[\begin{align} \operatorname{maximize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ \operatorname{subject~to} \quad & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}. \end{align}\] up to optimality tolerance \(\mathrm{optTol}> 0\) as follows. We terminate upon finding the optimal basic feasible solution \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^* \in \mathbb{R}^d\) to the intermediate linear program \[\begin{align} \operatorname{maximize} \quad & ({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \mathrm{optTol}\cdot \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}})^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\label{eq:intermediatelp}\\ \operatorname{subject~to} \quad & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\nonumber\\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}. \nonumber \end{align}\tag{10}\] Here, the conditions of are satisfied: the objective and constraint matrix are fixed, all rows of the constraint matrix have norm no greater than \(1\), the right-hand side and bound vector and the objective are sampled appropriately. Our Phase II algorithm returns its solution after an expected number of pivot steps no more than \[(1-\frac{1}{n})^{-1}(1-\frac{1}{d+n})^{-1}\cdot\left(102 + 316 d^{1.5}\ln(n+d)\sqrt{\frac{M}{\mathrm{feasTol}}\ln\left(\frac{6742d^3N\ln^3(n+d)}{\mathrm{optTol}\cdot\mathrm{feasTol}}\right)}\right),\] where \(N\) is defined in and indicates the expected maximum absolute objective value of any feasible solution.
Based on our observations, which can be found in , reasonable values for the remaining parameters are a mean width of \(M \leq 1000\) and tolerances of \(\mathrm{feasTol}= \mathrm{optTol}= 10^{-6}\). The expected maximum absolute objective value, which appears in the logarithm, should easily be bounded as \(N \leq 10^9\).
Besides the optimal basic feasible solution to the primal 5 , the algorithm will yield the optimal basic feasible solution \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{t}}} {\boldsymbol{\textstyle\boldsymbol{t}}} {\boldsymbol{\scriptstyle\boldsymbol{t}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{t}}}}^*) \in \mathbb{R}^{n + n + d}\) to the intermediate dual linear program \[\begin{align} \operatorname{minimize} \quad & \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}- \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}^\top t \\ \operatorname{subject~to} \quad & A^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}- {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{t}}} {\boldsymbol{\textstyle\boldsymbol{t}}} {\boldsymbol{\scriptstyle\boldsymbol{t}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{t}}}}= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \mathrm{optTol}\cdot \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\\ & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{t}}} {\boldsymbol{\textstyle\boldsymbol{t}}} {\boldsymbol{\scriptstyle\boldsymbol{t}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{t}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}. \end{align}\] We can now verify that the primal dual solution pair satisfy the approximate feasibility and complementary slackness conditions described in . The primal feasibility conditions \[\begin{align} A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^* &\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}+ \mathrm{feasTol}\cdot \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_n \\ -\mathrm{feasTol}\cdot\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d &\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^* \end{align}\] follow immediately from 4 . The dual feasibility conditions \[\begin{align} A^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^* &\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}- \mathrm{optTol}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d \\ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^* &\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}} \end{align}\] follow because \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\geq-\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}_d\). Note that in the optimal solution we have \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{t}}} {\boldsymbol{\textstyle\boldsymbol{t}}} {\boldsymbol{\scriptstyle\boldsymbol{t}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{t}}}}^* = A^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^* - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}- \mathrm{optTol}\cdot \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\). To certify the approximate optimality of the pair \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*\) we derive approximate complementary slackness conditions. Exact complementary slackness for 10 , along with 4 , gives us that \[\begin{align} \text{~for all~} i \in [n] &\text{,~if~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*_i > 0 \text{~then~} (A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*)_i = \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_i \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_i. \\ \text{~for all~} j \in [d] &\text{,~if~} (A^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*)_j > {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}_j + \mathrm{optTol}\cdot \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_j \text{~then~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_j^* = \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}_j \leq 0. \end{align}\] Note that \(\|\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\|=1\), which means that \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}_j < (A^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*)_j - \mathrm{optTol}\) implies \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}_j < (A^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*)_j - \mathrm{optTol}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}_j\), which in turn implies \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_j^* \leq 0\). It follows that the returned solution pair \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*,{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*\) satisfies the following approximate complementary slackness conditions: \[\begin{align} \text{~for all~} i \in [n] &\text{,~if~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*_i > 0 \text{~then~} (A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*)_i \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_i. \\ \text{~for all~} j \in [d] &\text{,~if~} (A^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^*)_j > c_j + \mathrm{optTol}\text{~then~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_j^* \leq 0. \end{align}\] This is what we promised the algorithm output would satisfy: feasibility and optimality for 3 up to tolerances as in 1 and 2 .
Note that we do not require to relax the dual nonnegativity constraint \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\). In real solvers this is likely relaxed to \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\geq -\mathrm{feasTol}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) because it is obtained through the (inexact) solving of a system of linear equations. Since we assume that our algorithm may solve such systems exactly, we are able to return solutions with \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\).
5 Running Time Bound↩︎
The proof will proceed as follows. In we use bound perturbations to prove that bases are either unlikely to be feasible or have large slacks on all non-tight constraints with constant probability when feasible. In we will use the semi-random shadow vertex method to prove that if bases are dual feasible for some objective, then with constant probability all of their multipliers will be large and positive for some (potentially distinct) intermediate objective. Finally in we use these two properties to prove an upper bound on the length of the semi-random shadow path with bound perturbations, which is the property stated in .
5.1 Bases with Large Slack↩︎
In this section we study the effect of the perturbations defined in . The bound perturbations will have the effect that vertices which are feasible will likely have large slacks on the non-tight constraints. We first prove this fact for fixed (not necessarily basic) solutions.
Lemma 8. Let \(A \in \mathbb{R}^{k \times d}\) be arbitrary and \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^k\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Let \(\omega > 0\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\in \mathbb{R}^{d}\) be fixed. Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}= \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}- A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\in \mathbb{R}^{k}\) denote the slack vector. Then we have \[\begin{align} \Pr\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \big| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big] \leq 62 \frac{\omega}{\eta} \ln(\frac{1}{\Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}]}). \end{align}\]
Proof. First, by independence of the random variables \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j\) we find that \[\begin{align} \label{align:exists} \Pr\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \big| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big] &\leq \sum_{j \in [k]} \Pr\big[ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \big | {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \geq 0 \big]. \end{align}\tag{11}\]
Second, let \(V \mathrel{\vcenter{:}}= \{j \colon {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j <{\mathchoice{\mbox{\boldmath\)\bf 0\(}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\}\) be the set of violated constraints. Since the numbers \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j\) are independent, the Chernoff bound (see, e.g., Theorem 4.5 in [85])gives us that \(\Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}] = \Pr\big[\abs{V} = 0\big] \leq \exp(-\mathbb{E}[\abs{V}]/2)\). Now we need to relate 11 with the set of violated constraints. Therefore we use (see [47]) which we can apply to the entries of the \((\overline{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}},\eta,\gamma)\)-exponential distribution directly and obtain that \(\Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_{j} \leq \omega | {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_{j} \geq 0] \leq 31 \frac{\omega}{\eta} \Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_{j} < 0]\). Putting everything together gives us \[\begin{align} \Pr\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \big| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big] &\leq \sum_{j \in [k]} \Pr\big[ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \big | {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_{j} \geq 0 \big] \\ & \leq \sum_{j \in [k]} 31 \frac{\omega}{\eta} \Pr\big[ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_{j} < 0 \big] \\ & = 31 \frac{\omega}{\eta} \mathbb{E}[\abs{V}] \\ & \leq 62 \frac{\omega}{\eta} \ln(\frac{1}{\Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}]}). \end{align}\] ◻
We now extend the above result to the slacks on non-tight constraints for basic feasible solutions.
Lemma 9. Let \(A \in \mathbb{R}^{n \times d}\) be arbitrary and \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed, and let \(\omega > 0\). Assume \(B \subset[n]\) is a basis and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B = A_B^{-1}\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B}\). If \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B \in \mathbb{R}^d\) satisfies \(\Pr[A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B \leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}] \geq n^{-d}\), then \[\begin{align} \Pr\big[ \exists j \in [n]\setminus B \; : \; A_j {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B \geq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_j - \omega \big| A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B \leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\big] \leq 124 d \frac{\omega}{\eta} \ln(n) + n^{-d}. \end{align}\]
Proof. Assume without loss of generality that \(B = \{n-d+1,\dots,n\}\). Write \(k = n-d\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\in \mathbb{R}^k\) with \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}= \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B} - A_{[n]\setminus B}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B\). This allows us to write \[\Pr\big[ \exists j \in [n]\setminus B \; : \; A_j {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_j - \omega \big| A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B \leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\big] = \Pr\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \big| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big].\] We separate the randomness in \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) into the part on \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B\) and the part on \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}\) and get \[\label{eq:splitrandomness} \Pr\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \text{~and~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big] = \mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B}\Big[\Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}}\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \text{~and~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big]\Big].\tag{12}\] In other words, by the principle of deferred decisions, we may condition on the value of \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B}\) and therefore assume it to be fixed. In particular, for any fixed value of \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B\), we will prove an upper bound on \(\Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}}\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \big| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big]\). If \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B\) satisfies \(\Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}] \geq n^{-2d}\) then we apply to find
\[\begin{align} \Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}}\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \text{~and~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big]&\leq 62 \frac{\omega}{\eta} \ln\left(\frac{1}{\Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}]}\right)\Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}]\\ &\leq 124 d \frac{\omega}{\eta} \ln(n)\Pr[s \geq 0]. \end{align}\] If \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B\) satisfies \(\Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}] \leq n^{-2d}\) then we upper bound \[\Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}}\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \text{~and~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big]\leq \Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}}\big[ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big] \leq n^{-2d}.\] Combining the two cases, we find that for every value of \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B\) it holds that \[\Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}}\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \text{~and~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big]\leq124 d \frac{\omega}{\eta} \ln(n)\Pr_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B}}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}] + n^{-2d}.\] In particular, by taking expectations over \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B}\) of both sides of this inequality, this pointwise upper bound can now be applied to 12 to find \[\Pr\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \text{~and~} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big] \leq 124 d \frac{\omega}{\eta} \ln(n)\Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}] + n^{-2d}\]
Now, by dividing both sides by \(\Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}]\) and using \(\Pr[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}] \geq n^{-d}\), we conclude that \[\Pr\big[ \exists j \in [k] \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \big| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big] \leq 124 d \frac{\omega}{\eta} \ln(n) + n^{-d},\] which finishes the proof. ◻
The lemmas above allow us to prove our main technical result about bound perturbations: for any fixed set of bases, the expected number of feasible bases among them is comparable to the expected number of feasible bases with large slacks.
Definition 3. Given arbitrary \(A \in \mathbb{R}^{n \times d}\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\), as well as a value \(\eta > 0\), define the set of feasible bases \(B\) as \[\mathcal{F}(A) = \left\{B \in \binom{[n]}{d} : \text{A_B invertible and~} A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\right\}.\] When \(A\) is clear from context or arbitrary, we may refer to \(\mathcal{F}(A)\) as just \(\mathcal{F}\).
We define \(\omega(\eta,n,d) = \frac{\eta}{1240 d \ln(n)}\), and we define the set of feasible bases with \(\eta\)-good slack* as \[\mathcal{F}^+(A,\eta) = \left\{B \in \mathcal{F} \colon A_{[n]\setminus B} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_B \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n]\setminus B} - \omega(\eta,n,d) \right\}.\] When \(\eta,n\) and \(d\) are clear from context, we may simply refer to \(\omega(\eta,n,d)\) as \(\omega\). When \(A\) and \(\eta\) are clear from context or arbitrary, we may refer to \(\mathcal{F}^+(A,\eta\)) as just \(\mathcal{F}^+\), and we may refer to it simply as the feasible bases with good slack.*
Theorem 3. (Slacks are large)Let \(A \in \mathbb{R}^{n \times d}\) be arbitrary and \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Let \(\mathcal{S} \subseteq \binom{[n]}{d}\) be arbitrary. Then \[\mathbb{E}[\abs{\mathcal{S} \cap \mathcal{F}^+}] \geq 0.9 \mathbb{E}[\abs{\mathcal{S} \cap \mathcal{F}}] - 2.\]
Proof. Let \(\mathcal{S}' := \{B \in \mathcal{S} : \Pr[B \in \mathcal{F}] \geq n^{-d}\}\) denote those bases in \(\mathcal{S}\) which have non-negligible probability of being feasible. Observe that \[\mathbb{E}[\abs{\mathcal{S} \cap \mathcal{F}^+}] \geq \mathbb{E}[\abs{\mathcal{S}' \cap \mathcal{F}^+}]\] follows directly from the fact that \(\mathcal{S}' \subseteq \mathcal{S}\).
Let \(B \in \mathcal{S}'\) index a basis and let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B = A_B^{-1}\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_B\). Recall that \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\mathrel{\vcenter{:}}= \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{[n] \setminus B} - A_{[n] \setminus B} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^B \in \mathbb{R}^{n-d}\). Because \(B \in \mathcal{S}'\) we have \(\Pr[B \in \mathcal{F}] \geq n^{-d}\), so we may apply to find for \(\omega = \omega(\eta,n,d)\) that \[\begin{align} \Pr[B \notin \mathcal{F}^+ \mid B \in \mathcal{F}] &= \Pr[\exists j \in [n]\setminus B \; : \; {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}_j \leq \omega \big| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{s}}} {\boldsymbol{\textstyle\boldsymbol{s}}} {\boldsymbol{\scriptstyle\boldsymbol{s}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{s}}}}\geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\big] \\ &\leq 124 \frac{\omega}{\eta} d \ln(n) + n^{-d} \leq 0.1 + n^{-d}. \end{align}\] Hence, \(\Pr[B \in \mathcal{F}^+ \mid B \in \mathcal{F}] \geq 0.9 - n^{-d}\). The result follows by summing up over all \(B \in \mathcal{S}\): \[\begin{align} \mathbb{E}[\abs{\mathcal{S}' \cap \mathcal{F}^+}] &= \sum_{B \in \mathcal{S}'} \Pr[B \in \mathcal{F}^+] \\ &= \sum_{B \in \mathcal{S}'} \Pr[B \in \mathcal{F}^+| B \in \mathcal{F}]\Pr[B \in \mathcal{F}] \\ &\geq \sum_{B \in \mathcal{S}'} (0.9 - n^{-d})\Pr[B \in \mathcal{F}] \\ &= (0.9-n^{-d}) \mathbb{E}[\abs{\mathcal{S}' \cap \mathcal{F}}] \\ &\geq 0.9 \mathbb{E}[\abs{\mathcal{S}' \cap \mathcal{F}}] - 1. \end{align}\] Putting the above together, we find \(\mathbb{E}[\abs{\mathcal{S} \cap \mathcal{F}^+}] \geq 0.9 \mathbb{E}[\abs{\mathcal{S}' \cap \mathcal{F}}] - 1\). Finally, we note that \[\mathbb{E}[\abs{(\mathcal{S}\setminus \mathcal{S}') \cap \mathcal{F}}] = \sum_{B \in \mathcal{S}\setminus \mathcal{S}'} \Pr[B \in \mathcal{F}] \leq 1\] holds by construction, which implies that \(\mathbb{E}[\abs{\mathcal{S} \cap \mathcal{F}^+}] \geq 0.9 \mathbb{E}[\abs{\mathcal{S} \cap \mathcal{F}}] - 2\) as required. ◻
5.2 Bases with Large Multipliers↩︎
In this subsection we prove one main ingredient for our proof of . We show that for any (normal) cone we have large multipliers with high probability.
Definition 4. A function \(f : \mathbb{R}^d \to \mathbb{R}_+\) is \(L\)-log-Lipschitz if for any \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}' \in \mathbb{R}^d\) it holds that \(f({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}})/f({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}') \leq \exp(L\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}-{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}'\|)\). A probability distribution is \(L\)-log-Lipschitz if it admits a probability density function with respect to the Lebesgue measure and if this probability density function is \(L\)-log-Lipschitz.
Definition 5. Let \(L > 0\). A random variable \(X \in \mathbb{R}^d\) is \(L\)-exponentially distributed on \(\mathbb{R}^d\) if there is a constant \(C_L\) such that \[\Pr[X \in S] = \int_{S} C_L e^{-L\|x\|} \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\] for every measurable set \(S \subset \mathbb{R}^d\)
The following fact is standard, we adapt the proof from [60].
Lemma 10. The normalizing constant \(C_L\) of the \(L\)-exponential distribution is \(C_L = \frac{L^d}{d! \operatorname{vol}_d(\operatorname{\mathbb{B}_2}^d)}\). For \(X\) exponentially distributed on \(\mathbb{R}^d\), the \(k\)’th moment of \(\|X\|\) is \(\mathbb{E}[\|X\|^k] = \frac{L^{-k} (k+d-1)!}{(d-1)!}\).
Proof. Without loss of generality, take \(L=1\). For the normalizing constant, \[\begin{align} C^{-1} &= \int_{\mathbb{R}^d} e^{-\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\|} \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}} = \int_{\mathbb{R}^d} \int_{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\|}^\infty e^{-t} {\rm dt} \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ &= \int_0^\infty e^{-t} \int_{\mathbb{R}^d} 1[\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\| \leq t] \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\operatorname{d}t = \int_0^\infty e^{-t} t^d \operatorname{vol}_d(\mathbb{B}^d) \operatorname{d}t = d! \operatorname{vol}_d(\mathbb{B}^d) \text{.} \end{align}\] To compute the \(k\)’th moment of the norm, \[\begin{align} \mathbb{E}[\|X\|^k ] &= C \int_{\mathbb{R}^d} \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\|^k e^{-\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\|} \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}} = C \int_{\mathbb{R}^d} \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\|^k \int_{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\|}^\infty e^{-t} {\rm dt} \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ &= C \int_0^\infty e^{-t} \int_{t\mathbb{B}^d} \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\|^k \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\operatorname{d}t = C \int_0^\infty e^{-t} t^{d+k} \operatorname{d}t \int_{\mathbb{B}^d} \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\|^k \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ &= C \int_0^\infty e^{-t} t^{d+k} \operatorname{d}t \int_{\mathbb{B}^d} \int_0^1 1[\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\| \geq s] k s^{k-1} \operatorname{d}s \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ &= C \int_0^\infty e^{-t} t^{d+k} \operatorname{d}t \operatorname{vol}_d(\mathbb{B}^d) \int_0^1 (1-s^d) k s^{k-1} \operatorname{d}s \\ &= C (d+k)! \int_0^1 k (1-s^{d+k-1}) \operatorname{vol}_d(\mathbb{B}^d) \operatorname{d}s \\ &= C (d+k)! \operatorname{vol}_d(\mathbb{B}^d) \frac{d}{d+k} = \frac{(d+k-1)!}{(d-1)!} \text{.} \qedhere \end{align}\] ◻
For the exponential distribution we have the following tail bound.
Lemma 11. Let \(X\) be \(L\)-exponentially distributed on \(\mathbb{R}^d\) with some \(L > 0\). Then we have \[\Pr[\|X\| \geq 2ed L^{-1} \ln n] \leq n^{-d}.\]
Proof. Without loss of generality we assume \(L=1\). Using Markov’s inequality we know for \(k = d \ln n\) and \(t=2ed\ln n\) that \[\begin{align} \Pr[\|X\| > t] &= \Pr[\|X\|^k > t^k] \\ &\leq \frac{\mathbb{E}[\|X\|^k]}{t^k} \\ &\leq \frac{(k+d)^k}{t^k} \\ &\leq \frac{(2d\ln n)^{d \ln n}}{(2ed\ln n)^{d \ln n}} = n^{-d}. \qedhere \end{align}\] ◻
For any \(L\)-log-Lipschitz distribution, and in particular for the \(L\)-exponential distribution, we will see that a line segment shifted by a random offset will tend to intersect any cone through its center.
Lemma 12. Let \(M \in \mathbb{R}^{d\times d}\) be an invertible matrix, and let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}\) be the sum of the row vectors of \(M\). Then we define for any \(m \geq 0\), the cone \(C_{m} = \{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\in \mathbb{R}^{d}: {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^\top M^{-1} \geq m \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\}\). Fix any \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}' \in \mathbb{R}^{d}\) and \(m > 0\). Then for any random vector \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\in \mathbb{R}^{d}\) with \(L\)-log-Lipschitz probability distribution \(\mu\), \[\begin{align} \Pr\Big[[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}' + {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}] \cap C_{m} \neq \emptyset\Big] \geq e^{-Lm\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}\|_{2}}\Pr\Big[[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}' + {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}] \cap C_{0} \neq \emptyset\Big] . \end{align}\]
Proof. Since \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}\) is the sum of the rows of \(M\) and \(M\) is invertible, \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}^\top M^{-1} = \sum_{i=1}^{d} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}^{i} = \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\). Then \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\in C_{0}\) if and only if \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^\top M^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\), which is true if and only if \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}+ m {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}})^\top M^{-1} \geq m \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\). This in turn is true if and only if \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}+ m {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}\in C_{m}\).
Note that \(\Pr[[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}'+{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}] \cap C_{0} \neq \emptyset] = \int_{-[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}'] + C_{0}} \mu({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}) d {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\). Then, since \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\in C_{m}\) if and only if \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}- m{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}\in C_{0}\), \[\begin{align} \int_{-[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}'] + C_{0}} \mu({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}) d {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}= \int_{-[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}'] + C_{m}} \mu({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}- m{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}) d{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}. \end{align}\] Then, by applying our assumption that \(\mu\) is \(L\)-log-Lipschitz, we get that \[\begin{align} \int_{-[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}'] + C_{m}} \mu({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}-m{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}) \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\leq \int_{-[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}'] + C_{m}} \mu({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}})e^{L m\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}\|_{2}} \operatorname{d}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}= e^{Lm\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{p}}} {\boldsymbol{\textstyle\boldsymbol{p}}} {\boldsymbol{\scriptstyle\boldsymbol{p}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{p}}}}\|_{2}}\Pr[[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}' + {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}] \cap C_{m} \neq \emptyset]. \end{align}\] ◻
Definition 6. Let \(A \in \mathbb{R}^{n \times d}\), let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\in \mathbb{R}^d\) and let \(t_2 > t_1 \geq 0\) and \(L > 0\) be arbitrary. We define \[\mathcal{M}_{[t_1,t_2]}(A,{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}) = \left\{B \in \binom{[n]}{d} : \text{A_B invertible and~} \exists \; t \in [t_1,t_2] \text{ such that } (t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^{\top}A_{B^{i}}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}}
{\boldsymbol{\textstyle\boldsymbol{0}}}
{\boldsymbol{\scriptstyle\boldsymbol{0}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\right\},\] and we call \(\mathcal{M}_{[t_1,t_2]}(A,{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})\) the set of bases with positive \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})\)-multipliers on \([t_1,t_2]\). When \(A\), \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\), \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\), \(t_1\) and \(t_2\) are clear from context or arbitrary, we may refer to \(\mathcal{M}_{[t_1,t_2]}(A,{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})\) by just \(\mathcal{M}\), and may refer to is simply as the set of bases with positive multipliers.
We define \[\mathcal{M}_{[t_1,t_2]}^+(A,{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}},L) = \left\{B \in \binom{[n]}{d} \colon \exists \; t \in [t_1,t_2] \text{ such that } (t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^{\top}A_{B^{i}}^{-1} \geq \frac{\ln(1/0.9)}{dL}\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}}
{\boldsymbol{\textstyle\boldsymbol{1}}}
{\boldsymbol{\scriptstyle\boldsymbol{1}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{1}}} \right\},\] and we call \(\mathcal{M}_{[t_1,t_2]}^+(A,{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}},L)\) the set of bases with \(L\)-good \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})\)-multipliers on \([t_1,t_2]\). When \(A\), \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\), \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\), \(t_1\), \(t_2\) and \(L\) are clear from context or arbitrary, we may refer to \(\mathcal{M}_{[t_1,t_2]}^+(A,{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}},L)\) by just \(\mathcal{M}^+\), and may refer to it simply as the set of bases with good multipliers.
Theorem 4. (Multipliers are large)Let \(\mathcal{S} \subseteq \binom{[n]}{d}\) and \(t_2 > t_1 \geq 0\) and \(L>0\) be arbitrary. Let \(A \in \mathbb{R}^{n \times d}\) be a matrix with rows of Euclidean norm at most \(1\), \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \mathbb{R}^d\) be arbitrary and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\in \mathbb{R}^d\) be \(L\)-log-Lipschitz distributed. Then \[\mathbb{E}[\abs{\mathcal{S} \cap \mathcal{M}^+}] \geq 0.9 \mathbb{E}[\abs{\mathcal{S} \cap \mathcal{M}}].\]
Proof. Let \(C_0^B = \{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\in \mathbb{R}^{d}: {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^\top A_B^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\}\) and \(C_{\frac{\ln(1/0.9)}{dL}}^B = \{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}\in \mathbb{R}^{d}: {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{y}}} {\boldsymbol{\textstyle\boldsymbol{y}}} {\boldsymbol{\scriptstyle\boldsymbol{y}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{y}}}}^\top A_B^{-1} \geq \frac{\ln(1/0.9)}{dL} \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\}\). Let further \(B \in \mathcal{S}\) be arbitrary. By the construction in the proof of we have that \[\Pr[B \in \mathcal{M}^+] = \Pr\Big[[t_1 {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}, t_2 {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}] \cap C^B_{\frac{\ln(1/0.9)}{dL}} \neq \emptyset\Big]\] and similarly \(\Pr[B \in \mathcal{M}] = \Pr\Big[[t_1 {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}, t_2 {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}] \cap C^B_0 \neq \emptyset\Big]\). By assumption we know that \(\|p\| \leq d\). Plugging the values of \(p\) and \(m\) into , it follows that \(\Pr[B \in \mathcal{M}^+] \geq 0.9 \Pr[B \in \mathcal{M}]\). Putting everything together, we find \[\begin{align} \mathbb{E}[\abs{\mathcal{S} \cap \mathcal{M}^+}] = \sum_{B \in \mathcal{S}} \Pr[B \in \mathcal{M}^+] \geq \sum_{B \in \mathcal{S}} 0.9 \Pr[B \in \mathcal{M}] &= 0.9 \mathbb{E}[\abs{\mathcal{S} \cap \mathcal{M}}] \end{align}\] as required. ◻
5.3 Bases with Both↩︎
In this subsection we first combine the good slack and good multiplier properties which we have investigated during the last two subsections. Afterwards we use this result for deducing our final .
Lemma 13. Let \(t_2 > t_1 \geq 0\) and \(L > 0\), let \(A \in \mathbb{R}^{n \times d}\) be a matrix with rows of Euclidean norm at most \(1\), let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \mathbb{R}^d\), and let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Assume that \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\in \mathbb{R}^d\) follows an \(L\)-log-Lipschitz probability distribution and that is independent from \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\). We have \[\mathbb{E}[\abs{\mathcal{M}^+ \cap \mathcal{F}^+}] \geq 0.8\mathbb{E}[\abs{\mathcal{M} \cap \mathcal{F}}] - 2\]
Proof. We use the principle of deferred decisions to separate the randomness over \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) and over \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) as follows \[\mathbb{E}[\abs{\mathcal{M}^+ \cap \mathcal{F}^+}] = \mathbb{E}_{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}\big[\mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}}[\abs{\mathcal{M}^+ \cap \mathcal{F}^+}] \big].\] Observe that \(\mathcal{M}^+\) does not depend on the value of \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\). Using with \(\mathcal{S} = \mathcal{M}^+\) we find for every value of \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) that \[\mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}}[\abs{\mathcal{M}^+ \cap \mathcal{F}^+}] \geq 0.9\mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}}[\abs{\mathcal{M}^+ \cap \mathcal{F}}] - 2.\] We can once again reorder expectations and find \[\begin{align} \mathbb{E}[\abs{\mathcal{M}^+ \cap \mathcal{F}^+}] &\geq \mathbb{E}_{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}\big[0.9\mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}}[\abs{\mathcal{M}^+ \cap \mathcal{F}}] - 2\big] \\ &= 0.9\mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}}\big[\mathbb{E}_{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}[\abs{\mathcal{M} \cap \mathcal{F}}]\big] - 2 \end{align}\] For any fixed value of \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\), we can observe that \(\mathcal{F}\) is fixed. As such, with \(\mathcal{S} =\mathcal{F}\) we can apply to show \[\mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}}\big[\mathbb{E}_{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}[\abs{\mathcal{M}^+ \cap \mathcal{F}}]\big] \geq \mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}}\big[0.9 \mathbb{E}_{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}[\abs{\mathcal{M}^+ \cap \mathcal{F}}]\big]\] In total this gets us \(\mathbb{E}[\abs{\mathcal{M}^+ \cap \mathcal{F}^+}] \geq 0.81 \mathbb{E}[\abs{\mathcal{M} \cap \mathcal{F}}] - 2\), which implies the lemma. ◻
To upper bound the total size of \(\mathcal{M} \cap \mathcal{F}\), we will need to make reference to those bases which are both in \(\mathcal{M}^+\) and in \(\mathcal{F}^+\). The argument upper bounding the length of the shadow path examines what happens when two such bases are adjacent. Since the inclusion criteria for neighboring bases are not independent, we require a small graph theoretical lemma that we simplify and adapt from [47].
Definition 7. Given a graph \((V,E)\), for \(v \in V\) we let \(\operatorname{deg}(v)\) denote the degree of \(v\). For subsets \(S_1, S_2 \subseteq V\) we define \(E[S_1,S_2] = \big\{\{u,v\} \in E : u \in S_1, v \in S_2\big\}\).
Lemma 14. Let \(P = (V,E)\) denote a path and \(V = R \cup B\) denote a partition of \(P\). Then \(\abs{V} \leq \abs{E[R,R]} + 2\abs{B} + 1\).
Proof. Since \(P\) is a path, a simple extension of the handshake lemma gives us \[2 \abs{R} \leq 2 + \sum_{v \in R}\operatorname{deg}(v) = 2 + \abs{E[R,B]} + 2\abs{E[R,R]}\] Then, since the maximum degree of a vertex in \(B\) is at most \(2\), we get that \(\abs{E[R,B]} \leq 2\abs{B}\). Combining the two inequalities, we find that \[\abs{R} \leq \frac{2\abs{E[R,R]} + \abs{E[R,B]}+ 2}{2} \leq \abs{E[R,R]} + \abs{B} + 1.\] Finally, the lemma follows from the above combined with \(\abs{V}=\abs{R}+\abs{B}\). ◻
Lemma 15. Let \(U\) be a fixed set and let \(R \subseteq V \subseteq U\) denote random sets satisfying \(\mathbb{E}[\abs{R}] \geq p \mathbb{E}[\abs{V}]\) for some \(p > 1/2\). Let \(P = (V,E)\) be an arbitrary path graph. Then we have \[\mathbb{E}[\abs{V}] \leq \frac{\mathbb{E}[\abs{E[R,R]}] + 1}{2p-1}.\]
Proof. Let \(B = V \setminus R\). From our assumption we know that \(\mathbb{E}[\abs{B}] \leq (1-p)\mathbb{E}[\abs{V}]\). By applying we learn \[\mathbb{E}[\abs{V}] \leq \mathbb{E}[\abs{E[R,R]}] + 2\mathbb{E}[\abs{B}] + 1 \leq \mathbb{E}[\abs{E[R,R]}] + 2(1-p)\mathbb{E}[\abs{V}]+1.\] Rearranging gives us our conclusion that \(\mathbb{E}[\abs{V}] \leq \frac{\mathbb{E}[\abs{E[R,R]}] + 1}{2p-1}\). ◻
We now have the correct concepts in place to upper bound the length of the shadow path. We can do this in two ways: either by using the objective value gap or by using the expected mean width of the feasible region. Both bounds will be used when proving .
Lemma 16. Let \(\rho > 0\), \(t_2 > t_1\geq 0\) and \(L>0\). Let \(A \in \mathbb{R}^{n \times d}\) be a matrix with rows of Euclidean norm at most \(1\), let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \mathbb{R}^d\), and let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\in \mathbb{R}^d\) be \(L\)-log-Lipschitz distributed. Note that throughout \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) is sampled independently of \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\). Let \(\mathcal{P} = (B^1,\dots,B^k)\) denote the shadow path from \(t_1 {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to \(t_2 {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) of \[\max\{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}:A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\}.\] Then we have \[\begin{align} \mathbb{E}[\abs{\mathcal{P}}] \leq 40 + \frac{2\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^k} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^1})]}{\rho\omega(\eta,n,d)} + \frac{2dL\rho(t_2-t_1)}{\ln(1/0.9)} \end{align}\]
Proof. In we characterized the shadow path as \(\mathcal{P} = \mathcal{M} \cap \mathcal{F}\). If \(\mathbb{E}[\abs{\mathcal{P}}] \leq 40\) then the result is trivial, so we may assume for the remainder of this proof that \(\mathbb{E}[\abs{\mathcal{P}}] > 40\). Then \(.05\mathbb{E}[|\mathcal{P}|] \geq 2\), so by , we have \[\mathbb{E}[\abs{\mathcal{M}^+ \cap \mathcal{F}^+}] \geq 0.8\mathbb{E}[\abs{\mathcal{P}}] - 2 \geq 0.75\mathbb{E}[\abs{\mathcal{P}}].\] Now define \(T = \{i \in [k-1] : \{B^i, B^{i+1}\} \subseteq \mathcal{M}^+ \cap \mathcal{F}^+\}.\)
We take the path graph \(V = \mathcal{P}\) and \(E = \{(B^i, B^{i+1}) : i \in [k-1]\}\). Taking \(R = \mathcal{F}^+ \cap \mathcal{M}^+\) makes \(\abs{E[R,R]} = \abs{T}\) and we can apply with \(p=0.75\) and find \[\mathbb{E}[\abs{\mathcal{P}}] \leq 2\mathbb{E}[\abs{T}] + 2.\]
The remainder of this proof is dedicated to showing that the inequality \[\begin{align} \abs{T} &\leq \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^k} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^1})}{\rho\omega} + \frac{dL\rho(t_2-t_1)}{\ln(0.9)} \label{eq:earlypath} \end{align}\tag{13}\] always holds. Consider an arbitrary \(i \in T\). Let \(p \in B^i\) denote the unique index for which \(p \notin B^{i+1}\). Write \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i = A_{B^i}^{-1} \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^i}\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} = A_{B^{i+1}}^{-1} \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^{i+1}}\). Finally, define \[\begin{align} t_{enter}^i = \min \{t \in [t_1,t_2] : (t{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\} \\ t_{exit}^i = \max \{t \in [t_1,t_2] : (t{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\}. \end{align}\]
Observe that \(t \mapsto \big((t{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1}\big)_p\) is linear, non-negative on \([t_{enter}^i, t_{exit}^i]\) and zero at \(t=t_{exit}^i\). It follows that \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^i}^{-1})_p \leq 0\). We make a case distinction on its value. If \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^i}^{-1})_p \leq -\rho\), then we consider the improvement in objective value along the edge and find that \[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i) = ({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^i}^{-1}) (A_{B^i}({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i)) \geq \rho\omega,\] because \(A_{B^i}({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i)\) is \(0\) on every coordinate except \(p\), where it is at most \(-\omega\). Thus in this case we have \[1 \leq \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i)}{\rho\omega}.\]
If \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^i}^{-1})_p > -\rho\), we consider the increase in the parameter \(t\) at the vertex. Because \(i \in \mathcal{M}^+\), there exists \(t_{middle}^i \in [t_{enter}^i,t_{exit}^i]\) for which we have \((t_{middle}^i {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1} \geq \frac{\ln(1/0.9)}{dL}\). Using the fact that \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^i}^{-1})_p > -\rho\), we find that for any \(\gamma > 0\) we have \[\Big(\big((t^i_{middle}+\gamma){\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\big)^\top A_{B^i}^{-1}\Big)_p > \frac{ \ln(1/0.9)}{dL} - \gamma \rho.\] For any \(t \in [t_{enter}^i, t_{middle}^i+\frac{\ln(1/0.9)}{dL\rho}]\) we have \(\big((t{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1}\big)_p > 0\). By construction we pivot out of \(B^i\) by relaxing constraint \(p\), meaning that the multiplier on \(p\) is the first to reach \(0\) as \(t\) increases. It follows that the entire vector of multipliers satisfies \((t{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})A_{B^i}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\). It follows that \(t_{exit}^i > t_{middle}^i + \frac{\ln(1/0.9)}{dL\rho}\) and so \(t_{exit}^i - t_{enter}^i > t_{exit}^i - t_{middle}^i > \frac{ \ln(1/0.9)}{dL\rho}\). We conclude for this case that \[1 < \frac{dL\rho(t_{exit}^i - t_{enter}^i)}{\ln(1/0.9)}.\] Thus, in either case we have \[1 \leq \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i)}{\rho\omega} + \frac{dL\rho(t_{exit}^i - t_{enter}^i)}{\ln(1/0.9)}\] We finish the proof of 13 by summing over all \(i\in T\) and telescoping the two terms: \[\begin{align} \abs{T} = \sum_{i \in T} 1 &\leq \sum_{i \in T}\frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i)}{\rho\omega} + \frac{dL\rho(t_{exit}^i - t_{enter}^i)}{\ln(1/0.9)}\\ &\leq \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{k} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1)}{\rho\omega} + \frac{dL\rho(t_2 - t_1)}{\ln(1/0.9)} \end{align}\] The first summand is a random variable, the second is not. The lemma’s statement for the case \(\mathbb{E}[\abs{\mathcal{P}}] \geq 40\) now follows as \(\mathbb{E}[\abs{\mathcal{P}}] \leq 2\mathbb{E}[\abs{T}]+2 \leq 2\frac{\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^k} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^1})]}{\rho\omega(\eta,n,d)} + 2\frac{dL\rho(t_2-t_1)}{\ln(1/0.9)} + 2\) ◻
Lemma 17. Let \(\rho > 0\), \(t_2 > t_1\geq 0\) and \(L>0\). Let \(A \in \mathbb{R}^{n \times d}\) be a matrix with rows of Euclidean norm at most \(1\), let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \mathbb{R}^d\), and let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\in \mathbb{R}^d\) be \(L\)-log-Lipschitz distributed. Let \(\mathcal{P} = (B^1,\dots,B^k)\) denote the shadow path from \(t_1 {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to \(t_2 {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) of \[\max\{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}:A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\}.\] Then we have that \[\mathbb{E}[\abs{\mathcal{P}}] \leq 40 + \frac{2\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^k})]}{\rho\omega(\eta,n,d)} + \frac{2dL\rho\ln(t_2/t_1)}{\ln(1/0.9)}.\]
Proof. In we characterized the shadow path as \(\mathcal{P} = \mathcal{M} \cap \mathcal{F}\). If \(\mathbb{E}[\abs{\mathcal{P}}] \leq 40\) then the result is trivial, so we may assume for the remainder of this proof that \(\mathbb{E}[\abs{\mathcal{P}}] > 40\). By , we have \[\mathbb{E}[\abs{\mathcal{M}^+ \cap \mathcal{F}^+}] \geq 0.8\mathbb{E}[\abs{\mathcal{M} \cap \mathcal{F}}] - 1 \geq 0.75\mathbb{E}[\abs{\mathcal{P}}].\] Now define \(T = \{i \in [k-1] : \{B^i, B^{i+1}\} \subseteq \mathcal{M}^+ \cap \mathcal{F}^+\}.\)
We take the path graph \(V = \mathcal{P}\) and \(E = \{(B^i, B^{i+1}) : i \in [k-1]\}\). Taking \(R = \mathcal{F}^+ \cap \mathcal{M}^+\) makes \(\abs{E[R,R]} = \abs{T}\) and we can apply with \(p=0.75\) and find \[\mathbb{E}[\abs{\mathcal{P}}] \leq 2\mathbb{E}[\abs{T}] + 2.\]
The biggest part of the remainder of this proof is dedicated to showing the deterministic statement \[\begin{align} \abs{T} &\leq \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^k} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^1})}{\rho\omega} + \frac{dL\rho\ln(t_2/t_1)}{\ln(0.9)}\label{eq:latepath} \end{align}\tag{14}\] holds. Consider an arbitrary \(i \in T\). Let \(p \in B^i\) denote the unique index for which \(p \notin B^{i+1}\). Write \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i = A_{B^i}^{-1} \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^i}\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} = A_{B^{i+1}}^{-1} \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^{i+1}}\). Finally, define \[\begin{align} t_{enter}^i = \min \{t \in [t_1,t_2] : (t{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\} \\ t_{exit}^i = \max \{t \in [t_1,t_2] : (t{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\}. \end{align}\]
Recall that, as before, \(t \mapsto \big((t{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1}\big)_p\) is linear, non-negative on \([t_{enter}^i, t_{exit}^i]\) and zero at \(t=t_{exit}^i\). It follows that \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top A_{B^i}^{-1})_p \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\). We make a case distinction on its value. In case of \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top A_{B^i}^{-1})_p \geq \rho\), we consider the decrease in auxiliary objective value along the edge and find that \[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i) = ({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top A_{B^i}^{-1}) (A_{B^i}({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i)) \leq -\rho\omega,\] because \(A_{B^i}({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i)\) is \(0\) on every coordinate except \(p\), where it is at most \(-\omega\). Thus in this case we have \[1 \leq \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1})}{\rho\omega}.\] In case of \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top A_{B^i}^{-1})_p \leq \rho\), we once again look at the parameter increase from \(t_{exit}^i\) to \(t_{enter}^i\). We aim to lower bound \(t_{exit}^i/ t_{enter}^i\). Since \(i \in \mathcal{M}^+\), there exists \(t \in [t_{enter}^i,t_{exit}^i]\) be such that \((t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1} \geq \frac{\ln(1/0.9)}{dL}\). Then, since \(\left({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^{\top} A_{B^{i}}^{-1}\right)_{p} \leq \rho\), by assumption, \(-\left({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^{\intercal} A_{B^{i}}^{-1}\right)_p \geq -\rho\) meaning that \[\begin{align} \left(\left(t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}- \frac{\ln(1/0.9)}{d L \rho} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\right)^{\top} A_{B^{i}}^{-1} \right)_{p} &= \left(\left(t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})\right)^{\top} A_{B^{i}}^{-1}\right)_{p} - \left(\frac{\ln(1/0.9)}{d L \rho} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^{\intercal} A_{B^{i}}^{-1}\right)_{p} \\ &\geq \frac{\ln(1/.9)}{dL} -\frac{\ln(1/.9)}{d L \rho} \rho \\ &= 0. \end{align}\] By construction, since \(p\) is the variable leaving the basis when pivoting from \(B^{i}\), \(p\) is the first coordinate to decrease to \(0\) when decrementing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\). In particular, \(t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}- \frac{\ln(1/.9)}{d L \rho} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) and \(t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) are both in the normal cone given by the non-negative row span of \(A_{B^{i}}\). That is \[\label{eq:movez} (t{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}- \frac{\ln(1/0.9)}{dL\rho}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}.\tag{15}\] By convexity of the normal cone, if \(1 - \frac{\ln(1/.9)}{d L \rho} \leq 0\) then \(t {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\) would also be in that cone, but by assumption, \(B^{i}\) is not the final basis of the path. Therefore, we must also have \(1 - \frac{\ln(1/.9)}{d L \rho} > 0\).
As a consequence, we may rescale by a factor \((1-\frac{\ln(1/0.9)}{dL\rho})^{-1}\) and observe that 15 is equivalent to the assertion that \((t'{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}})^\top A_{B^i}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\) for \(t' = \frac{t}{1-\frac{\ln(1/0.9)}{dL\rho}}\). This implies \[t_{enter}^i \leq t = t'(1-\frac{\ln(1/0.9)}{dL\rho}) \leq t_{exit}^i(1-\frac{\ln(1/0.9)}{dL\rho}),\] which we can rewrite to obtain \(t_{exit}^i/t_{enter}^i \geq \frac{1}{1-\frac{\ln(1/0.9)}{dL\rho}}\) and then, since \(\ln(1/(1-x)) \geq x\) for \(x \in [0,1)\), \[\ln(t_{exit}^i/t_{enter}^i) \geq \ln(\frac{1}{1-\frac{\ln(1/0.9)}{dL\rho}}) \geq \frac{\ln(1/0.9)}{dL\rho}.\]
This once again brings us to an upper bound for the unit \[1 \leq \frac{dL\rho\ln(t_{exit}^i/t_{enter}^i)}{\ln(1/0.9)}.\] As for the other inequality, we telescope the sums to find \[\begin{align} \abs{T} = \sum_{i \in T} 1 &\leq \sum_{i \in T} \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^i - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{i+1})}{\rho\omega} + \frac{dL\rho\ln(t_{exit}^i/t_{enter}^i)}{\ln(1/0.9)}\\ &\leq \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1 - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^k)}{\rho\omega} + \frac{dL\rho\ln(t_2/t_1)}{\ln(1/0.9)}. \end{align}\] This finishes the proof of 14 . The first summand is a random variable, the second is not. The lemma’s statement for the case \(\mathbb{E}[\abs{\mathcal{P}}] \geq 40\) now follows as \(\mathbb{E}[\abs{\mathcal{P}}] \leq 2\mathbb{E}[\abs{T}]+2 \leq 2\frac{\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^1} - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{B^k})]}{\rho\omega(\eta,n,d)} + 2\frac{dL\rho\ln(t_2/t_1)}{\ln(1/0.9)} + 2\). ◻
In order to state our main result, we need two measurements of the feasible set. The first is the mean width:
Definition 8. Let \(A \in \mathbb{R}^{n \times d}\), \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \mathbb{R}^d\) and let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Assume that \(P=\{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\in \mathbb{R}^d : A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\}\) is bounded. For random \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) distributed spherically symmetrically, let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1\) denote the vertex of \(P\) maximizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\), or \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1={\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\) when \(P=\emptyset\). We define the expected mean width* to be the quantity \[M = 2\mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}\bigg[\frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|}\bigg] = \frac{\mathbb{E}_{\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1]}{\mathbb{E}_{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}\big[\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|\big].}\] The number \(M\) will only be used when \(A\) and \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) are clear from context.*
Note that the mean width will require the random objective to be spherically symmetrically distributed. There are indeed distributions which are both spherically symmetric and \(L\)-log-Lipschitz:
Remark 5. The \(L\)-exponential distribution () is both spherically symmetric and \(L\)-log-Lipschitz.
The second measurement we require is for the width in the objective direction:
Lemma 18. Let \(A \in \mathbb{R}^{n \times d}\), \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \mathbb{R}^d\) and let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Let \(B^0\) be a basis for which \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^0}^{-1} \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\) and \(B^1\) be a basis for which \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\). Define \(N_{B^0,B^1} = {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top(A_{B^1}^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^1} - A_{B^0}^{-1} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^0}) + \gamma\eta(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^{1}}^{-1}\|_{1} + \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^{0}}^{-1}\|_1)\). Define \(N = N(A,{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}},\gamma,\eta) = \min_{B^0,B^1} N_{B^0,B^1}\), where the minimum is taken over all valid choices of \(B^0, B^1\).
Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,1}\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,0}\) denote optimal vertex solutions to respectively \[\begin{align} \operatorname{maximize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}& \text{and}&& \operatorname{minimize} \quad & {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\\ \operatorname{subject~to} \quad & A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}&&& \operatorname{subject~to} \quad & A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}} \end{align}\] Then \(\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,1}-{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,0})] \leq N\).
Proof. We prove the inequality for any valid choice of \(B^0, B^1\). For any realization of \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) we have \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,1}\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^0}^{-1}\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^0} \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,0}\) by weak duality. This yields \[\begin{align} \mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,1}] &\leq \mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^1}] \\ &= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}\mathbb{E}[\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^1}] \\ &= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^1} + \gamma\eta\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}) \\ &\leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^1} + \gamma\eta\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}\|_1, \end{align}\] where the last inequality follows because \(B^1\) is optimal and hence \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{0}}} {\boldsymbol{\textstyle\boldsymbol{0}}} {\boldsymbol{\scriptstyle\boldsymbol{0}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{0}}}}\). Similarly we have \(\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,0}] \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}_{B^1} - \gamma\eta\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}\|_1\). Taking the difference we find \(\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,1}] - \mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{*,0}] \leq N\) as needed. ◻
Theorem 6. Let \(\varepsilon> 0\) satisfy \(\varepsilon^{-1} \geq \frac{1}{1000 d^2 L \gamma \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|_2}\). Let \(A \in \mathbb{R}^{n \times d}\) be a matrix with rows of Euclidean norm at most \(1\), let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \mathbb{R}^d\), and let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\in \mathbb{R}^d\) be \(L\)-log-Lipschitz and spherically symmetrically distributed. If \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) are independent then the shadow path of \[\max\{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}:A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\}\] from a vertex maximizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to a vertex maximizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) has expected length at most \[101+ 9\sqrt{\frac{\mathbb{E}[\| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|] M dL}{\omega} \ln\left( \frac{dLN}{\omega \varepsilon} \right) },\] where \(\omega = \omega(\eta,n,d)\) was defined as \(\omega(\eta,n,d) = \frac{\eta}{1240 d \ln(n)}\), \(M\) is the expected mean width, and \(N\) is as defined in .
Proof. We separate the path from \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) into two subpaths. For \(0 < t' \leq \varepsilon^{-1}\) to be determined, the first path goes from \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to \(t'{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) and the second goes from \(t'{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to \(\varepsilon^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\). Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1\) be the vertex maximizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\), let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^2\) be the vertex maximizing \(t'{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\), let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^3\) be the vertex maximizing \(\varepsilon^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\), and finally let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{\ast}\) be the vertex on the shadow path maximizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\).
By applying with \(t_0=0, t_1=t'\) and choosing \(\rho = \frac{\ln(1/0.9)}{dLt'}\), we know that the first path has expected length at most \[\begin{align} 40 + 2\frac{\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^2 - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1)]\cdot dLt'}{\ln(1/0.9)\omega} + 2 &\leq 42 + 19\frac{\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^2 - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1)] \cdot dLt'}{\omega} \\ &\leq 42 + 19\frac{\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^* - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1)] \cdot dLt'}{\omega} \\ & \leq 42 + 19 \frac{NdLt'}{\omega}, \end{align}\] where the last inequality followed from . We choose \(t' = \frac{ \omega}{dLN}\) to make sure that the first sub-path has expected length at most \(42 + 19 = 61\).
Note that \(t' = \frac{ \omega}{dLN} = \frac{\eta}{1240 d^2LN\ln(n)} \leq \frac{\eta}{1000 d^2L \cdot \gamma\eta\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}\|_1}\), where \(B^1\) is the optimal feasible basis for maximizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\). The rows of \(A\) reach have 2-norm at most \(1\), which makes that \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top A_{B^1}^{-1}\|_1 \geq \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|_2\) by the triangle inequality. Hence \(t' \leq \frac{1}{1000d^2 L \gamma \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|_2} \leq \varepsilon^{-1}\) as promised.
Next, note that the shadow path from \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\) is monotonically decreasing in \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\). In particular, since \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{\ast}\) is at the end of the shadow path and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{1}\) is at the start, \[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^{\top} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{1} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^{\top} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{2} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^{\top} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{3} \geq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^{\top} {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^{\ast}.\] In particular this gives \(0 \leq \mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^2 - {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^3)] \leq \mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1-{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^\ast)]\). For the second sub-path, which goes from \(t'{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to \(\varepsilon^{-1}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\), we use and find it to have expected length at most \[40 + \frac{2\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^2 -{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^3)]}{\rho\omega} + \frac{2dL\rho\ln(1/(t'\varepsilon))}{\ln(1/0.9)} \leq 40 + 2\frac{\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1] - \mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*]}{\rho\omega} + \frac{2dL\rho\ln(1/(t'\varepsilon))}{\ln(1/0.9)}.\] Note that \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*\) is independent of \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\). Since \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) is distributed symmetrically around the origin that implies \(\mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^*] = 0\). The vertex \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1\) was defined as the maximizer of \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\), meaning that \(2 \mathbb{E}[{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}^1] = \mathbb{E}[\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|] \cdot M\) This means that the second sub-path has expected length at most \[40 + \frac{\mathbb{E}\big[\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|\big]\cdot M}{\rho\omega} + \frac{2dL\rho\ln\big(\frac{dLN}{\omega\varepsilon}\big)}{\ln(1/0.9)}.\] Now we choose \(\rho =\sqrt{\frac{2\mathbb{E}\big[\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|\big] \cdot M \ln(1/.9)}{\omega dL \ln\left(\frac{dLN}{\omega \varepsilon}\right)}}\) to minimize this sum. Then the resulting minimizer is \[40 + 2\sqrt{\frac{2\mathbb{E}[\| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|] M dL \ln\left(\frac{dLN}{\omega \varepsilon} \right)}{ \omega \ln(1/.9)} } \leq 40 + 9\sqrt{\frac{\mathbb{E}[\| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|] M dL \ln\left(\frac{dLN}{\omega \varepsilon} \right)}{ \omega } }.\] Adding this to the expected length \(61\) of the first subpath yields the desired result. ◻
is the main technical version of our shadow bound. We will derive two corollaries from it that will be easier to use.
Theorem 7. Let \(\varepsilon> 0\) satisfy \(\varepsilon^{-1} \geq \frac{1}{1000 d^3 \gamma \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|_2}\). Let \(A \in \mathbb{R}^{n \times d}\) be a matrix with rows of Euclidean norm at most \(1\), let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \mathbb{R}^d\), and let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Let \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^d\) be distributed uniformly at random, independent from \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\). If the LP \[\max\{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}:A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\}\] has expected mean width \(M\), then the shadow path from a vertex maximizing \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) to a vertex maximizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) has expected length at most \[102 + 316\sqrt{\frac{M d^3 \ln(n)}{\eta} \ln\left(\frac{6742 d^{3} N \ln^2(n)}{\eta \varepsilon} \right)},\] where \(N\) is as defined in .
Proof. Let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) be an \(L\)-exponentially distributed random vector, sampled in such a manner that \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}= \frac{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}}{\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|}\). Instead of measuring the path from \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) to \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\), we instead consider the path from \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\) to \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon \frac{1}{2edL^{-1}\ln(n)}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\). By Theorem 6, the expected number of steps on this latter path is at most \[101 + 9 \sqrt{\frac{\mathbb{E}[\| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|] M dL}{\omega} \ln\left( \frac{dLN (2 e d L^{-1} \ln(n))}{\omega \varepsilon} \right) } = 101 + 9d \sqrt{\frac{\mathbb{E}[\| {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|] M dL}{\omega} \ln\left(\frac{2ed^{2}N \ln(n)}{\omega \varepsilon} \right)},\] where we fill in \(\omega = \frac{\eta}{1240 d \ln(n)}\) as specified in . By , we have \(\mathbb{E}\big[\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\|\big] \cdot L = d\) which yields the complete bound of \[101 + 9\sqrt{\frac{1240 M d^3 \ln(n)}{\eta} \ln\left(\frac{2480ed^{3}N \ln^2(n)}{\eta \varepsilon} \right)} \leq 101 + 316\sqrt{\frac{M d^3 \ln(n)}{\eta} \ln\left(\frac{6742 d^{3}N\ln^2(n)}{\eta \varepsilon} \right)}.\]
Now consider the expected number of bases which are on the former path (\(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) to \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\)) but not on the latter path (\(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) to \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon\frac{1}{2edL^{-1}\ln(n)}{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\)). If there are any such bases, then there are at most \(\binom{n}{d} \leq n^d\) many. However, such bases exist only when \(\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\| \geq 2edL^{-1}\ln(n)\). By Lemma 11, we have \(\Pr[\|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\| \geq 2 e d L^{-1} \ln(n)] \leq n^{-d}\). Hence the expected number of bases on the former path but not on the latter is at most \(1\). This finishes the proof. ◻
Finally, we require a variant of which allows for two non-perturbed variable bounds. This will allow us to use the semi-random shadow vertex method on LP’s of the form \[\begin{align} \operatorname{maximize} \quad &{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\nonumber \\ \operatorname{subject~to} \quad & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\nonumber \\ &0 \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_d \leq U\label{eq:oneboundfix} \end{align}\tag{16}\] with \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}_d, -{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}_d\}\).
Theorem 8. Assume \(n > d \geq 2\). Let \(\varepsilon> 0\) satisfy \(\varepsilon^{-1} \geq \frac{1}{1000 d^3 \gamma \|{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\|_2}\). Let \(A \in \mathbb{R}^{n \times d}\) be a matrix with rows of Euclidean norm at most \(1\), let \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\in \{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}_d, -{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}_d\}\), \(U > 0\) and let \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\in \mathbb{R}^n\) be \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}},\eta,\gamma)\)-exponentially distributed. Let \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\in \mathbb{S}^{d-1}\) be distributed uniformly at random, independent from \(\hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\). Let \((-\hat{L}, \hat{U})\) be \(((0,U),\eta,\gamma)\)-exponentially distributed, again independent of what came before. If the feasible set of \[\begin{align} \operatorname{maximize} \quad &{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\nonumber \\ \operatorname{subject~to} \quad & A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\nonumber \\ &\hat{L} \leq {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_d \leq \hat{U} \label{eq:oneboundperturb} \end{align}\qquad{(2)}\] has expected mean width \(M\), then the shadow path from a vertex maximizing \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) to a vertex maximizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon \bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) on the feasible set 16 has expected length at most \[113 + 344d \sqrt{\frac{d \ln(n+2) M}{\eta} \ln\left(\frac{6742d^{3}(U + 2\eta\gamma)\ln^2 (n+2)}{\eta \varepsilon} \right)}.\]
Proof. Since \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}= {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}_d\) or \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}= -{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}_d\), the shadow path will leave or enter the faces of the feasible set with \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_d=0\) or \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_d=U\) at most once each. For all other pivot steps, the bases visited are also bases of the relaxed feasible set \(\{{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}: A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \hat{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{b}}} {\boldsymbol{\textstyle\boldsymbol{b}}} {\boldsymbol{\scriptstyle\boldsymbol{b}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{b}}}}\}\). Whenever \(\hat{0} \leq 0\) and \(U \leq \hat{U}\), these same bases are also feasible bases on the same shadow path on ?? . By this condition happens with probability at least \(1-\frac{2}{(n+d)^2}\). As such, the expected number of pivot steps on the shadow path from \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) to \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon \bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) for this new LP gives an upper bound on the length of the path we seek to bound. Using we find that we can expect at most \[102 + 316\sqrt{\frac{M d^3 \ln(n+2)}{\eta} \ln\left(\frac{6742 d^{3} N \ln^2(n+2)}{\eta \varepsilon} \right)}\] bases on the shadow path from \(\bar\mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) to \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}+ \varepsilon \bar \mathchoice{\boldsymbol{\displaystyle\boldsymbol{\theta}}} {\boldsymbol{\textstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptstyle\boldsymbol{\theta}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{\theta}}}\) on the feasible set of ?? . Resampling \(\hat{0}\) and \(\hat{U}\) until \(\hat{0} \leq 0\) and \(\hat{U} \leq U\) are satisfied requires an average of at most \(\frac{1}{1-\frac{2}{(n+d)^2}}\) samples, and Wald’s equation tells us that our final path has expected length no more than \[2+\frac{1}{1-\frac{2}{(n+d)^2}}\left(102 + 316\sqrt{\frac{M d^3 \ln(n+2)}{\eta} \ln\left(\frac{6742 d^{3} N \ln^2(n+2)}{\eta \varepsilon} \right)}\right).\] The extra \(2\) is from the pivots to and from the faces \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_d=0\) and \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_d=U\). We compute new constants using \(n > d \geq 2\).
The value of \(N\) can be computed exactly. Any feasible basis maximizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}_d\) has objective value \(U\) on the unperturbed problem and contains the inequality \(s \leq \hat{U}\), which will be the sole constraint to have a non-zero Lagrange multiplier and that multiplier is \(1\). Similarly for any feasible basis minimizing \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{e}}} {\boldsymbol{\textstyle\boldsymbol{e}}} {\boldsymbol{\scriptstyle\boldsymbol{e}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{e}}}}_d\) over the unperturbed problem having objective value \(0\) and a multiplier of \(1\) on the basic constraint \(-s \leq 0\). Thus \(N = U + 2\gamma\eta\). ◻
6 Discussion↩︎
In this section we evaluate how well the present analysis comports with further observation and how well our overall methodological structure ultimately comported with the general by-the-book analysis framework. Of course, in any mathematical analysis compromises are inevitable in order to accommodate the analyst’s ability to produce proofs. Since the impetus for performing a by-the-book analysis (and for the by-the-book analysis framework itself) is to produce theoretical results which have greater capacity to explain what is observed in practice, we propose that it is essential to a by-the-book analysis to evaluate how successful the analysis was in comporting with its observations. This is in part to provide a realistic assessment of the explanatory power of our results, but for this work, in which we propose the by-the-book analysis framework, it is also to differentiate the strengths and weaknesses of the by-the-book framework as a general algorithm analysis framework from the strengths and weaknesses of the particular by-the-book analysis we performed.
6.1 Input Assumptions↩︎
Our bound is written as polynomial in: \[d,n,\eta,M, \ln(N), \ln(\kappa), \mathrm{feasTol}, \log(\frac{1}{\mathrm{optTol}}).\] The standard input size measures for any linear program are \(d\) and \(n\), the number of variables and number of inequalities respectively. For weakly polynomial run-time bounds, \(\ln(\kappa)\) and \(\ln(N)\) are also standard to include as part of the measure of the input-size as they are bounded by the bit-complexity of the linear program. The remaining parameters \(\eta\), \(\text{optTol}\), and \(\text{feasTol}\) are all directly related to our modeling assumptions. We contend that, as supported by our research and computational experiments, it is reasonable to assume the chosen bounds on these numbers in a real-world computational setting. Likewise, our assumption that the rows of the input constraint matrix have Euclidean norm at most 1 is reasonable and unobtrusive. This is due to the fact that common LPs are hyper sparse, often containing only a constant number of non-zero elements in every inequality constraint, and each of these entries being bounded in magnitude. The mean width \(M\) was already discussed in detail in .
6.2 Model Assumptions↩︎
Our analysis has strong requirements on the pivot rule and ratio test, needing variants that are considered inefficient in practice and needing both to be performed exactly, without any loss of numerical precision.
We require that bounds are perturbed once and never shifted again. In particular, this requires a numerically exact implementation of Dantzig’s ratio test, ruling out the possibility of using Harris’ ratio test. Since any state of the art implementation of the simplex method benefits from Harris’ ratio test [1], this is a notable limitation of the current analysis.
We use a priori perturbations of the bounds and right-hand side. In our observations, we found that HiGHS does a priori perturbations not in the primal simplex method, but only to the cost vector in the dual simplex method. A more realistic model of the simplex method would allow for pivot steps which are slightly infeasible, and would allow to add and subtract perturbations throughout the running time.
When perturbing, our analysis requires a particular choice of probability density function for the individual perturbations. This is different from current leading implementations, which sample uniformly from an interval. We do not currently know how our probability distribution compares to the uniform distribution in practice. As such, we can not evaluate whether this is an unrealistic modeling choice or a possible improvement over the state-of-the-art.
For the pivot rule, we require that the semi-random shadow vertex path is followed. For our analysis, the randomness in the perturbations and pivot rule should furthermore be independent of each other and of the input data. This rules out a wide range of sophisticated pricing strategies, including steepest edge pivoting, partial pricing and multiple pricing (see [55] and [1]), as well as long-step pivoting rules used in Phase I of the simplex method [55]. In the dual, it rules out the bound-flipping ratio test.
The shadow vertex rule has been criticized for the difficulty of following it with appropriate pivot tolerances in a numerical setting [55]. As such, it does not see much use. The pivot rule used most commonly in practice is the steepest-edge rule (enhanced with partial pricing in the primal or bound-flipping ratio test in the dual). An analysis similar to the one presented here which uses steepest-edge would be highly desirable. Unfortunately, theoretical analysis is currently hamstrung by a broad lack of theoretical tools for analyzing the progress of steepest-edge outside of bounding step-by-step optimality gap reduction, which is not sufficient for proving strong upper bounds on the running time since any one pivot may reduce the optimality gap by an arbitrarily small amount. General improvements in our ability to theoretically analyze steepest-edge would open up the possibility of improved by-the-book analyses of simplex.
There are various other techniques in active use among high-quality simplex implementations that we did not bring into consideration for this paper. Most of these pertain to sparsity, numerical error, and cheap update formulas for weights or factorizations.
Automatic row/column rescaling, while often explained as being performed for the purpose of numerical stability, is likely to be an important component to include in future models. Rescaling affects the mean width of the feasible set, impacts the pivoting choices made by steepest-edge and semi-random shadow vertex rules, and influences the geometry of the feasible set when tolerances are accepted.
6.3 Methodology↩︎
Although we broadly succeeded in following the proposed methodology of by-the-book analysis, we did lightly deviate from this structure during the third phase. We originally planned to let an assumption of bounded geometric diameter act as a model and proxy for the input being well-scaled. It was only after beginning mathematical analysis in the third phase that we discovered the mean width may be a better parameter: it would yield stronger bounds, and moreover is much easier to measure. In one sense this is in conflict with the principle that our assumptions should be informed solely by a priori observations, and not retroactively informed by the proofs.
On the other hand, we believe that it is natural and even inevitable to discover over the course of the analysis that early choices of assumptions for modeling one’s observations can be improved by better ones. In this case, we did not so much introduce a new assumption as refine the parameter we were using to model an existing assumption (in this case, the assumption that our LP was well-scaled in some concrete way). Moreover, in the spirit of by-the-book analysis, we were only content to allow our bounds to depend on \(M\) provided we could verify that it is bounded on real instances. This is actually a strength of our shift from geometric diameter to mean width: the mean width is more easily measured (or rather, accurately estimated) than the geometric diameter. However, it remains to be seen whether the simplex method’s performance truly has a meaningful dependence on mean width; as we note in , this requires further investigation.
6.4 Outcomes↩︎
Although our results are a significant step forward in the study of the simplex method, they do not constitute a complete quantitative explanation of its real-world performance. We can evaluate the analysis and the theorem statement both quantitatively and qualitatively.
6.4.0.1 Scaling with problem size
Our bound contains a large constant factor dependence, much bigger than the scaling with problem size observed in practice. In the course of this work, only limited effort was expended to reduce this constant factor.
Our running time bound scales faster with the problem size \(d\) than the nearly linear growth observed in [2]–[7]. At the moment of writing it is unclear which losses happen due to inefficiencies in the proof, and which happen due to insufficient accounting of the geometry of the feasible set.
6.4.0.2 Size of perturbations.
Larger and more aggressive perturbations are known to improve performance [57]. A common technique in solvers, used when the simplex method stalls with repeated zero-length pivot steps, is to add increasingly large perturbations until stalling resolves. Although these perturbations do help the simplex method make progress, it comes at a cost: the perturbations must later be removed in order to return a solution that is feasible up to the desired tolerance.
The running time bounds in our theorems contain a one-over-square-root dependence on the feasibility tolerance. Interpreted naively, this would suggest that changing the feasibility tolerance by a factor \(100\) would change the running time by as much as a factor \(10\). To the best of our knowledge, no effect of such magnitude has been reported. The likely source of this disparity is that, in our analysis, step length is governed only by the perturbations: we did not use the geometry of the feasible set.
In order to improve the running time bounds, and to have a bound which does not scale as strongly with the tolerance, taking into account this geometry is necessary. Otherwise, a running time dependence of \(\sqrt{M/\eta}\) is tight, matching a lower bound construction up to polynomial factors in \(d\) and poly-logarithmic factors in \(M/\eta\).
This lower bound is obtained by taking \(n=(c/\eta)^{d/2}\) constraint vectors, for some absolute constant \(c > 1\), roughly equally spaced on the unit sphere. With a right-hand side equal to \(\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\), the feasible set will approximate a Euclidean ball. Any simplex method navigating from the minimizer to the maximizer of a given objective will require at least \(\Omega(\eta^{-1/2})\) pivot steps, provided that every solution visited satisfies the relaxed constraints \(A {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq (1+\eta)\mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\) and lies outside the interior of \(A{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\leq \mathchoice{\boldsymbol{\displaystyle\boldsymbol{1}}} {\boldsymbol{\textstyle\boldsymbol{1}}} {\boldsymbol{\scriptstyle\boldsymbol{1}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{1}}}\). This holds even for \(d=2\). For more on such constructions and analyses, see [34], [47], [86].
6.4.0.3 Length and curvature
In our mathematical analysis, we prove that vertices visited by the semi-random shadow vertex method tend to take up space in the primal, by having at least one long shadow edge incident, or take up space in the dual, by creating curvature of the shadow polygon. Furthermore, the analysis suggests that this effect is strongest when close to the maximizer of the random objective, and gets weaker clo Without randomizing the shadow plane with respect to the data, the theory would not predict this effect to happen.
This prediction aligns with findings from Fischer. In his Diplomarbeit [87], he used the shadow vertex method to draw projections of the Netlib LP problems. On the horizontal axis he plotted the value of the objective \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^T {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\). For the vertical axis he either used fixed objectives \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_0, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_1,\) or \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{a}}} {\boldsymbol{\textstyle\boldsymbol{a}}} {\boldsymbol{\scriptstyle\boldsymbol{a}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{a}}}}_i^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\), or random objectives \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}} {\boldsymbol{\textstyle\boldsymbol{x}}} {\boldsymbol{\scriptstyle\boldsymbol{x}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}\) with \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\in [-1,1]^d\) uniformly random. Four consecutive shadow paths were followed \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\to {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\to -{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\to -{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}} {\boldsymbol{\textstyle\boldsymbol{z}}} {\boldsymbol{\scriptstyle\boldsymbol{z}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}\to {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}} {\boldsymbol{\textstyle\boldsymbol{c}}} {\boldsymbol{\scriptstyle\boldsymbol{c}}} {\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}\) to find the complete shadow image. In cases where the shadow is unbounded, only the vertices are drawn. Upon request, we obtained the images from G. M. Ziegler.
Figure 4: Two-dimensional projections of scagr25, LP objective for horizontal axis [87].. a — \({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_0\) for vertical axis, b — Random objective for vertical axis
For most instances we received only one plot for a single auxiliary objective vector. However for the instance scagr25 Fischer drew two plots: one with a deterministic shadow \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}_0, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}})\) and one with a semi-random shadow \(({\mathchoice{\boldsymbol{\displaystyle\boldsymbol{z}}}
{\boldsymbol{\textstyle\boldsymbol{z}}}
{\boldsymbol{\scriptstyle\boldsymbol{z}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{z}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}}, {\mathchoice{\boldsymbol{\displaystyle\boldsymbol{c}}}
{\boldsymbol{\textstyle\boldsymbol{c}}}
{\boldsymbol{\scriptstyle\boldsymbol{c}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{c}}}}^\top{\mathchoice{\boldsymbol{\displaystyle\boldsymbol{x}}}
{\boldsymbol{\textstyle\boldsymbol{x}}}
{\boldsymbol{\scriptstyle\boldsymbol{x}}}
{\boldsymbol{\scriptscriptstyle\boldsymbol{x}}}})\). These can be seen in which depict the two projections. For the semi-random shadow in (b), we see a strong correspondence: when a given part of the boundary with constant length has more vertices
(and thus, shorter edges), then it is more curved (taking up dual space). For the deterministic shadow in (a), we do not see the same correspondence. At least on the qualitative level, Fischer’s plots agree with the prediction from our analysis.
7 Conclusion↩︎
In this paper we introduced the framework of by-the-book analysis and used it to study the performance of the simplex method. We have presented a running time analysis for a simplex method with bound perturbations, obtaining an upper bound on the expected running time of \[O\left(d^{1.5}\ln(n) \sqrt{\frac{ M}{\mathrm{feasTol}} \ln\left(\frac{d\kappa N\ln(n)}{\mathrm{feasTol}\cdot \mathrm{optTol}} \right)}\right)\] pivot steps required to solve an LP up to given feasibility and optimality tolerance. This result demonstrates how incorporating bound perturbations can provably ensure the simplex method requires only few pivot steps.
Although this work is predominantly dedicated to introducing and performing a by-the-book analysis of the simplex method, we note that this bound on the expected running time holds for arbitrary inputs and is mathematically sound even outside of the context of by-the-book analysis. We highlight the two most significant improvements over earlier work.
First, from a purely mathematical lens, our analysis works for sparse constraint data, in contrast to smoothed and average-case analysis. Since all practical LPs are sparse (see, e.g., [55], [1] or [49], [50]), this is an important feature. Second, due to the fact that we followed the by-the-book framework proposed herein, the assumptions we make are grounded in observations from simplex method implementations, LP modeling best practices, and measurements from practical benchmark instances. Our running time primarily depends on the LP being well scaled and the simplex method featuring bound perturbations. Both of these properties are essential to any large-scale LP solvers [1]. In this manner, the results of this by-the-book analysis correspond well with established knowledge on the simplex method as it is used in practice. This is an advantage of our bound, even when it is interpreted purely as a highly parameterized theoretical result; the parameters themselves were chosen not out of mathematical convenience but due to the fact that our research and experiments indicate that these parameters are bounded in practice, and may indeed play a genuine role in the behavior of the simplex method.
The exact role of scaling for linear programs has previously not been well understood, but is known to reduce the number of pivot steps [1] [88]. There is no broad agreement on how to define when an LP is well scaled [89]. We take our theorems and experiments to indicate that the mean width of the feasible set is bounded for “well-scaled” instances. A more in-depth investigation is in order to substantiate or refute that belief. Are the instances with small mean width indeed the ones that would be subjectively judged as well-scaled? Are instances with large mean width typically harder to solve for a simplex method? These are only some of the questions that need to be answered in order to confidently speak about whether small mean width is indeed a desirable property of linear programs.
Despite the remaining limitations highlighted in , we believe that the present work is a significant step forward towards a theoretical understanding of the simplex method’s real-world performance. Moreover, we expect our proposed framework of by-the-book analysis to be applicable to many more algorithms. Any algorithm could be the subject of a by-the-book analysis, provided that there is enough computational experience and/or high-quality open-source code available. A by-the-book analysis may be especially desirable in situations where existing theoretical results struggle to match the known practical efficiency of an algorithm or require unrealistic assumptions.
8 Acknowledgements↩︎
This work is supported by ANR JCJC grant ANR-24-CE48-2762. We thank François Lamothe for feedback on the initial manuscript and for helpful discussions about Phase 1 of the simplex algorithm. Our deepest gratitude goes out to the many developers who made time to explain us the inner workings of their simplex codes, including from COPT, FICO, Google, Gurobi, Hexaly, HiGHS, MOSEK, OMP, and SCIP, as well as Laurent Poirrier. Any flaws in our descriptions are ours alone.
9 Additional Proofs↩︎
The following lemma from [47] is reproduced to make the present document self-contained.
Lemma 19 (General condition-reversing interval lemma). Let \(L > 0\) be arbitrary. Suppose \(s \in \mathbb{R}\) is a continuous random variable whose probability density function \(f : \mathbb{R}\to \mathbb{R}\) satisfies the following limited log-Lipschitz property: For every \(x_1, x_2 \in [t-2/L,t+2/L]\) we have \(f(x_1)/f(x_2) \leq e^{L\abs{x_1-x_2}}\). Then for any \(\varepsilon\in [0,1/L]\) we have \[\Pr[s \geq t-\varepsilon\mid s \leq t] \leq 31 \varepsilon L \cdot \Pr[s \geq t].\]
Proof. We start by proving that \(\Pr\big[s \in [t-\varepsilon, t]\big] \leq e^2 \varepsilon L \cdot \Pr\big[s \in [t, t+1/L]\big]\). In the final paragraphs of this proof we will extend this statement into the desired conclusion.
Since \(\varepsilon\leq 1/L\) we are within the log-Lipschitzness range to bound our intended left-hand side as \[\Pr\big[s \in [t-\varepsilon, t]\big] = \int_{t-\varepsilon}^t f(x) \operatorname{d}x \leq \int_{t-\varepsilon}^t f(t) e^{L\abs{x-t}} \operatorname{d}x \leq e \varepsilon f(t).\] Similarly, we may use this log-Lipschitzness property to lower bound the probability in our intended right-hand side and find \[\begin{align} \Pr\big[ s \in [t, t + 1/L]\big] &= \int_t^{t + 1/L} f(x) \operatorname{d}x \\ &\geq \int_t^{t + 1/L} f(t) e^{-L\cdot\abs{x-t}} \operatorname{d}x \\ &\geq e^{-1} f(t) L^{-1}. \end{align}\] Putting these two inequalities together, we find \[\label{eq:numeratorbound} \Pr\big[s \in [t-\varepsilon, t]\big] \leq e^2 \varepsilon L \Pr\big[s \in [t,t+1/L] \big].\tag{17}\] This is the initial statement mentioned at the start of this proof. We now define the affine transformation \(T : \mathbb{R}\to \mathbb{R}\) to satisfy \(T(t-2/L) = t-2/L\) and \(T(t+1/L) = t\). For this transformation we observe that the Jacobian is \(\frac{2}{3}\), and for any point \(y \in [t-2/L, t+1/L]\) we have \(\abs{y-T(y)} \leq 1/L\). We use the transformation to give a change of variables and find \[\begin{align} \Pr[s \leq t] &= \int_{-\infty}^t f(x) \operatorname{d}x \nonumber\\ &= \int_{-\infty}^{t-2/L} f(x) \operatorname{d}x + \int_{t-2/L}^t f(x) \operatorname{d}x \nonumber\\ &= \int_{-\infty}^{t-2/L} f(x) \operatorname{d}x + \frac{2}{3}\int_{t-2/L}^{t+1/L} f( T(y) ) \operatorname{d}y \nonumber\\ &\geq \int_{-\infty}^{t-2/L} f(x) \operatorname{d}x + \frac{2}{3e}\int_{t-2/L}^{t+1/L} f(y) \operatorname{d}y \nonumber\\ &\geq \frac{2}{3e} \int_{-\infty}^{t+1/L} f(y) \operatorname{d}y \nonumber\\ &= \frac{2}{3e} \Pr[s \leq t+1/L]\nonumber\\ &\geq \frac{2}{3e}\Pr[s\leq t+1/L].\label{eq:denominatorbound} \end{align}\tag{18}\] Using the above two inequalities in order to bound the numerator and the denominator, we can now prove the lemma as follows \[\begin{align} \Pr[s \geq t-\varepsilon\mid s \leq t] &= \frac{\Pr\big[s \in [t-\varepsilon, t]\big]}{\Pr[s \leq t]} \\ &\leq \frac{3e\Pr\big[s \in [t-\varepsilon, t]\big]}{2\Pr[s \leq t+1/L]} } \\ &\leq \frac{3e^3 \varepsilon L \cdot \Pr\big[s \in [t, t+1/L]\big]}{2\Pr[s \leq t+1/L]} } \\ &\leq \frac{31 \varepsilon L \cdot \Pr\big[s \in [t, t+1/L]\big]}{\Pr[s \leq t+1/L]} \\ &= 31\varepsilon L \cdot \Pr[s \geq t \mid s \leq t + 1/L]. \end{align}\] In order to establish our final inequality, we use \(\Pr[s \geq t \mid s \leq t+1/L] \leq 1\) to find \[\begin{align} \Pr[s \geq t] &= \Pr[s > t+1/L] + \Pr[s \geq t \mid s \leq t+1/L] \Pr[s \leq t+1/L]\\ &\geq \Pr[s \geq t \mid s \leq t+1/L]\cdot\Big(\Pr[s > t+1/L] + \Pr[s \leq t+1/L]\Big)\\ &= \Pr[s \geq t \mid s \leq t+1/L]. \end{align}\] These two inequalities, then, prove the lemma as \[\Pr[s \geq t-\varepsilon\mid s\leq t] \leq 31\varepsilon L\cdot \Pr[s \geq t \mid s \leq t + 1/L] \leq 31\varepsilon\Pr[s \geq t].\] ◻
References↩︎
https://github.com/google/or-tools/blob/9b770/ortools/glop/parameters.proto#L481↩︎
https://github.com/google/or-tools/blob/9b770/ortools/glop/parameters.proto#L183↩︎
https://github.com/google/or-tools/blob/9b770/ortools/glop/parameters.proto#L251↩︎
https://github.com/google/or-tools/blob/9b770/ortools/glop/reduced_costs.cc#L255↩︎
https://github.com/google/or-tools/blob/9b770/ortools/glop/parameters.proto#L415↩︎
https://ergo-code.github.io/HiGHS/dev/options/definitions/#option-simplex-scale-strategy↩︎
https://github.com/ERGO-Code/HiGHS/blob/c67f06df422ce1faff4089033513874b8/highs/lp_data/HighsLpUtils.cpp#L861↩︎
https://ergo-code.github.io/HiGHS/dev/options/definitions/#option-small-matrix-value↩︎
https://github.com/ERGO-Code/HiGHS/blob/c67f06df422ce1faff4089033513874b8/highs/simplex/HEkkPrimal.cpp#L2793↩︎
https://github.com/ERGO-Code/HiGHS/blob/c67f06df422ce1faff4089033513874b8/highs/simplex/HEkkDual.cpp#L126↩︎
See also https://github.com/ERGO-Code/HiGHS/blob/c67f06df422ce1faff4089/highs/simplex/HEkk.cpp#L2488↩︎
https://docs.mosek.com/latest/pythonapi/presolver.html#index-5↩︎
https://docs.mosek.com/latest/pythonapi/parameters.html#mosek.dparam.data_tol_aij_large↩︎
https://docs.mosek.com/latest/pythonapi/parameters.html#mosek.dparam.data_tol_x↩︎
https://docs.mosek.com/latest/pythonapi/parameters.html#mosek.dparam.basis_tol_x↩︎