1 Introduction↩︎

Stochastic convex optimization (SCO) is a central problem in machine learning and statistics, where for a sample space \(\mathbb{S}\), parameter space \(\mc{X}\subset\R^d\), and a collection of convex losses \(\crl{F(\cdot;s): s\in\mathbb{S}}\), one wishes to solve \[\label{eq:sco} \minimize_{x\in\mc{X}} f(x) \defeq \E_{S\sim P}\brk*{F(x;S)} = \int_{\mathbb{S}}F(x;s)\mathrm{d}P(s)\tag{1}\] using an observed dataset \(\mc{S} = S_1^n \simiid P\). While as formulated, the problem is by now fairly well-understood [3]–[7], it is becoming clear that, because of considerations beyond pure statistical accuracy—memory or communication costs [8]–[10], fairness [11], [12], personalization or distributed learning [13]—problem 1 is simply insufficient to address modern learning problems. To that end, researchers have revisited SCO under the additional constraint that the solution preserves the privacy of the provided sample [14]–[18]. A waypoint is [19], who provide a private method with optimal convergence rates for the related empirical risk minimization problem, with recent papers focus on SCO providing (worst-case) optimal rates in various settings: smooth convex functions [2], [20], non-smooth functions [21], non-Euclidean geometry [22], [23] and under more stringent privacy constraints [24].

Yet these works ground their analyses in worst-case scenarios and provide guarantees for the hardest instance of the class of problems they consider. Conversely, they argue that their algorithms are optimal in a minimax sense: for any algorithm, there exists a hard instance on which the error achieved by the algorithm is equal to the upper bound. While valuable, these results are pessimistic—the exhibited hard instances are typically pathological—and fail to reflect achievable performance.

In this work, we consider the problem of adaptivity when solving 1 under privacy constraints. Importantly, we wish to provide private algorithms that adapt to the hardness of the objective \(f\). A loss function \(f\) may belong to multiple problem classes, each exhibiting different achievable rates, so a natural desideratum is to attain the error rate of the easiest sub-class. As a simple vignette, if one gets an arbitrary \(1\)-Lipschitz convex loss function \(f\), the worst-case guarantee of any \(\varepsilon\)-DP algorithm is \(\Theta(1/\sqrt{n} + d/(n\varepsilon))\). However, if one learns that \(f\) exhibits some growth property—say \(f\) is \(1\)-strongly convex—the regret guarantee improves to the faster \(\Theta(1/n + (d / (n\varepsilon))^2)\) rate with the appropriate algorithm. It is thus important to provide algorithms that achieves the rates of the “easiest” class to which the function belongs [25]–[27].

To that end, consider the nested classes of functions \(\mc{F}^\kappa\) for \(\kappa \in [1, \infty]\) such that, if \(f\in \mc{F}^\kappa\) then there exists \(\lambda > 0\) such that for all \(x \in \mc{X}\), \[f(x) - \inf_{x'\in\mc{X}} f(x') \ge \frac{\lambda}{\kappa}\norm{x-x^\star}_2^\kappa.\] For example, strong convexity implies growth with parameter \(\kappa = 2\). This growth assumption closely relates to uniform convexity [25] and the Polyak-Kurdyka-Łojasiewicz inequality [28], and we make these connections precise in Section 2. Intuitively, smaller \(\kappa\) makes the function much easier to optimize: the error around the optimal point grows quickly. Objectives with growth are widespread in machine learning applications: among others, the \(\ell_1\)-regularized hinge loss exhibits sharp growth (i.e.\(\kappa = 1\)) while \(\ell_1\)- or \(\ell_\infty\)-constrained \(\kappa\)-norm regression —i.e.\(s = (a, b) \in \R^d\times\R\) and \(F(x;s) = \abs{b - \tri{a, x}}^\kappa\)—has \(\kappa\)-growth for any \(\kappa\) integer greater than \(2\) [29]. In this work, we provide private adaptive algorithms that adapt to the actual growth of the function at hand.

We begin our analysis by examining Asi and Duchi’s inverse sensitivity mechanism [1] on ERM as a motivation. While not a practical algorithm, it achieves instance-optimal rates for any one-dimensional function under mild assumptions, quantifying the best bound one could hope to achieve with an adaptive algorithm, and showing (in principle) that adaptive private algorithms can exist. We first show that for any function with \(\kappa\)-growth, the inverse sensitivity mechanism achieves privacy cost \((d/(n\varepsilon))^{\kappa / (\kappa - 1)}\); importantly, without knowledge of the function class \(\mc{F}^\kappa\), that \(f\) belongs to. This constitutes grounding and motivation for our work in three ways: (i) it validates our choice of sub-classes \(\mc{F}^\kappa\) as the privacy rate is effectively controlled by the value of \(\kappa\), (ii) it exhibits the rate we wish to achieve with efficient algorithms on \(\mc{F}^\kappa\) and (iii) it showcases that for easier functions, privacy costs shrink significantly—to illustrate, for \(\kappa = 5/4\) the privacy rate becomes \((d/(n\varepsilon))^5\).

We continue our treatment of problem 1 under growth in Section 4 and develop practical algorithms that achieve the rates of the inverse sensitivity mechanism. Moreover, for approximate \((\varepsilon,\delta)\)-differential privacy, our algorithms improve the rates, achieving roughly \((\sqrt{d}/(n\varepsilon))^{\kappa / (\kappa - 1)}\). Our algorithms hinge on a reduction to SCO: we show that by solving a sequence of increasingly constrained SCO problems, one achieves the right rate whenever the function exhibits growth at the optimum. Importantly, our algorithm only requires a lower bound \(\underline{\kappa} \le \kappa\) (where \(\kappa\) is the actual growth of \(f\)).

We provide optimality guarantees for our algorithms in Section 5 and show that both the inverse sensitivity and the efficient algorithms of Section 4 are simultaneously minimax optimal over all classes \(\mc{F}^\kappa\) whenever \(\kappa = 1+\Theta(1)\) and \(d=1\) for \(\varepsilon\)-DP algorithms. Finally, we prove that in arbitrary dimension, for both pure- and approximate-DP constraints, our algorithms are also simultaneously optimal for all classes \(\mc{F}^\kappa\) with \(\kappa \ge 2\).

On the way, we provide results that may be of independent interest to the community. First, we develop optimal algorithms for SCO under pure differential privacy constraints, which, to the best of our knowledge, do not exist in the literature. Secondly, our algorithms and analysis provide high-probability bounds on the loss, whereas existing results only provide (weaker) bounds on the expected loss. Finally, we complete the results of [30] on (non-private) optimization lower bounds for functions with \(\kappa\)-growth by providing information-theoretic lower bounds (in contrast to oracle-based lower bounds that rely on observing only gradient information) and capturing the optimal dependence on all problem parameters (namely \(d, L\) and \(\lambda\)).

1.1 Related work↩︎

Convex optimization is one of the best studied problems in private data analysis [17]–[19], [31]. The first papers in this line of work mainly study minimizing the empirical loss, and readily establish that the (minimax) optimal privacy rates are \({d}/{n \varepsilon}\) for pure \(\varepsilon\)-DP and \({\sqrt{d \log(1/\delta)}}/{n \varepsilon}\) for \((\varepsilon,\delta)\)-DP [17], [19]. More recently, several works instead consider the harder problem of privately minimizing the population loss [2], [20]. These papers introduce new algorithmic techniques to obtain the worst-case optimal rates of \(1/\sqrt{n}+ {\sqrt{d \log(1/\delta)}}/{n \varepsilon}\) for \((\varepsilon,\delta)\)-DP. They also show how to improve this rate to the faster \(1/n + {{d \log(1/\delta)}}/{(n \varepsilon)^2}\) in the case of \(1\)-strongly convex functions. Our work subsumes both of these results as they correspond to \(\kappa = \infty\) and \(\kappa = 2\) respectively. To the best of our knowledge, there has been no work in private optimization that investigates the rates under general \(\kappa\)-growth assumptions or adaptivity to such conditions.

In contrast, the optimization community has extensively studied growth assumptions  [25], [30], [32] and show that on these problems, carefully crafted algorithms improves upon the standard \(1/\sqrt{n}\) for convex functions to the faster \((1/\sqrt{n})^{\kappa/(\kappa - 1)}\). [25] derives worst-case optimal (in the first-order oracle model) gradient algorithms in the uniformly convex case (i.e. \(\kappa \ge 2\)) and provides technique to adapt to the growth \(\kappa\), while [30], drawing connections between growth conditions and active learning, provides upper and lower bounds in the first-order stochastic oracle model. We complete the results of the latter and provide information-theoretic lower bounds that have optimal dependence on \(d, \lambda\) and \(n\)—their lower bound only holding for \(\lambda\) inversely proportional to \(d^{1/2-1/\kappa}\), when \(\kappa \ge 2\). Closest to our work is [32] who studies instance-optimality via local minimax complexity [33]. For one-dimensional functions, they develop a bisection-based instance-optimal algorithm and show that on individual functions of the form \(t \mapsto \kappa^{-1}\abs{t}^\kappa\), the local minimax rate is \((1/\sqrt{n})^{\kappa / (\kappa - 1)}\).

2 Preliminaries↩︎

We first provide notation that we use throughout this paper, define useful assumptions and present key definitions in convex analysis and differential privacy.

2.0.0.1 Notation.

\(n\) typically denotes the sample size and \(d\) the dimension. Throughout this work, \(x\) refers to the optimization variable, \(\mc{X} \subset \R^d\) to the constraint set and \(s\) to elements (\(S\) when random) of the sample space \(\mathbb{S}\). We usually denote by \(F:\mc{X}\times\mathbb{S}\to\R\) the (convex) loss function and for a dataset \(\mc{S} = (s_1, \ldots, s_n) \subset \mathbb{S}\), we define the empirical and population losses \[\femp(x) \defeq \frac{1}{n}\sum_{i\le n}F(x;s_i) ~~and~~ f(x) \defeq \E_{S\sim P}\brk*{F(x;S)}.\] We omit the dependence on \(P\) as it is often clear from context. We reserve \(\varepsilon, \delta \ge 0\) for the privacy parameters of Definition 1. We always take gradients with respect to the optimization variable \(x\). In the case that \(F(\cdot;s)\) is not differentiable at \(x\), we override notation and define \(\nabla F(x;s) = \argmin_{g\in \partial F(x;s)}\norm{g}_2\), where \(\partial F(x;s)\) is the subdifferential of \(F(\cdot;s)\) at \(x\). We use \(\msf{A}\) for (potentially random) mechanism and \(S_1^n\) as a shorthand for \(\prn*{S_1, \ldots, S_n}\). For \(p \ge 1\), \(\norm{\cdot}_p\) is the standard \(\ell_p\)-norm, \(\ball_p^d(R)\) is the corresponding \(d\)-dimensional \(p\)-ball of radius \(R\) and \(p^\star\) is the dual of \(p\), i.e. such that \(1/p^\star + 1/p = 1\). Finally, we define the Hamming distance between datasets \(\dham(\mc{S}, \mc{S}') \defeq \inf_{\sigma \in \mathfrak{S}_n} \ones\crl{s_i \neq s'_{\sigma(i)}}\), where \(\mathfrak{S}_n\) is the set of permutations over sets of size \(n\).

2.0.0.2 Assumptions.

We first state standard assumptions for solving 1 . We assume that \(\mc{X}\) is a closed, convex domain such that \(\mathsf{diam}_2(\mc{X}) = \sup_{x, y \in \mc{X}}\norm{x-y}_2 \le D< \infty\). Furthermore, we assume that for any \(s\in\mathbb{S}\), \(F(\cdot;s)\) is convex and \(L\)-Lipschitz with respect to \(\norm{\cdot}_2\). Central to our work, we define the following \(\kappa\)-growth assumption.

2.0.0.3 Uniform convexity and inequality.

Assumption 1 is closely related to two fundamental notions in convex analysis: uniform convexity and the inequality. Following [34], we say that \(h:\mc{Z}\subset\R^d \to \R\) is \((\sigma, \kappa)\)-uniformly convex with \(\sigma > 0\) and \(\kappa \ge 2\) if \[for all~~x,y \in\mc{Z}, \quad h(y) \ge h(x) + \tri{\nabla h(x), y-x} + \frac{\sigma}{\kappa}\norm{x-y}_2^\kappa.\] This immediately implies that (i) sums (and expectations) preserve uniform convexity (ii) if \(f\) is uniformly convex with \(\lambda\) and \(\kappa\), then it has \((\lambda, \kappa)\)-growth. This will be useful when constructing hard instances as it will suffice to consider \((\lambda, \kappa)\)-uniformly convex functions which are generally more convenient to manipulate. Finally, we point out that, in the general case that \(\kappa \ge 1\), the literature refers to Assumption 1 as the inequality [28] with, in their notation, \(\varphi(s) = (\kappa / \lambda)^{1/\kappa}s^{1/\kappa}\). Theorem 5-(ii) in [28] says that, under mild conditions, Assumption 1 implies the following inequality between the error and the gradient norm for all \(x \in \mc{X}\) \[\label{eq:kl} f(x) - \inf_{x'\in\mc{X}} f(x') \le \frac{e}{\lambdae}\nrm*{\nabla f(x)}_2^{\tfrac{\kappa}{\kappa - 1}},\tag{2}\] This is a key result in our analysis of the inverse sensitivity mechanism of Section 3.

2.0.0.4 Differential privacy.

We begin by recalling the definition of \((\varepsilon,\delta)\)-differential privacy.

We use the following standard results in differential privacy.

Next, we consider the Laplace mechanism. We will let \(Z \sim \laplace_d(\sigma)\) denote a \(d\)-dimensional vector \(Z \in \R^d\) such that \(Z_i \simiid \laplace(\sigma)\) for \(1 \le i \le d\).

Finally, we need the Gaussian mechanism for \((\varepsilon,\delta)\)-DP.

2.0.0.5 Inverse sensitivity mechanism.

Our goal is to design private optimization algorithms that adapt to the difficulty of the underlying function. As a reference point, we turn to the inverse sensitivity mechanism of [1] as it enjoys general instance-optimality guarantees. For a given function \(h: \mathbb{S}^n \to \mathcal{T} \subset \R^d\) that we wish to estimate privately, define the inverse sensitivity at \(x\in\mc{T}\) \[\label{eq:inv-sens} \invmodcont_h(\mathcal{S};x) = \inf_{\mathcal{S}'} \{ \dham(\mathcal{S}',\mathcal{S}) : h(\mathcal{S}')=x \},\tag{3}\] that is, the inverse sensitivity of a target parameter \(y \in \mathcal{T}\) at instance \(\mathcal{S}\) is the minimal number of samples one needs to change to reach a new instance \(\mathcal{S}'\) such that \(h(\mathcal{S}') = y\). Having this quantity, the inverse sensitivity mechanism samples an output from the following probability density \[\label{pvmxcofu} \pdf_{\Ainvsm(\mathcal{S})} (x) \propto e^{- \varepsilon\invmodcont_h(\mathcal{S};x) }.\tag{4}\] The inverse sensitivity mechanism preserves \(\varepsilon\)-DP and enjoys instance-optimality guarantees in general settings [1]. In contrast to (worst-case) minimax optimality guarantees which measure the performance of the algorithm on the hardest instance, these notions of instance-optimality provide stronger per-instance optimality guarantees.

3 Adaptive rates through inverse sensitivity for \(\varepsilon\)-DP↩︎

To understand the achievable rates when privately optimizing functions with growth, we begin our theoretical investigation by examining the inverse sensitivity mechanism in our setting. We show that, for instances that exhibit \(\kappa\)-growth of the empirical function, the inverse sensitivity mechanism privately solves ERM with excess loss roughly \(({d}/{n \varepsilon} )^{\frac{\kappa}{\kappa-1}}\).

In our setting, we use a gradient-based approximation of the inverse sensitivity mechanism to simplify the analysis, while attaining similar rates. Following [36] with our function of interest \(h(\mathcal{S}) \defeq \argmin_{x \in \mc{X}} f_\mathcal{S}(x)\), we can lower bound the inverse sensitivity \(\invmodcont_h(\mathcal{S};x) \ge n \ltwo{\nabla f_\mathcal{S}(x)}/2 L\) under natural assumptions. We define a \(\rho\)-smoothed version of this quantity which is more suitable to continuous domains \[G^\rho_\mathcal{S}(x) = \inf_{y \in \mc{X}: \ltwo{y-x} \le \rho} \ltwo{\nabla f_\mathcal{S}(y)},\] and define the \(\rho\)-smooth gradient-based inverse sensitivity mechanism \[\label{eq:grad-inv-sens} \pdf_{\Agrinvsm(\mathcal{S})} (x) \propto e^{- \varepsilon n G^\rho_\mathcal{S}(x)/2 L}.\tag{5}\]

Note that while exactly sampling from the un-normalized density \(\pdf_{\Agrinvsm\prn{\mathcal{S}}}\) is computationally intractable, analyzing its performance is an important step towards understanding the optimal rates for the family of functions with growth that we study in this work. The following theorem demonstrates the adaptivity of the inverse sensitivity mechanism to the growth of the underlying instance. We defer the proof to 6.

The rates of the inverse sensitivity in [thm:inv-sens] provide two main insights regarding the landscape of the problem with growth conditions. First, these conditions allow to improve the worst-case rate \(d/n\varepsilon\) to \(({d}/{n \varepsilon} )^{\frac{\kappa}{\kappa-1}}\) for pure \(\varepsilon\)-DP and therefore suggest a better rate \(({\sqrt{d \log(1/\delta)}}/{n \varepsilon} )^{\frac{\kappa}{\kappa-1}}\) is possible for approximate \((\varepsilon,\delta)\)-DP. Moreover, the general instance-optimality guarantees of this mechanism [1] hint that these are the optimal rates for our class of functions. In the sections to come, we validate the correctness of these predictions by developing efficient algorithms that achieve these rates (for pure and approximate privacy), and prove matching lower bounds which demonstrate the optimality of these algorithms.

4 Efficient algorithms with optimal rates↩︎

While the previous section demonstrates that there exists algorithms that improve the rates for functions with growth, we pointed out that \(\Agrinvsm\) was computationally intractable in the general case. In this section, we develop efficient algorithms—e.g.that are implementable with gradient-based methods—that achieve the same convergence rates. Our algorithms build on the recent localization techniques that [2] used to obtain optimal rates for DP-SCO with general convex functions. In Section 4.1, we use these techniques to develop private algorithms that achieve the optimal rates for (pure) DP-SCO with high probability, in contrast to existing results which bound the expected excess loss. These results are of independent interest.

In Section 4.2, we translate these results into convergence guarantees on privately optimizing convex functions with growth by solving a sequence of increasingly constrained SCO problems—the high-probability guarantees of 4.1 being crucial to our convergence analysis of these algorithms.

4.1 High-probability guarantees for convex DP-SCO↩︎

We first describe our algorithm (Algorithm 1) then analyze its performance under pure-DP (Proposition [thm:sco-hb-pure]) and approximate-DP constraints (Proposition [thm:sco-hb-appr]). Our analysis builds on novel tight generalization bounds for uniformly-stable algorithms with high probability [37]. We defer the proofs to 7.

Similarly, by using a different choice for the parameters and noise distribution, we have the following guarantees for approximate \((\varepsilon,\delta)\)-DP.

4.2 Algorithms for DP-SCO with growth↩︎

Building on the algorithms of the previous section, we design algorithms that recover the rates of the inverse sensitivity mechanism for functions with growth, importantly without knowledge of the value of \(\kappa\). Inspired by epoch-based algorithms from the optimization literature [5], [38], our algorithm iteratively applies the private procedures from the previous section. Crucially, the growth assumption allows to reduce the diameter of the domain after each run, hence improving the overall excess loss by carefully choosing the hyper-parameters. We provide full details in 2.

The following theorem summarizes our main upper bound for DP-SCO with growth in the pure privacy model, recovering the rates of the inverse sensitivity mechanism in 3. We defer the proof to 7.3.

Moreover, we can improve the dependence on the dimension for approximate \((\varepsilon,\delta)\)-DP, resulting in the following bounds.

5 Lower bounds↩︎

In this section, we develop (minimax) lower bounds for the problem of SCO with \(\kappa\)-growth under privacy constraints. Note that taking \(\varepsilon\to \infty\) provides lower bound for the unconstrained minimax risk. For a sample space \(\mathbb{S}\) and collection of distributions \(\mc{P}\) over \(\mathbb{S}\), we define the function class \(\mc{F}^\kappa(\mc{P})\) as the set of convex functions from \(\R^d\to\R\) that are \(L\)-Lipschitz and has \(\kappa\)-growth (Assumption 1). We define the constrained minimax risk [39] \[\mathfrak{M}_n(\mc{X}, \mc{P}, \mc{F}^\kappa, \varepsilon, \delta) \defeq \inf_{\what{x}_n\in\mc{A}^{\varepsilon, \delta}} \sup_{(F, P) \in \mc{F}^\kappa\times\mc{P}} \E\brk*{ f(\what{x}_n(S_1^n)) - \inf_{x'\in\mc{X}}f(x') },\] where \(\mc{A}^{\epsilon, \delta}\) is the collection of \((\varepsilon, \delta)\)-DP mechanisms from \(\mathbb{S}^n\) to \(\mc{X}\). When clear from context, we omit the dependency on \(\mc{P}\) of the function class and simply write \(\mc{F}^\kappa\). We also forego the dependence on \(\delta\) when referring to pure-DP constraints, i.e. \(\mathfrak{M}_n(\mc{X}, \mc{P}, \mc{F}^\kappa, \varepsilon, \delta=0) \eqdef \mathfrak{M}_n(\mc{X}, \mc{P}, \mc{F}^\kappa, \varepsilon)\). We now proceed to prove tight lower bounds for \(\varepsilon\)-DP in 5.1 and \((\varepsilon,\delta)\)-DP in 5.2.

5.1 Lower bounds for pure \(\varepsilon\)-DP↩︎

Although in 4 we show that the same algorithm achieves the optimal upper bounds for all values of \(\kappa >1\), the landscape of the problem is more subtle for the lower bounds and we need to delineate two different cases to obtain tight lower bounds. We begin with \(\kappa \ge 2\), which corresponds to uniform convexity and enjoys properties that make the problem easier (e.g., closure under summation or addition of linear terms). The second case, \(1 < \kappa <2\), corresponds to sharper growth and requires a different hard instance to satisfy the growth condition.

5.1.0.1 \(\kappa\)-growth with \(\kappa \ge 2\).

We begin by developing lower bounds under pure DP for \(\kappa \ge 2\)

First of all, note that \(L \ge \lambda 2^\kappa R^{\kappa - 1}\) is not an overly-restrictive assumption. Indeed, for an arbitrary \((\lambda, \kappa)\)-uniformly convex and \(L\)-Lipschitz function, it always holds that \(L \ge \tfrac{\lambda}{2}R^{\kappa - 1}\). This is thus equivalent to assuming \(\kappa = \Theta(1)\). Note that when \(\kappa \gg 1\), the standard \(n^{-1/2} + d/(n\varepsilon)\) lower bound holds. We present the proof in 8.1.1 and preview the main ideas here.

5.1.0.2 \(\kappa\)-growth with \(\kappa \in (1, 2]\).

As the construction of the hard instance is more intricate for \(\kappa < 2\), we provide a one-dimensional lower bound and leave the high-dimensional case for future work. In this case we directly obtain the result with a private version of Le Cam’s method [39]–[41], however with a different family of functions.

The issue with the construction of the previous section is that the function does not exhibit sharp growth for \(\kappa < 2\). Indeed, the added linear function shifts the minimum away from \(0\) where the function is differentiable and as a result it locally behaves as a quadratic and only achieves growth \(\kappa = 2\). To establish the lower bound, we consider a different sample function \(F\) that has growth exactly \(1\) on one side and \(\kappa\) on the other side. This yields the following

5.2 Lower bounds under approximate privacy constraints↩︎

We conclude our treatment by providing lower bounds but now under approximate privacy constraints, demonstrating the optimality of the risk bound of Theorem [thm:sco-growth-appr]. We prove the result via a reduction: we show that if one solves ERM with \(\kappa\)-growth with error \(\Delta\), this implies that one solves arbitrary convex ERM with error \(\phi(\Delta)\). Given that a lower bound of \(\Omega\prn{\sqrt{d}/(n\varepsilon)}\) holds for ERM, a lower bound of \(\phi^{-1}\prn{\sqrt{d}/(n\varepsilon)}\) holds for ERM with \(\kappa\)-growth. However, for this reduction to hold, we require that \(\kappa \ge 2\). Furthermore, we consider \(\kappa\) to be roughly a constant—in the case that \(\kappa\) is too large, standard lower bounds on general convex functions apply.

1 implies that the same lower bound (up to logarithmic factors) applies to SCO via the reduction of [20]. Before proving the theorem, let us state (and prove in 8.2) the following reduction: if an \((\varepsilon,\delta)\)-DP algorithm achieves excess error (roughly) \(\Delta\) on ERM for any function with \(\kappa\)-growth, there exists an \((\varepsilon,\delta)\)-DP algorithm that achieves error \(\Delta^{(\kappa-1) / \kappa}\) for any convex function. We construct the latter by iteratively solving ERM problems with geometrically increasing \(\norm{\cdot}_2^\kappa\)-regularization towards the previous iterate to ensure the objective has \(\kappa\)-growth.

With this proposition, the proof of the theorem directly follows as [19] prove a lower bound \(\Omega(\sqrt{d}/(n\varepsilon))\) for ERM with \((\varepsilon,\delta)\)-DP.

Discussion↩︎

In this work, we develop private algorithms that adapt to the growth of the function at hand, achieving the convergence rate corresponding to the “easiest” sub-class the function belongs to. However, the picture is not yet complete. First of, there are still gaps in our theoretical understanding, the most interesting one being \(\kappa = 1\). On these functions, appropriate optimization algorithms achieve linear convergence [29] and raise the question, can we achieve exponentially small privacy cost in our setting? Finally, while our optimality guarantees are more fine-grained than the usual minimax results over convex functions, they are still contigent on some predetermined choice of sub-classes. Studying more general notions of adaptivity is an important future direction in private optimization.

6 Proofs for 3↩︎

6.1 Proof of [thm:inv-sens]↩︎

Let us first prove privacy. The sensitivity of \(\ltwo{\nabla f_\mathcal{S}(x)}\) is \(2L/n\) as \(F\) is \(L\)-Lipschitz, therefore following the privacy proof of the smooth inverse sensitivity mechanism [1] we get that \(\Agrinvsm\) 5 is \(\varepsilon\)-DP.

Let us now prove the claim about utility. Denote \(\hat{x} = \Agrinvsm(\mathcal{S})\) and \(E = \frac{ 2 LK}{n \varepsilon}\) with \(K\) to be chosen presently. We argue that it is enough to show that \(\Pr(G_\rho(\hat{x}) \ge E) \le \beta\). Indeed then with probability at least \(1-\beta\) we have \(G_\rho(\hat{x}) \le E\), which implies there is \(y\) such that \(\ltwo{\hat{x} -y}\le \rho\) and \(\ltwo{\nabla f_\mathcal{S}(y)} \le E\), hence using the inequality 2 \[\begin{align} f_\mathcal{S}(\hat{x}) - f_\mathcal{S}(x^\star) & = f_\mathcal{S}(\hat{x}) - f_\mathcal{S}(y) + f_\mathcal{S}(y) - f_\mathcal{S}(x^\star) \\ & \le L\rho + \frac{e}{\lambda^{\frac{1}{\kappa-1}}} \ltwo{\nabla f_\mathcal{S}(y)}^{\frac{\kappa}{\kappa-1}} \\ & \le L\rho + \frac{e}{\lambda^{\frac{1}{\kappa-1}}} E^{\frac{\kappa}{\kappa-1}}. \end{align}\]

It remains to prove that \(\Pr(G_\rho(\hat{x}) \ge E) \le \beta\). Let \(S_0 = \{ x \in \R^d : \ltwo{x - x^\star} \le \rho \}\) and \(S_1 = \{ x \in \R^d : G_\rho(x) \ge E \}\). Note that \(G_\rho(x) = 0\) for any \(x \in S_0\) as \(x^\star\) is in the interior of \(\mc{X}\) which implies \(\nabla f_\mathcal{S}(x^\star) = 0\). Hence the definition of the smooth inverse sensitivity mechanism 5 implies \[\begin{align} \Pr( \Agrinvsm(\mathcal{S}) \in S_1) & \le \frac{\vol(\{x \in \R^d : \ltwo{x - x^\star} \le D+ \rho \}) e^{-\frac{n \varepsilon}{2 L} E}}{ \vol(\{x \in \R^d : \ltwo{x - x^\star} \le \rho \})} \\ & \le e^{-K} \left(1 + \frac{D}{\rho} \right)^d \le \beta, \end{align}\] where the last inequality follows by choosing \(K = \log(1/\beta) + d \log(1 + D/\rho)\).

7 Proofs for 4↩︎

We need to the following result on the generalization properties of uniformly stable algorithms [37].

7.1 Proof of [thm:sco-hb-pure]↩︎

We begin by proving the privacy claim. We show that each iterate is \(\varepsilon\)-DP and therefore post-processing implies the claim as each sample is used in exactly one iterate. To this end, let \(\lambda_i = 1/\eta_i n_0\) and note that the minimizer \(\hat{x}_i\) has \(\ell_2\) sensitivity \(2L/\lambda_i n_0 \le 4 L\eta_i\) [2], hence the \(\ell_1\)-sensitivity is at most \(4 L\eta_i \sqrt{d}\). Standard properties of the Laplace mechanism [35] now imply that \(x_i\) is \(\varepsilon\)-DP which give the claim about privacy.

Now we proceed to prove utility which follows similar arguments to the localization-based proof in [2]. Letting \(\hat{x}_0 = x^\star\), we have: \[\begin{align} f(x_k) - f(x^\star) & = \sum_{i=1}^k f(\hat{x}_i) - f(\hat{x}_{i-1}) + f(x_k) - f(\hat{x}_k). \end{align}\] First, by using standard properties of Laplace distributions [42], we know that for \(\zeta_i \sim \laplace(\sigma_i)\), \[\Pr(\ltwo{\zeta_i} \ge t ) \le \Pr(\linf{\zeta_i} \ge t/\sqrt{d} ) \le d e^{-t/\sqrt{d} \sigma_i},\] which implies (as \(\beta \le 1/(n+d)\)) that with probability \(1-\beta/2\) we have \(\ltwo{\zeta_i} \le 10\sqrt{d} \sigma_i \log(1/\beta)\) for all \(1 \le i \le k\). Hence \[\begin{align} f(x_k) - f(\hat{x}_k) & \le L\ltwo{x_k - \hat{x}_k} \\ & \le L\sigma_k \sqrt{d} \log(1/\beta) \\ & \le 4 L^2 d \frac{\eta_i }{\varepsilon} \\ & \le 4 L^2 d \frac{\eta}{\varepsilon 2^{4i}} \le \frac{4 LD}{n^2}, \end{align}\] where the last inequality follows since \(\eta= \frac{D\varepsilon}{Ld \log(k/\beta)}\). Now we use high-probability generalization guarantees of uniformly-stable algorithms. We use 2 with \(F(x;s_j) + \frac{\ltwo{x- x_{i-1}}^2}{\eta_i n_0}\) to get that with probability \(1-\beta/2\) for each \(i\) \[\begin{align} f(\hat{x}_i) - f(\hat{x}_{i-1}) \le \frac{\ltwo{\hat{x}_{i-1} - x_{i-1}}^2}{\eta_i n_0} + {c L^2 \log(n) \log(1/\beta) \eta_i} + \frac{c LD\sqrt{\log(1/\beta)}}{\sqrt{n_0}}. \end{align}\] Thus, \[\begin{align} \sum_{i=1}^k f(\hat{x}_i) - f(\hat{x}_{i-1}) & \le \sum_{i=1}^k \crl*{\frac{\norm{\hat{x}_{i-1} - x_{i-1}}_2^2 }{\eta_i n_0} + {c L^2 \log(n) \log(1/\beta) \eta_i} + \frac{c LD\sqrt{\log(1/\beta)}}{\sqrt{n_0}}} \\ & \le \frac{D^2}{\eta n_0} + \brk*{\sum_{i=2}^k \frac{\sigma_{i-1}^2 d \log^2(1/\beta)}{\eta_i n_0}} + {2 c L^2 \log(n) \log(1/\beta) \eta} + \frac{c LD\sqrt{\log(1/\beta)} k}{\sqrt{n_0}} \\ & = \frac{D^2 }{\eta n_0} + \brk*{\sum_{i=2}^k \frac{C L^2 \eta_{i-1} d^2 \log^2(1/\beta)}{ n_0 \varepsilon^2}} + {2 c L^2 \log(n) \log(1/\beta) \eta} + \frac{c LD\sqrt{\log(1/\beta)} k}{\sqrt{n_0}} \\ & = \frac{D^2}{\eta n_0} + \frac{C L^2 \eta d^2 \log^2(1/\beta)}{ n_0 \varepsilon^2}\brk*{\sum_{i=2}^k 2^{-i}} + {2 c L^2 \log(n) \log(1/\beta) \eta} + \frac{c LD\sqrt{\log(1/\beta)} k}{\sqrt{n_0}} \\ & \le LD\cdot O \left( \frac{ \sqrt{\log(1/\beta) \log(n) } + \sqrt{\log(1/\beta)} \log^{3/2}(n) }{\sqrt{n}} + \frac{d \log(1/\beta) \log(n)}{n \varepsilon} \right), \end{align}\] where the last inequality follows by choosing \(\eta= \frac{D}{L} \min \left(\frac{1}{\sqrt{n \log(1/\beta)}} ,\frac{ \varepsilon}{ d \log(1/\beta)} \right)\)

7.2 Proof of [thm:sco-hb-appr]↩︎

The proof is similar to the proof of [thm:sco-hb-pure]. For privacy, we show in the proof of [thm:sco-hb-pure] that the \(\ell_2\)-sensitivity of \(\hat{x}_i\) is upper bounded by \(2L/\lambda_i n_0 \le 4 L\eta_i\) hence standard properties of the Gaussian mechanism [35] now imply that \(x_i\) is \((\varepsilon,\delta)\)-DP which implies the final algorithm is \((\varepsilon,\delta)\)-DP using post-processing.

The utility proof follows the same arguments as in the proof of [thm:sco-hb-pure], except that for \(\zeta_i \sim \normal(0,\sigma_i^2)\) we have [43] (since \(\zeta_i\) is \(2\sqrt{2}\sigma_i\sqrt{d}\)-norm-sub-Gaussian)\[\Pr(\ltwo{\zeta_i} \ge t \sqrt{d} ) \le 2 e^{-\tfrac{t^2}{16\sigma_i^2}},\] implying that \(\ltwo{\zeta_i} \le 4\sqrt{d} \sigma_i \log(4/\beta)\) for all \(1 \le i \le k\) with probability \(1-\beta/2\).

7.3 Proofs of Theorems [thm:sco-growth-pure] and [thm:sco-growth-appr]↩︎

We first restate Theorems [thm:sco-growth-pure] and [thm:sco-growth-appr].

We start by proving privacy. Since each sample \(s_i\) is used in exactly one iterate, we only need to show that each iterate is \((\varepsilon,\delta)\)-DP, which will imply the main claim using post-processing. The privacy of each iterate follows directly from the privacy guarantees of 1. We proceed to prove utility.

We will prove the utility claim assuming the subroutine used in 2 satisfies the following: the output \(x_{k+1}\) has error \[f(x_{k+1}) - \min_{x \in \mc{X}} f(x) \le D_k \cdot \rho,\] for some \(\rho >0\). Note that in our setting, [thm:sco-hb-pure] implies that \(\rho \le L\cdot O ( \frac{\sqrt{\log(1/\beta) } \log n_0}{\sqrt{n_0}} + \frac{ d \log(1/\beta)}{n_0 \varepsilon} )\) for pure-DP and similarly [thm:sco-hb-appr] gives the corresponding \(\rho\) for \((\varepsilon,\delta)\)-DP.

The proof has two stages. In the first stage (4), we prove that as long as \(i \le i_0\) for some \(i_0 >0\), then \(x^\star\in \mc{X}_i\) and the performance of the algorithm keeps improving. We show that at the end of this stage, the points \(x_{i_0+1}\) has optimal excess loss. Then, in the second stage (5), we show that the iterates would not move much as the radius \(D_i\) of the domain is sufficiently small, hence the final accumulated error along these iterations is small.

Let us begin with the first stage. Let \(i_0\) be the largest \(i\) such that \(D_i \ge (\frac{\kappa 2^\kappa \rho}{\lambda})^{\frac{1}{\kappa-1}}\). We prove that \(x^\star\in \mc{X}_i\) for all \(0 \le i \le i_0\) where we recall that \(\mc{X}_i = \crl{x \in \mc{X}: \ltwo{x - x_i} \le D_i}\) and \(D_i = 2^{-i} D_0\).

We now proceed to the second stage. The following lemma shows that the accumulated error along the iterates \(i > i_0\) is small and therefore \(x_T\) obtains the same error as \(x_{i_0+1}\) (up to constant factors).

Assuming \(T \ge i_0 + 1\), [thm:sco-growth-pure] and [thm:sco-growth-appr] now follow immediately from 5. Indeed, for the case of pure-DP (\(\delta=0\)), the choice of hyper-parameters in 2 and the guarantees of 1 ([thm:sco-hb-pure]) imply that \(\rho \le L\cdot O ( \frac{\sqrt{\log(1/\beta) } \log n_0}{\sqrt{n_0}} + \frac{ T d \log(1/\beta)}{n_0 \varepsilon} )\), which proves [thm:sco-growth-pure]. Similarly, [thm:sco-growth-appr] follows by using the guarantees of of 1 for approximate \((\varepsilon,\delta)\)-DP, that ism [thm:sco-hb-appr], which gives \(\rho \le L\cdot O \left( \frac{\sqrt{\log(1/\beta) } \log n_0}{\sqrt{n_0}} + \frac{ T \sqrt{d \log(1/\delta)} \log(1/\beta)}{n_0 \varepsilon} \right)\). Note that our choice of stepsize at each iterate implies that [thm:sco-growth-pure] guarantees the desired utility with probability at least \(1-\beta^2\), hence the final utility guarantee holds with probability at least \(1 - T \beta^2 \ge 1 - \beta\).

It remains to verify \(T \ge i_0 + 1\). Note that by choosing \(T \ge \frac{2 \log(D_0^{\kappa-1} \lambda/\rho)}{\underline{\kappa}-1}\), we get that \(D_T \le (\frac{\kappa 2^\kappa \rho}{\lambda})^{\frac{1}{\kappa-1}}\), hence \(T \ge i_0 + 1\). As we have \(\rho \ge L/\sqrt{n_0}\) (non-private error) and \(D_0^{\kappa - 1} \le L/\lambda\) in our setting, we get that choosing \(T = \frac{2 \log n}{\underline{\kappa}-1}\) gives the claim.

8 Proofs of Section 5↩︎

In this section, we provide the proofs for our lower bound under privacy constraints for functions with growth. This section is organized as follows: we prove in 8.1, the lower bounds under pure-DP and in 8.2, the lower bounds under approximate-DP. Within 8.1, we distinguish between \(\kappa \ge 2\) (8.1.1) and \(\kappa \in \prn{1, 2}\) (8.1.2).

8.1 Proofs of Section 5.1↩︎

8.1.1 Proof of [thm:lb-sco-private-large-kappa]↩︎

As we preview in the main text, the proof combines the (non-private) information-theoretic lower bounds of 3 with the (private) lower bound on ERM of 4. Finally, we show in [sec:sec:lb-erm-sco] that privately solving SCO is harder than privately solving ERM, concluding the proof of the theorem. We restate the theorem and prove these results in sequence.

8.1.1.1 Non-private lower bound

We begin the proof of [thm:lb-sco-private-large-kappa] by proving a (non-private) information-theoretic lower bound for minimizing functions with \(\kappa\ge 2\)-growth. We use the standard reduction from estimation to testing [44] in conjunction with Fano’s method [40], [41].

8.1.1.2 Privatizing the lower bound via a packing argument

We now show how this construction yields a private lower bound via a packing argument. For \(d\ge 1\), considering the ERM problem, the following private lower bound holds.

8.1.1.3 Reduction from \(\varepsilon\)-DP ERM to \(\varepsilon\)-DP SCO

We conclude the proof of the theorem by proving that SCO under privacy constraints is strictly harder than ERM. This is similar to Appendix C in [20] but we require it for pure-DP constraints. We make this formal in here.

We have the following lemma.

8.1.2 Proof of [thm:lb-small-kappa]↩︎

8.2 Proof for Section 5.2↩︎

References↩︎