Semidefinite programming

Semidefinite programming (SDP) is a subfield of convex optimization concerned with the optimization of a linear objective function over the intersection of the cone of positive semidefinite matrices with an affine space.

Semidefinite programming is a relatively new field of optimization which is of growing interest for several reasons. Many practical problems in operations research and combinatorial optimization can be modeled or approximated as semidefinite programming problems. In automatic control theory, SDP's are used in the context of linear matrix inequalities. SDPs are in fact a special case of cone programming and can be efficiently solved by interior point methods.All linear programs can be expressed as SDPs, and via hierarchies of SDPs the solutions of polynomial optimization problems can be approximated. Finally, semidefinite programming can aid in the design of quantum computing circuits Fact|date=February 2008, which makes it interesting as a future subject.

Definition

Notation

Denote by $mathbb\{S\}\_n$ the space of all $n\; imes\; n$ symmetric matrices.The space is equipped with the inner product (where $\{\; m\; tr\}$ denotes the trace)$langle\; A,B\; angle\_\{mathbb\{S\}^n\}\; =\; \{\; m\; tr\}(A^T\; B)\; =\; sum\_\{i=1,j=1\}^n\; A\_\{ij\}B\_\{ij\}.$A symmetric matrix is positive semidefinite if all its eigenvaluesare nonnegative; we write $Asucccurlyeq\; 0$. Similarly, $Asucc0$, $Apreccurlyeq\; 0$, and $Aprec\; 0$ means that $A$ is positivedefinite, negative semidefinite, and negative definite,respectively. Denote by $mathbb\{S\}\_+^n$ the convex cone of positivesemidefinite $n\; imes\; n$ matrices. This cone defines a partialorder for $A,B\; in\; mathbb\{S\}^n$ by $Asucccurlyeq\; B$ whenever $A-B$ ispositive semidefinite, $A-Bsucccurlyeq\; 0$.

Primal and dual form

Linear semidefinite programming (SDP) deals with optimizationproblems of the type:$min\; langle\; C,X\; angle\_\{mathbb\{S\}^n\}$:$mbox\{subject\; to\}$:$quad\; langle\; A\_i,X\; angle\_\{mathbb\{S\}^n\}\; =\; b\_i,quad\; i=1,ldots,m$:$quad\; X\; succcurlyeq\; 0$in variable $Xinmathbb\{S\}^n$. We refer to this problem as a"primal" semidefinite program (P-SDP).

Analogously to linear programming, we introduce a "dual"semidefinite program (D-SDP):$max\; langle\; b,y\; angle\_\{R^n\}$:$mbox\{subject\; to\}$:$quad\; sum\_\{i=1\}^m\; y\_i\; A\_i\; preccurlyeq\; C$in variable $yinmathbb\{R\}^m$.

For convenience, an SDP will often be specified in a slightly different, but equivalent form. For example, linear expressions involving nonnegative scalar variables may be added to the program specification. This remains an SDP because each variable can be incorporated into the matrix $X$ as a diagonal entry ($X\_\{ii\}$ for some $i$). To ensure that $X\_\{ii\}\; geq\; 0$, constraints $X\_\{ij\}\; =\; 0$ can be added for all $j\; eq\; i$. As another example, note that for any positive semidefinite matrix $X$, there exists a set of vectors $\{\; v\_i\; \}$ such that the $i$, $j$ entry of $X$ is $X\_\{ij\}\; =\; (v\_i,\; v\_j)$ the scalar product of $v\_i$ and $v\_j$. Therefore, SDPs are often formulated in terms of linear expressions on scalar products of vectors. Given the solution to the SDP in the standard form, the vectors $\{\; v\_i\; \}$ can be recovered in $O(n^3)$ time (e.g., by using an incomplete Cholesky decomposition of X).

Duality Theorem

(i) Suppose the primal problem (P-SDP) is bounded below and strictlyfeasible (i.e., there exists$X\_0inmathbb\{S\}^n,\; X\_0succ\; 0$ such that $langleA\_i,X\_0\; angle\_\{mathbb\{S\}^n\}\; =\; b\_i$, $i=1,ldots,m$).Then there is an optimal solution $y^*$ to (D-SDP) and:$langle\; C,X^*\; angle\_\{mathbb\{S\}^n\}\; =\; langle\; b,y^*\; angle\_\{R^n\}.$

(ii) Suppose the dual problem (D-SDP) is bounded above and strictlyfeasible (i.e.,$sum\_\{i=1\}^m\; (y\_0)\_i\; A\_iprec\; 0$ for some $y\_0inR^m$).Then there is an optimal solution $X^*$ to (P-SDP) andthe equality from (i) holds.

Examples

Example 1

Consider three random variables $A$, $B$, and $C$. By definition, their correlation coefficients $ho\_\{AB\},\; ho\_\{AC\},\; ho\_\{BC\}$ are valid if and only if

:

Suppose that we know from some prior knowledge (empirical results of an experiment, for example) that $-0.2\; leq\; ho\_\{AB\}\; leq\; -0.1$ and $0.4\; leq\; ho\_\{BC\}\; leq\; 0.5$. The problem of determining the smallest and largest values that $ho\_\{AC\}$ can take is given by:

:minimize/maximize $x\_\{13\}$:subject to:$-0.2\; leq\; x\_\{12\}\; leq\; -0.1$:$0.4\; leq\; x\_\{23\}\; leq\; 0.5$:$x\_\{11\}\; =\; x\_\{22\}\; =\; x\_\{33\}\; =\; 1$:

we set $ho\_\{AB\}\; =\; x\_\{12\},\; ho\_\{AC\}\; =\; x\_\{13\},\; ho\_\{BC\}\; =\; x\_\{23\}$ to obtain the answer. This can be formulated by an SDP. We handle the inequality constraints by augmenting the variable matrix and introducing slack variables, for example

Solving this SDP gives the minimum and maximum values of $ho\_\{AC\}\; =\; x\_\{13\}$ as $-0.978$ and $0.872$ respectively.

Example 2

Consider the problem

: minimize $frac\{(c^T\; x)^2\}\{d^Tx\}$: subject to $Ax\; +bgeq\; 0$

where we assume that $d^Tx>0$ whenever $Ax+bgeq\; 0$.

Introducing an auxiliary variable $t$ the problem can be reformulated:

: minimize $t$: subject to $Ax+bgeq\; 0,\; ,\; frac\{(c^T\; x)^2\}\{d^Tx\}leq\; t$

In this formulation, the objective is a linear function of the variables $x,t$.

The first restriction can be written as

: $extbf\{diag\}(Ax+b)geq\; 0$

where the matrix $extbf\{diag\}(Ax+b)$ is the square matrix with values in the diagonal equalto the elements of the vector $Ax+b$.

The second restriction can be written as

: $td^Tx-(c^Tx)^2geq\; 0$

or equivalently

: det

Thus $Dgeq\; 0$.

The semidefinite program associated with this problem is

: minimize $t$: subject to

Example 3 (Goemans-Williamson MAX CUT approximation algorithm)

Semidefinite programs are important tools for developing approximation algorithms for NP-hard maximization problems. The first approximation algorithm based on an SDP is due to Goemans and Williamson (JACM, 1995). They studied the MAX CUT problem: Given a graph $G\; =\; (V,\; E)$, output a partition of the vertices $V$ so as to maximize the number of edges crossing from one side to the other. This problem can be expressed as an integer quadratic program: Maximize $sum\_\{(i,j)\; in\; E\}\; frac\{1-v\_\{i\}\; v\_\{j\{2\}$ such that each $v\_i$ is either $1$ or $-1$. Unless $P\; =\; NP$, we cannot solve this maximization problem efficiently. However, Goemans and Williamson observed a general three-step procedure for attacking this sort of problem: 1. Relax the integer quadratic program into an SDP. 2. Solve the SDP (to within an arbitrarily small additive error $epsilon$). 3. Round the SDP solution to obtain an approximate solution to the original integer quadratic program. For MAX CUT, the most natural relaxation is $max\; sum\_\{(i,j)\; in\; E\}\; frac\{1-langle\; v\_\{i\},\; v\_\{j\}\; angle\}\{2\}$ such that $lVert\; v\_i\; Vert^2\; =\; 1$, where the maximization is over vectors $\{v\_i\}$ instead of integer scalars. (This is an SDP because the objective function and constraints are all linear functions of vector inner products.) Solving the SDP gives a set of unit vectors in $mathbf\{R^n\}$; since the vectors are not required to be collinear, the value of this relaxed program can only be higher than the value of the original quadratic integer program. Finally, a rounding procedure is needed to obtain a partition. Goemans and Williamson simply choose a uniformly random hyperplane through the origin and divide the vertices according to which side of the hyperplane the corresponding vectors lie. Straightforward analysis shows that this procedure achieves an expected approximation ratio (performance guarantee) of $0.87856\; -\; epsilon$. (The expected value of the cut is the sum over edges of the probability that the edge is cut, which is simply arccos of the angle between the vectors at the endpoints of the edge over $pi$. Comparing this probability to $frac\{1-langle\; v\_\{i\},\; v\_\{j\}\; angle\}\{2\}$, the ratio is always at least $0.87856$.) Assuming the Unique Games Conjecture, it can be shown that this approximation ratio is essentially optimal.

Since the original paper of Goemans and Williamson, SDPs have been applied to develop numerous approximation algorithms. Recently, Prasad Raghavendra has developed a general framework for constraint satisfaction problems based on the Unique Games Conjecture (STOC 2008).

Algorithms

There are several types of algorithms for solving SDPs. These algorithms output the value of the SDP up to an additive error $epsilon$ in time that is polynomial in the program description size and $log\; (1/epsilon)$.

Interior point methods

Most codes are based on interior point methods (CSDP, SeDuMi, SDPT3, DSDP, SDPA). Robust and efficient for general linear SDP problems. Restricted by the fact that the algorithms are second-order methods and need to store and factorize a large (and often dense) matrix.

Bundle method

The code SBmethod formulates the SDP problem as a nonsmooth optimization problem and solves it by the Spectral Bundle method of nonsmooth optimization. This approach can only be used for a special class of linear SDP problems but then it is very efficient.

Other

Algorithms based on augmented Lagrangian method (PENSDP) are similar in behavior to the interior point methods and can be specialized to some very large scale problems. Other algorithms use low-rank information and reformulation of the SDP as a nonlinear programming problem (SPDLR).

Software

The following codes are available for SDP:

SDPA, CSDP, SDPT3, SeDuMi, DSDP, PENSDP, SDPLR, SBmeth

SeDuMi runs on MATLAB and uses the Self-Dual method for solving general convex optimization problems.

Applications

Semidefinite programming has been applied to find approximate solutions to combinatorial optimization problems, such as the solution of the max cut problem with an approximation ratio of 0.87856. SDPs are also used in geometry to determine tensegrity graphs, and arise in control theory as LMIs.

See also

*Sum of squares

References

* Lieven Vandenberghe, Stephen Boyd, "Semidefinite Programming", SIAM Review 38, March 1996, pp. 49-95. [http://stanford.edu/~boyd/papers/pdf/semidef_prog.pdf pdf]

* Monique Laurent, Franz Rendl, "Semidefinite Programming and Integer Programming", Report PNA-R0210, CWI, Amsterdam, April 2002. [http://www.optimization-online.org/DB_HTML/2002/12/585.html optimization-online]

* E. de Klerk, "Aspects of Semidefinite Programming: Interior Point Algorithms and Selected Applications", Kluwer Academic Publishers, March 2002, ISBN 1-4020-0547-4.

External links

* [http://www-user.tu-chemnitz.de/~helmberg/semidef.html Christoph Helmberg's page giving links to introductions and events in the field]

;Software
* [http://www-user.tu-chemnitz.de/~helmberg/sdp_software.html Christoph Helmberg's overview over existing software.]
* [http://sedumi.mcmaster.ca/ SeDuMi: (Self-Dual-Minimization) solves optimization problems over symmetric cones (this includes semidefinite optimization).]
* [http://control.ee.ethz.ch/~joloef/yalmip.php YALMIP: MATLAB optimization toolbox and frontend for many sdp solver]
* [http://www.stanford.edu/~boyd/cvx/ cvx: a MATLAB frontend that uses either SeDuMi or SDPT3]
* [http://abel.ee.ucla.edu/cvxopt cvxopt: a python solver]