1. CSE 203B: Convex Optimization Week 4 Discuss Session
Xinyuan Wang
01/30/2019

2. Contents Convex functions (Ref. Chap.3)
Review: definition, first order condition, second order condition, operations that preserve convexity
Epigraph
Conjugate function

3. Review of Convex Function
Definition ( often simplified by restricting to a line)
First order condition: a global underestimator
Second order condition
Operations that preserve convexity
Nonnegative weighted sum
Composition with affine function
Pointwise maximum and supremum
Composition
Minimization
Perspective
See Chap 3.2, try to prove why the convexity is preserved with those operations.

4. Epigraph 𝛼-sublevel set of 𝑓: 𝑅 𝑛 →𝑅 𝐶 𝛼 = 𝑥∈𝑑𝑜𝑚 𝑓 𝑓 𝑥 ≤𝛼}
𝐶 𝛼 = 𝑥∈𝑑𝑜𝑚 𝑓 𝑓 𝑥 ≤𝛼}
sublevel sets of a convex function are convex for any value of 𝛼.
Epigraph of 𝑓: 𝑅 𝑛 →𝑅 is defined as
𝐞𝐩𝐢 𝑓= (𝑥,𝑡) 𝑥∈𝑑𝑜𝑚 𝑓,𝑓 𝑥 ≤𝑡}⊆ 𝑅 𝑛+1
A function is convex iff its epigraph is a convex set.

5. Relation between convex sets and convex functions
A function is convex iff its epigraph is a convex set.
Consider a convex function 𝑓 and 𝑥, 𝑦∈𝑑𝑜𝑚 𝑓
𝑡≥𝑓 𝑦 ≥𝑓 𝑥 +𝛻𝑓 𝑥 𝑇 (𝑦−𝑥)
The hyperplane supports epi 𝒇 at (𝑥, 𝑓 𝑥 ), for any
𝑦,𝑡 ∈𝐞𝐩𝐢 𝑓⇒
𝛻𝑓 𝑥 𝑇 𝑦−𝑥 +𝑓 𝑥 −𝑡≤0
⇒ 𝛻𝑓 𝑥 −1 𝑇 𝑦 𝑡 − 𝑥 𝑓 𝑥 ≤0
First order condition for convexity
epi 𝒇 𝑡 𝑥
Supporting hyperplane, derived from first order condition

6. Pointwise Supremum
If for each 𝑦∈𝑈, 𝑓(𝑥, 𝑦): 𝑅 𝑛 →𝑅 is convex in 𝑥, then function
𝑔(𝑥)= sup 𝑦∈𝑈 𝑓(𝑥,𝑦)
is convex in 𝑥.
Epigraph of 𝑔 𝑥 is the intersection of epigraphs with 𝑓 and set 𝑈
𝐞𝐩𝐢 𝑔= ∩ 𝑦∈𝑈 𝐞𝐩𝐢 𝑓(∙ ,𝑦)
knowing 𝐞𝐩𝐢 𝑓(∙ ,𝑦) is a convex set (𝑓 is a convex function in 𝑥 and regard 𝑦 as a const), so 𝐞𝐩𝐢 𝑔 is convex.
An interesting method to establish convexity of a function: the pointwise supremum of a family of affine functions. (ref. Chap 3.2.4)

7. Conjugate Function Given function 𝑓: 𝑅 𝑛 →𝑅 , the conjugate function
𝑓 ∗ 𝑦 = sup 𝑥∈𝑑𝑜𝑚 𝑓 𝑦 𝑇 𝑥 −𝑓(𝑥)
The dom 𝑓 ∗ consists 𝑦∈ 𝑅 𝑛 for which sup 𝑥∈𝑑𝑜𝑚 𝑓 𝑦 𝑇 𝑥 −𝑓(𝑥) is bounded
Theorem: 𝑓 ∗ (𝑦) is convex even 𝑓 𝑥 is not convex.
Supporting hyperplane: if 𝑓 is convex in that domain
Slope 𝑦=−1
( 𝑥 , 𝑓( 𝑥 )) where 𝛻 𝑓 𝑇 𝑥 =𝑦 if 𝑓 is differentiable
Slope 𝑦=0
(0, − 𝑓 ∗ (0))
(Proof with pointwise supremum)
𝒚 𝑻 𝒙 −𝒇(𝒙) is affine function in 𝒚

8. Examples of Conjugates
Derive the conjugates of 𝑓:𝑅→𝑅
𝑓 ∗ 𝑦 = sup 𝑥∈𝑑𝑜𝑚 𝑓 𝑦𝑥 −𝑓(𝑥)
Affine
quadratic
Norm
See the extension to domain 𝑹 𝒏 in Example 3.21

9. Examples of Conjugates
Log-sum-exp: 𝑓 𝑥 = log ( Σ 𝑖=1 n 𝑒 𝑥 𝑖 ) , 𝑥∈ 𝑅 𝑛 . The conjugate is
𝑓 ∗ 𝑦 = sup 𝑥∈𝑑𝑜𝑚 𝑓 𝑦 𝑇 𝑥 −𝑓(𝑥)
Since 𝑓(𝑥) is differentiable, first calculate the gradient of 𝑔 𝑥 = 𝑦 𝑇 𝑥 −𝑓(𝑥)
𝛻𝑔 𝑥 =𝑦− 𝑍 𝟏 𝑇 𝑍
where 𝑍= 𝑒 𝑥 1 , ⋯ 𝑒 𝑥 𝑛 𝑇 . The maximum of 𝑔 𝑥 is attained at 𝛻𝑔 𝑥 =0, so that 𝑦= 𝑍 𝟏 𝑇 𝑍 .
𝑦 𝑖 = 𝑒 𝑥 𝑖 Σ 𝑖=1 n 𝑒 𝑥 𝑖 , 𝑖=1,…, 𝑛
Does the bounded supremum (not →∞) exist for all 𝒚∈ 𝑹 𝒏 ?

10. Examples of Conjugates
Does the bounded supremum (not →∞) exist for all 𝒚∈ 𝑹 𝒏 ?
If 𝑦 𝑖 <0, then the gradient on 𝛻 𝑖 𝑔= 𝑦 𝑖 − 𝑒 𝑥 𝑖 Σ 𝑖=1 n 𝑒 𝑥 𝑖 <0 for all 𝑥 𝑖 ∈𝑅. Then 𝑔 𝑥 reaches maximum at ∞ at 𝑥→−∞. So 𝑦≽0.
To have 𝑦= 𝑍 1 𝑇 𝑍 solvable, we notice that 𝟏 𝑇 𝑦= 𝟏 𝑇 𝑍 1 𝑇 𝑍 =1.
If 𝟏 𝑇 𝑦≠1 and 𝑦≽0, we choose 𝑥=𝑡𝟏 s.t.
𝑔 𝑥 = 𝑦 𝑇 𝑥 −𝑓 𝑥 =𝑡 𝑦 𝑇 𝟏− log 𝑛 𝑒 𝑡
=𝑡 𝑦 𝑇 𝟏−1 − log 𝑛
𝑔 𝑥 reaches maximum at ∞ when we let 𝑡→∞(−∞), which is not bounded.
The conjugate function 𝑓 ∗ 𝑦 = Σ 𝑖=1 n 𝑦 𝑖 log 𝑦 𝑖 if 𝟏 𝑇 𝑦=1 and 𝑦≽0 ∞ otherwise