1. CIS 700: “algorithms for Big Data”
Lecture 9:
Compressed Sensing
Slides at
Grigory Yaroslavtsev

2. Compressed Sensing
Given a sparse signal 𝑥∈ ℝ 𝑛 can we recover it from a small number of measurements?
Goal: design 𝐴∈ ℝ 𝑑×𝑛 which allows to recover any 𝑠-sparse 𝑥∈ ℝ 𝑛 from 𝐴𝑥.
𝐴 = matrix of i.i.d. Gaussians 𝑁(0,1)
Application: signals are usually sparse in some Fourier domain

3. Reconstruction Reconstruction: min 𝑥 0 , subject to: 𝐴𝑥=𝑏
Uniqueness: If there are two 𝑠-sparse solutions 𝑥 1 , 𝑥 2 :
𝐴 𝑥 1 − 𝑥 2 =0
then 𝐴 has 2𝑠 linearly dependent columns
If 𝑑=Ω( 𝑠 2 ) and 𝐴 is Gaussian then unlikely to have linearly dependent columns
𝑥 0 not convex, NP-hard to reconstruct
𝑥 0 → 𝑥 1 : min 𝑥 1 , subject to: 𝐴𝑥=𝑏
When does this give sparse solutions?

4. Subgradient min 𝑥 1 , subject to: 𝐴𝑥=𝑏
𝑥 1 is convex but not differentiable
Subgradient 𝛻f:
equal to gradient where 𝑓 is differentiable
any linear lower bound where 𝑓 is not differentiable
∀ 𝑥 0 ,Δ𝑥: 𝑓 𝑥 0 +Δ𝑥 ≥𝑓 𝑥 𝛻𝑓 𝑇 Δ𝑥
Subgradient for 𝑥 1 :
𝛻 𝑥 𝑖 =𝑠𝑖𝑔𝑛( 𝑥 𝑖 ) if 𝑥 𝑖 ≠0
𝛻 𝑥 𝑖 ∈ −1,1 if 𝑥 𝑖 =0
For all Δ𝑥 such that 𝐴Δ𝑥=0 satisfies 𝛻 𝑇 Δ𝑥≥0
Sufficient: ∃𝑤 such that 𝛻= 𝐴 𝑇 𝑤 so 𝛻 𝑇 Δ𝑥= 𝑤𝐴Δ𝑥=0

5. Exact Reconstruction Property
Subgradient Thm. If 𝐴 𝑥 0 =𝑏 and there exists a subgradient 𝛻 for 𝑥 1 such that 𝛻= 𝐴 𝑇 𝑤 and columns of 𝐴 corresponding to 𝑥 0 are linearly independent then 𝑥 0 minimizes 𝑥 1 and is unique.
(Minimum): Assume 𝐴𝑦=𝑏. Will show
𝑦 1 ≥ 𝑥
𝑧=𝑦− 𝑥 0 ⇒𝐴𝑧=𝐴𝑦−𝐴 𝑥 0 =0
𝛻 𝑇 𝑧=0⇒
𝑦 1 = 𝑥 0 +𝑧 ≥ 𝑥 𝛻 𝑇 𝑧= 𝑥

6. Exact Reconstruction Property
(Uniqueness): assume 𝑥 0 is another minimum
𝛻 at 𝑥 0 is also a subgradient at 𝑥 0
∀Δ𝑥:𝐴Δ𝑥=0:
𝑥 0 +Δ𝑥 1 = 𝑥 0 + 𝑥 0 − 𝑥 0 +Δ𝑥
≥ 𝑥 𝛻 𝑇 ( 𝑥 0 − 𝑥 0 +Δ𝑥)
= 𝑥 𝛻 𝑇 𝑥 0 − 𝑥 0 + 𝛻 T Δ𝑥
𝛻 𝑇 𝑥 0 − 𝑥 0 = 𝑤 𝑇 𝐴 𝑥 0 − 𝑥 0 = 𝑤 𝑇 𝑏−𝑏 =0
𝑥 0 +Δ𝑥 1 ≥ 𝑥 𝛻 T Δ𝑥
𝛻 i =sign (x 0 i )=sign ( 𝑥 0 i ) if either is non-zero, otherwise equal to 0
⇒ 𝑥 0 and 𝑥 0 have same sparsity pattern
By linear independence of columns of 𝐴: 𝑥 0 = 𝑥 0

7. Restricted Isometry Property
Matrix 𝐴 satisfies restricted isometry property (RIP), if for any 𝑠-sparse 𝑥 there exists 𝛿 𝑠 :
1 − 𝛿 𝑠 𝑥 ≤ 𝐴𝑥 ≤ 1+ 𝛿 𝑠 𝑥 2 2
Exact isometry:
all eigenvalues are ±1
for orthogonal 𝑥,𝑦: 𝑥 𝑇 𝐴 𝑇 𝐴𝑦=0
Let 𝐴 𝑆 be the set of columns of 𝐴 in set 𝑆
Lem: If 𝐴 satisfies RIP and 𝛿 𝑠 1 + 𝑠 2 ≤ 𝛿 𝑠 1 + 𝛿 𝑠 2 :
For 𝑆 of size 𝑠 singular values of 𝐴 𝑆 in [1− 𝛿 𝑠 ,1+ 𝛿 𝑠 ]
For any orthogonal 𝑥,𝑦 with supports of size 𝑠 1 , 𝑠 2 :
𝑥 𝑇 𝐴 𝑇 𝐴𝑦 ≤ 𝑥 | 𝑦 |( 𝛿 𝑠 1 + 𝛿 𝑠 2 )

8. Restricted Isometry Property
Lem: If 𝐴 satisfies RIP and 𝛿 𝑠 1 + 𝑠 2 ≤ 𝛿 𝑠 1 + 𝛿 𝑠 2 :
For 𝑆 of size 𝑠 singular values of 𝐴 𝑆 in [1− 𝛿 𝑠 ,1+ 𝛿 𝑠 ]
For any orthogonal 𝑥,𝑦 with supports of size 𝑠 1 , 𝑠 2 :
𝑥 𝑇 𝐴 𝑇 𝐴𝑦 ≤3/2 𝑥 𝑦 ( 𝛿 𝑠 1 + 𝛿 𝑠 2 )
W.l.o.g 𝑥 = 𝑦 =1 so 𝑥+𝑦 2 =2
2 1− 𝛿 𝑠 1 + 𝑠 2 ≤ 𝐴 𝑥+𝑦 2 ≤ 2 1+ 𝛿 𝑠 1 + 𝑠 2
2 1− 𝛿 𝑠 1 + 𝛿 𝑠 2 ≤ 𝐴 𝑥+𝑦 2 ≤ 𝛿 𝑠 1 + 𝛿 𝑠 2
1 − 𝛿 𝑠 1 ≤ 𝐴𝑥 2 ≤(1+ 𝛿 𝑠 1 )
1 − 𝛿 𝑠 2 ≤ 𝐴𝑦 2 ≤(1+ 𝛿 𝑠 2 )

9. Restricted Isometry Property
2 𝑥 𝑇 𝐴 𝑇 𝐴𝑦
= 𝑥+𝑦 𝑇 𝐴 𝑇 𝐴 𝑥+𝑦 − 𝑥 𝑇 𝐴 𝑇 𝐴𝑥− 𝑦 𝑇 𝐴 𝑇 𝐴𝑦
= 𝐴 𝑥+𝑦 2 − 𝐴𝑥 2 − 𝐴𝑦 2
2 𝑥 𝑇 𝐴 𝑇 𝐴𝑦≤2 1+ 𝛿 𝑠 1 + 𝛿 𝑠 2 − 1 − 𝛿 𝑠 1 − 1 − 𝛿 𝑠 2 =3( 𝛿 𝑠 1 + 𝛿 𝑠 2 )
𝑥 𝑇 𝐴 𝑇 𝐴𝑦≤ 𝑥 ⋅ 𝑦 ⋅( 𝛿 𝑠 1 + 𝛿 𝑠 2 )

10. Reconstruction from RIP
Thm. If 𝐴 satisfies RIP with 𝛿 𝑠+1 ≤ 𝑠 and 𝑥 0 is 𝑠-sparse and satisfies 𝐴 𝑥 0 =𝑏. Then a 𝛻( ⋅ 1 ) exists at 𝑥 0 which satisfies conditions of the “subgradient theorem”.
Implies that 𝑥 0 is the unique minimum 1-norm solution to 𝐴𝑥=𝑏.
𝑆= 𝑖 𝑥 0 𝑖 ≠0 , 𝑆 = 𝑖 𝑥 0 𝑖 =0
Find subgradient 𝑢 search for 𝑤: 𝑢= 𝐴 𝑇 𝑤
for 𝑖∈𝑆: 𝑢 𝑖 =𝑠𝑖𝑔𝑛 𝑥 0
2-norm of the coordinates in 𝑆 is minimized

11. Reconstruction from RIP
Let 𝑧 be a vector with support 𝑆:
𝑧 𝑖 =sign (𝑥 0 𝑖 )
Let 𝑤= 𝐴 𝑆 𝐴 𝑆 𝑇 𝐴 𝑆 −1 𝑧
𝐴 𝑆 has independent columns by RIP
For coordinates in 𝑆:
𝐴 𝑇 𝑤 𝑆 = 𝐴 𝑆 𝑇 𝐴 𝑆 𝐴 𝑆 𝑇 𝐴 𝑆 −1 𝑧=𝑧
For coordinates in 𝑆 :
𝐴 𝑇 𝑤 𝑆 = 𝐴 𝑆 𝑇 𝐴 𝑆 𝐴 𝑆 𝑇 𝐴 𝑆 −1 𝑧
Eigenvalues of 𝐴 𝑆 𝑇 𝐴 𝑆 are in 1− 𝛿 𝑠 2 , 1+ 𝛿 𝑠 2
|| 𝐴 𝑆 𝑇 𝐴 𝑆 −1 ||≤ 1 1− 𝛿 𝑠 2 , let 𝑝= 𝐴 𝑆 𝑇 𝐴 𝑆 −1 𝑧, 𝑝 ≤ 𝑠 1− 𝛿 𝑠 2
𝐴 𝑆 𝑝=𝐴𝑞 where 𝑞 has all coordinates in 𝑆 equal 0
For 𝑗∈ 𝑆 : 𝐴 𝑇 𝑤 𝑗 = 𝑒 𝑗 𝑇 𝐴 𝑇 𝐴𝑞 so | 𝐴 𝑇 𝑤 𝑗 |≤ 𝛿 𝑠 + 𝛿 1 𝑠 1− 𝛿 𝑠 2 ≤ 𝛿 𝑠 𝑠 1− 𝛿 𝑠 2 ≤ 1 2