Information Complexity Lower Bounds Rotem Oshman, Princeton CCI Based on: Bar-Yossef,Jayram,Kumar,Srinivasan’04 Braverman,Barak,Chen,Rao’10

Communication Complexity 𝑓 𝑋,𝑌 = ? 𝑋 𝑌 Yao ‘79, “Some complexity questions related to distributive computing”

Communication Complexity Applications: Circuit complexity Streaming algorithms Data structures Distributed computing Property testing …

Deterministic Protocols A protocol Π specifies, at each point: Which player speaks next What should the player say When to halt and what to output Formally, Π: 0,1 ∗ → 𝐴,𝐵,⊥ × 0,1 what we’ve said so far who speaks next: 𝐴= Alice, 𝐵= Bob, ⊥ = halt what to say/output

Randomized Protocols Can use randomness to decide what to say Private randomness: each player has a separate source of random bits Public randomness: both players can use the same random bits Goal: for any 𝑋,𝑌 compute 𝑓 𝑋,𝑌 correctly with probability ≥1 −𝜖 Communication complexity: worst-case length of transcript in any execution

Randomness Can Help a Lot Example: Equality 𝑋,𝑌 Input: 𝑋,𝑌∈ 0,1 𝑛 Output: is 𝑋=𝑌 ? Trivial protocol: Alice sends 𝑋 to Bob For deterministic protocols, this is optimal!

Equality Lower Bound 0 𝑛 1 𝑛 … 1 #rectangles ≤ 2 transcript

Randomized Protocol Protocol with public randomness: Select random 𝑍∈ 0,1 𝑛 Alice sends 𝑋,𝑍 = 𝑖=1 𝑛 𝑋 𝑖 𝑍 𝑖 mod 2 Bob accepts iff 𝑌,𝑍 = 𝑋,𝑍 If 𝑋=𝑌: always accept If 𝑋≠𝑌: 𝑋,𝑍 + 𝑌,𝑍 mod 2= 𝑋+𝑌,𝑍 mod 2 Reject with probability 1/2 non-zero vector

Set Disjointness Input: 𝑋,𝑌⊆ 1,…,𝑛 Output: 𝑋∩𝑌=∅ ? Theorem [Kalyanasundaran, Schnitger ‘92, Razborov ‘92]: randomized CC = Ω 𝑛 Easy to see for deterministic protocols Today we’ll see a proof by Bar-Yossef, Jayram, Kumar, Srinivasan ‘04

Application: Streaming Lower Bounds Streaming algorithm: Example: how many distinct items in the data? Reduction from Disjointness [Alon, Matias, Szegedy ’99] How much space is required to approximate f(data)? algorithm data

Reduction from Disjointness: Fix a streaming algorithm for Distinct Elements with space 𝑆, universe size 𝑛 Construct a protocol for Disj. with 𝑛 elements: 𝑋={ 𝑥 1 ,…, 𝑥 𝑘 } 𝑌={ 𝑦 1 ,…, 𝑦 ℓ } algorithm 𝑋∩𝑌=∅ ⇔ #distinct elements in 𝑋∪𝑌 is 𝑋 + 𝑌 State of the algorithm and 𝑋 (#bits = 𝑆+log 𝑛)

Application 2: KW Games Circuit depth lower bounds: How deep does the circuit need to be? ∧ ∨ 𝑥 1 … 𝑥 𝑛 𝑓(𝑥 1 ,…, 𝑥 𝑛 )

Application 2: KW Games Karchmer-Wigderson’93,Karchmer-Raz-Wigderson’94: find 𝑖 such that 𝑋 𝑖 ≠ 𝑌 𝑖 𝑋 :𝑓 𝑋 =0 𝑌:𝑓 𝑌 =1

Application 2: KW Games Claim: if 𝐾 𝑊 𝑓 has deterministic CC ≥𝑑, then 𝑓 requires circuit depth ≥𝑑. Circuit with depth 𝑑 ⇒ protocol with length 𝑑 1 ∧ ∨ 𝑥 1 … 𝑥 𝑛 1 1 1 𝑋 :𝑓 𝑋 =0 𝑌:𝑓 𝑌 =1

Information-Theoretic Lower Bound on Set Disjointness

Some Basic Concepts from Info Theory Entropy of a random variable: 𝐻 𝑋 =− 𝑥 Pr 𝑋=𝑥 log Pr 𝑋=𝑥 Important properties: 𝐻 𝑋 ≥0 𝐻 𝑋 =0 ⇒ 𝑋 is deterministic 𝐻(𝑋) = expected # bits needed to encode 𝑋

Some Basic Concepts from Info Theory Conditional entropy: 𝐻 𝑋 𝑌 = 𝐸 𝑦 𝐻( 𝑋 |𝑌=𝑦 ) Important properties: 𝐻 𝑋|𝑌 ≤𝐻(𝑋) 𝐻 𝑋|𝑌 =𝐻 𝑋 ⇒ 𝑋,𝑌 are independent Example: 𝑋,𝑍∼Bernoulli 1 2 𝐻 𝑋 =− 1 2 log 1 2 − 1 2 log 1 2 =1 If 𝑍=0 then 𝑌=𝑋, if 𝑍=1 then 𝑌=1−𝑋 𝐻 𝑋 𝑌,𝑍 = 1 2 𝐻 𝑋|𝑋 + 1 2 𝐻 𝑋|1−𝑋 =0 𝐻 𝑋 𝑌 =1

Some Basic Concepts from Info Theory Mutual information: 𝐼 𝑋;𝑌 =𝐻 𝑋 −𝐻 𝑋 𝑌 =𝐻 𝑌 −𝐻(𝑌|𝑋) Conditional mutual information: 𝐼 𝑋;𝑌|𝑍 =𝐻 𝑋|𝑍 −𝐻 𝑋 𝑌,𝑍 =𝐻 𝑌|𝑍 −𝐻(𝑌|𝑋,𝑍) Important properties: 𝐼 𝑋;𝑌 ≥0 𝐼 𝑋;𝑌 =0 ⇒ 𝑋,𝑌 are independent

Some Basic Concepts from Info Theory Chain rule for mutual information: 𝐼 𝑋 1 , 𝑋 2 ;𝑌 =𝐼 𝑋 1 ;𝑌 +𝐼 𝑋 2 ;𝑌 𝑋 1 More generally, 𝐼 𝑋 1 ,…, 𝑋 𝑘 ;𝑌 = 𝑖=1 𝑘 𝐼 𝑋 𝑖 ;𝑌 𝑋 1 ,…, 𝑋 𝑖−1

Information Cost of Protocols Fix an input distribution 𝜇 on 𝑋,𝑌 Given a protocol Π, let Π also denote the distribution of Π’s transcript Information cost of Π: 𝐼𝐶 Π =𝐼 Π;𝑌 𝑋 +𝐼 Π;𝑋 𝑌 Information cost of a function 𝑓: 𝐼 𝐶 𝜖 𝑓 = inf Π solves 𝑓 𝑤/error≤𝜖 𝐼𝐶 Π

Information Cost of Protocols Important property: 𝐼𝐶 Π ≤|Π| Proof: by induction. Let Π= Π 1 … Π 𝑡 . ∀𝑟≤𝑡 : 𝐼 Π ≤𝑟 ;𝑌 𝑋 +𝐼 Π ≤𝑟 ;𝑋 𝑌 ≤𝑟. 𝐼 Π ≤𝑟 ;𝑌 𝑋 +𝐼 Π ≤𝑟 ;𝑋 𝑌 what we know after r rounds =𝐼 Π <𝑟 ;𝑌 𝑋 +𝐼 Π <𝑟 ;𝑋 𝑌 what we knew after r-1 rounds + 𝐼 Π 𝑟 ;Y X, Π <𝑟 +𝐼 Π 𝑟 ;X Y, Π <𝑟 what we learn in round r, given what we already know

Information vs. Communication Want: 𝐼 Π 𝑟 ;Y X, Π <𝑟 +𝐼 Π 𝑟 ;X Y, Π <𝑟 ≤1. Suppose Π 𝑟 is sent by Alice. What does Alice learn? Π 𝑟 is a function of Π <𝑟 and 𝑋, so 𝐼 Π 𝑟 ;Y X, Π <𝑟 =0. What does Bob learn? 𝐼 Π 𝑟 ;Y X, Π <𝑟 ≤ Π 𝑟 =1.

Information vs. Communication Important property: 𝐼𝐶 Π ≤|Π| Lower bound on information cost ⇒ lower bound on communication complexity In fact, IC lower bounds are the most powerful technique we know

Information Complexity of Disj. Disjointness: is 𝑋∩𝑌=∅ ? Disj 𝑋,𝑌 = 𝑖=1 𝑛 𝑋 𝑖 ∧ 𝑌 𝑖 Strategy: for some “hard distribution” 𝜇, Direct sum: 𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Prove that 𝐼 𝐶 𝜇 And ≥Ω(1).

Hard Distribution for Disjointness For each coordinate 𝑖∈ 𝑛 : 𝑋 𝑖 =0 𝑋 𝑖 =1 𝑌 𝑖 =0 1/3 1/3 𝑌 𝑖 =1 1/3

𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Let Π be a protocol for Disj on 𝑋,𝑌∈ 0,1 𝑛 Construct Π ′ for And as follows: Alice and Bob get inputs 𝑈,𝑉∈ 0,1 Choose a random coordinate 𝑖∈ 𝑛 , set 𝑋 𝑖 =𝑈, 𝑌 𝑖 =𝑉 Sample 𝑋 −𝑖 , 𝑌 −𝑖 and run Π For each 𝑗≠𝑖, 𝑋 𝑗 ∧ 𝑌 𝑗 =0 ⇒ Disj 𝑋,𝑌 =And 𝑋 𝑖 , 𝑌 𝑖 𝑋 𝑌 𝑈 𝑉

𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Let Π be a protocol for Disj on 𝑋,𝑌∈ 0,1 𝑛 Construct Π ′ for And as follows: Alice and Bob get inputs 𝑈,𝑉∈ 0,1 Choose a random coordinate 𝑖∈ 𝑛 , set 𝑋 𝑖 =𝑈, 𝑌 𝑖 =𝑉 Bad idea: publicly sample 𝑋 −𝑖 , 𝑌 −𝑖 𝑋 Suppose in Π, Alice sends 𝑋 1 ⊕…⊕ 𝑋 𝑛 . In Π, Bob learns one bit ⇒ in Π ′ he should learn 1/𝑛 bit But if 𝑋 −𝑖 is public Bob learns 1 bit about 𝑈! 𝑈 𝑌 𝑉

𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Let Π be a protocol for Disj on 𝑋,𝑌∈ 0,1 𝑛 Construct Π ′ for And as follows: Alice and Bob get inputs 𝑈,𝑉∈ 0,1 Choose a random coordinate 𝑖∈ 𝑛 , set 𝑋 𝑖 =𝑈, 𝑌 𝑖 =𝑉 Another bad idea: publicly sample 𝑋 −𝑖 , Bob privately samples 𝑌 −𝑖 given 𝑋 −𝑖 But the players can’t sample 𝑋 −𝑖 , 𝑌 −𝑖 independently…

𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Let Π be a protocol for Disj on 𝑋,𝑌∈ 0,1 𝑛 Construct Π ′ for And as follows: Alice and Bob get inputs 𝑈,𝑉∈ 0,1 Choose a random coordinate 𝑖∈ 𝑛 , set 𝑋 𝑖 =𝑈, 𝑌 𝑖 =𝑉 Publicly sample 𝑋 1 ,…, 𝑋 𝑖−1 Privately sample 𝑋 (𝑖+1) ,…, 𝑋 𝑛 𝑋 𝑈 Privately sample 𝑌 1 ,…, 𝑌 𝑖−1 Publicly sample 𝑌 𝑖+1 ,…, 𝑌 𝑛 𝑌 𝑉

Direct Sum Theorem Transcript of Π ′ =𝑖, 𝑋 <𝑖 , 𝑌 >𝑖 ,Π Need to show: 𝐼 𝜇 Π ′ ;𝑉 𝑈 + 𝐼 𝜇 Π ′ ;𝑈 𝑉 ≤ 𝐼 𝜇 𝑛 Π;𝑌 𝑋 + 𝐼 𝜇 𝑛 Π;𝑋 𝑌 /𝑛 𝐼 𝜇 Π ′ ;𝑉 𝑈 = 𝐼 𝜇 𝑛 𝑖, 𝑋 <𝑖 , 𝑌 >𝑖 ,Π; 𝑌 𝑖 𝑋 𝑖 = 𝐼 𝜇 𝑛 Π; 𝑌 𝑖 𝑋 ≤𝑖 , 𝑌 >𝑖 ,𝑖 + 𝐼 𝜇 𝑛 𝑖, 𝑋 <𝑖 , 𝑌 >𝑖 ; 𝑌 𝑖 𝑋 𝑖 ≤𝐼 Π, 𝑋 >𝑖 ; 𝑌 𝑖 𝑋 ≤𝑖 , 𝑌 >𝑖 ,𝑖 =𝐼 𝑋 >𝑖 ; 𝑌 𝑖 𝑋 ≤𝑖 , 𝑌 >𝑖 ,𝑖 +𝐼 Π; 𝑌 𝑖 𝑋, 𝑌 >𝑖 ,𝑖 =(1/𝑛) 𝑖=1 𝑛 𝐼 Π; 𝑌 𝑖 𝑋, 𝑌 >𝑖 =𝐼(Π;𝑌│𝑋)/𝑛.

Information Complexity of Disj. Disjointness: is 𝑋∩𝑌=∅ ? Disj 𝑋,𝑌 = 𝑖=1 𝑛 𝑋 𝑖 ∧ 𝑌 𝑖 Strategy: for some “hard distribution” 𝜇, Direct sum: 𝐼 𝐶 𝜇 𝑛 Disj ≥𝑛⋅𝐼 𝐶 𝜇 (And) Prove that 𝐼 𝐶 𝜇 And ≥Ω(1). 

Hardness of And 𝐼𝐶 And =𝐼 Π;𝑌 𝑋 + 𝐼 Π;𝑋 𝑌 ≥Ω 1 2 3 𝐼 Π;𝑌 𝑋=0 + 1 3 𝐼 Π;𝑌 𝑋=1 + 2 3 𝐼 Π;𝑋 𝑌=0 + 1 3 𝐼 Π;𝑋 𝑌=1 11 01 00 10 1/3 transcript on 11 should be “very different” =0 =0

Hellinger Distance ℎ 2 𝑃,𝑄 =1− 𝜔 𝑃 𝜔 𝑄 𝜔 Examples: ℎ 2 𝑃,𝑄 =1− 𝜔 𝑃 𝜔 𝑄 𝜔 Examples: ℎ 2 𝑃,𝑃 =1− 𝜔 𝑃 𝜔 2 =1− 𝜔 𝑃 𝜔 =0 If 𝑃,𝑄 have disjoint support, ℎ 𝑃,𝑄 =1

Hellinger Distance Hellinger distance is a metric ℎ 𝑃,𝑄 ≥0, with equality iff 𝑃=𝑄 ℎ 𝑃,𝑄 =ℎ 𝑄,𝑃 Triangle inequality: ℎ 𝑃,𝑄 ≤ℎ 𝑃,𝑅 +ℎ 𝑅,𝑄 𝑃 𝑄 𝑅

Hellinger Distance If for some 𝜔 we have 𝑃 𝜔 −𝑄 𝜔 =𝛿, then ℎ 2 𝑃,𝑄 ≥ 𝛿 2 11 01 00 10 ℎ≥ 2 3 2

Hellinger Distance vs. Mutual Info Let 𝑃 0 , 𝑃 1 be two distributions Select 𝑍 by choosing 𝐽∼Bernoulli 1 2 , then drawing 𝑍∼ 𝑃 𝐽 Then 𝐼 𝑍;𝐽 ≥ ℎ 2 𝑃 0 , 𝑃 1 11 01 00 10 1/3 𝐼 Π;𝑌 𝑋=0 ≥ ℎ 2 Π 00 , Π 01 𝐼 Π;𝑋 𝑌=0 ≥ ℎ 2 Π 00 , Π 10

Hardness of And 01 11 00 10 1/3 1/3 1/3 Same for Bob until Alice acts differently 1/3 01 ℎ≥ 2 3 2 11 Same for Alice until Bob acts differently 1/3 1/3 00 10

“Cut-n-Paste Lemma” ℎ Π 00 , Π 11 =ℎ Π 01 , Π 10 ℎ Π 00 , Π 11 =ℎ Π 01 , Π 10 Recall: ℎ 2 Π 𝑋𝑌 , Π 𝑋 ′ 𝑌 ′ =1− 𝑡 Π 𝑋𝑌 𝑡 Π 𝑋 ′ 𝑌 ′ 𝑡 Enough to show: we can write Π 𝑋𝑌 𝑡 = 𝑞 𝐴 𝑡,𝑋 ⋅ 𝑞 𝐵 𝑡,𝑌 ℎ 2 Π 00 , Π 11 =1− 𝑡 𝑞 𝐴 𝑡,0 𝑞 𝐵 𝑡,0 𝑞 𝐴 𝑡,1 𝑞 𝐵 𝑡,1 =1− 𝑡 𝑞 𝐴 𝑡,0 𝑞 𝐵 𝑡,1 𝑞 𝐴 𝑡,0 𝑞 𝐵 𝑡,1 = ℎ 2 Π 01 , Π 11

“Cut-n-Paste Lemma” We can write Π 𝑋𝑌 𝑡 = 𝑞 𝐴 𝑡,𝑋 ⋅ 𝑞 𝐵 𝑡,𝑌 Proof: Π 𝑋𝑌 𝑡 = 𝑞 𝐴 𝑡,𝑋 ⋅ 𝑞 𝐵 𝑡,𝑌 Proof: Π induces a distribution on “partial transcripts” of each length 𝑘: Π 𝑋𝑌 𝑘 𝑡 = probability that first 𝑘 bits are 𝑡 By induction: Π 𝑋𝑌 𝑘 𝑡 = 𝑞 𝐴 𝑘 𝑡,𝑋 ⋅ 𝑞 𝐵 𝑘 𝑡,𝑌 Base case: Π 𝑋𝑌 𝑘 𝜖 =1 Set 𝑞 𝐴 𝑘 𝜖,𝑋 = 𝑞 𝐵 𝑘 𝜖,𝑌 =1

“Cut-n-Paste Lemma” Step: Π 𝑋𝑌 𝑘+1 𝑡 = Π 𝑋𝑌 𝑘 𝑡 ≤𝑘 ⋅ Pr next bit= 𝑡 𝑘+1 Suppose after 𝑡 ≤𝑘 it is Alice’s turn to speak What Alice says depends on: Her input Her private randomness The transcript so far, 𝑡 ≤𝑘 So Pr next bit= 𝑡 𝑘+1 =𝑓 𝑡 ≤𝑘 ,𝑋, 𝑡 𝑘+1 =𝑓 𝑡,𝑋 Set 𝑞 𝐴 𝑘+1 𝑡,𝑋 = 𝑞 𝐴 𝑘 𝑡 ≤𝑘 ,𝑋 ⋅𝑓 𝑡,𝑋 , 𝑞 𝐵 𝑘+1 𝑡,𝑋 = 𝑞 𝐵 𝑘 𝑡 ≤𝑘 ,𝑋

Hardness of And 1/3 𝐼𝐶 And =𝐼 Π;𝑌 𝑋 + 𝐼 Π;𝑋 𝑌 = 2 3 𝐼 Π;𝑌 𝑋=0 + 2 3 𝐼 Π;𝑋 𝑌=0 ≥const⋅ ℎ 2 Π 00 , Π 01 + ℎ 2 Π 00 , Π 10 ≥const′⋅ ℎ Π 00 , Π 01 +ℎ Π 00 , Π 10 2 ≥const′⋅ ℎ 2 Π 01 , Π 10 =const′⋅ ℎ 2 Π 00 , Π 11 ≥Ω 1 01 ℎ≥ 2 3 2 11 1/3 1/3 00 10

Multi-Player Communication Complexity

The Coordinator Model 𝑓 𝑋 1 ,…, 𝑋 𝑘 = ? 𝑘 sites 𝑛 bits 𝑋 1 𝑋 2 𝑋 𝑘 …

Multi-Party Set Disjointness Input: 𝑋 1 ,…, 𝑋 𝑘 ⊆ 𝑛 Output: is ⋂ 𝑋 𝑖 =∅? Braverman,Ellen,O.,Pitassi,Vaikuntanathan’13: lower bound of Ω 𝑛𝑘 bits

Reduction from Disj to graph connectivity Given 𝑋 1 ,…, 𝑋 𝑘 , we want to Choose vertices 𝑉 Design inputs 𝐸 1 ,…, 𝐸 𝑘 such that 𝐺 𝑉, 𝐸 1 ∪…∪ 𝐸 𝑘 𝑉, 𝐸 1 ∪…∪ 𝐸 𝑘 is connected iff ⋂ 𝑋 𝑖 =∅

Reduction from Disj to graph connectivity 𝑝 1 𝑝 2 𝑝 𝑘 (Players) (Elements) 1 2 3 4 5 6 𝑋 𝑖 𝑛 ∖ ⋃ 𝑋 𝑖 input graph connected ⇔ ⋃ 𝑋 𝑖 ≠ 𝑛 ⋂ 𝑋 𝑖 ≠∅

Other Stuff Distributed computing

Other Stuff Compressing down to information cost Number-on-forehead lower bounds Open questions in communication complexity