1. Calibrating Noise to Sensitivity in Private Data Analysis
Kobbi Nissim
BGU
With Cynthia Dwork, Frank McSherry, Adam Smith, Enav Weinreb

2. The Setting San x1 x2 x3  xn-1 xn Users x  Dn query X = answer
Can I combine these to learn some private info?
I just want to learn a few harmless global statistics xn
xn-1  x3 x2 x1
Users
(government, researchers,
marketers, …)
query
San
X =
answer
x  Dn
(n rows each of domain D)

3. What is privacy? Clearly we cannot undo the harm done by others
Can we minimize the additional harm while providing utility?
Goal: Whether or not I contribute my data does not affect my privacy

4. Output Perturbation San x1 x2 x3  xn-1 xn f f(x) + noise ¢
random coins
f(x) + noise
San Controls:
which functions f
kind of perturbation

5. When Can I Release f(x) accurately?
Intuition: global information is “insensitive” to individual data and is safe
f(x1,…,xn) is sensitive if changing a few entries can drastically change its value

6. Talk Outline
A framework for output perturbation based on “sensitivity”
Formalize “sensitivity” and relate it to privacy definitions
Examples of sensitivity based analysis
New ideas
Basic models for privacy
Local vs. global
Noninteractive vs. Interactive

7. Related Work
Relevant work in Statistics, Data mining, Computer Security, Databases
Largely: no precise definitions and analysis of privacy
Recently: A foundational approach
[DN03,EGS03,DN04,BDMN05, KMN05 CDMSW05,CDMT05,MS06,CM06,…]
This work extends [DN03,DN04,BDMN05]

8. Privacy as Indistinguishabilityxn
xn-1  x3 x2 x1 x=
query 1
answer 1
query T
answer T 
transcript
T(x)
San
random coins
Differ in 1 row
Distributions at “distance” < 
x’= xn
xn-1  x3
x2’ x1
query 1
answer 1
query T
answer T 
transcript
T(x')
San
random coins

9. -Indistinguishability
A sanitizer is -indistinguishable if
for all pairs x,x’  Dn which differ on at most one entry
for all adversaries A
for all transcripts t
Pr[TA(x) = t]
 e 
Pr[TA(x’) = t]

10. Semantically Flavored Definitions
Indistinguishability - easy to work with but does not directly say what the adversary can do an learn
“Ideal” semantic definition:
Adversary does not change his beliefs about me
Problem: dependencies, e.g. in form of side information
Say you know that I am 20 pounds heavier than average Israeli…
You will learn my weight from census results
Whether or not I participate
Ways to get around:
Assume “independence” of X1,…,Xn [DN03,DN04,BDMN05]
Compare “what A knows now” vs “what A would have learned anyway” [DM]

11. Incremental Risk Suppose adversary has prior “beliefs” about x
Probability distribution, r.v. X= (X1,…,Xn)
Given transcript t, adversary updates “beliefs” according to Bayes’ rule
New distribution Xi’| T(X)=t

12. It’s the same whether you participate or not
Incremental Risk
Bugger!
It’s the same whether you participate or not
“Proof:” indistinguishability guarantees that updates are the same within 1± 
Two options:
I participate in census (input = X)
I do not participate (input Yi = X1,…,Xi-1,*,Xi+1,…,Xn )
Privacy: whether I participate or not does not significantly influence adversary’s posterior beliefs:
For all transcripts t, for all i: X’i |T(X)=t ¼ X’i |T(Yi)=t
San Yi
San X

13. Recall – -Indistinguishability
For all pairs x,x’  Dn s.t. dist(x,x’) = 1
For all transcripts t
Pr[TA(x) = t]
 e 
Pr[TA(x’) = t]

14. An Example – Sum Queries
Pls let me know fA(x)=iA xi xn
xn-1  x3 x2 x1
San
random coins
fA(x) + noise
x [0,1]n

15. Sum Queries – Answering a Query
x 2 [0,1]n fA(x)=iA xi
Can be used as a basis for other tasks: clustering, learning, classification… [BDMN05]
Answer:
xi + Y where Y » Lap(1/)
Laplace Distribution:
h(y)  e-|y|
Note: |fA(x)-fA(x’)|  1

16. Sum Queries – Proof of -Indistinguishability
Property of Lap
 x,y: h(x)/h(y)  e|x-y|
Pr[T(x)=t]  e|fA(x)-t|
Pr[T(x’)=t]  e|fA(x’)-t|
Pr[T(x)=t] / Pr[T(x’)=t]  e | fA(x)- fA(x’)|  e
f(x’)
f(x)
max |fA(x)-fA(x’)| = 1

17. Sensitivity We chose noise magnitude to cover for max |f(x)-f(x’)|
Sf = max ||f(x)-f(x’)||1
Local Sensitivity
LSf(x) = max ||f(x)-f(x’)||1 f xn
xn-1  x3 x2 x1
San x=
x’=
x2’
f(x) + noise
f(x’) + noise
dist(x,x’)=1
dist(x,x’)=1

18. Calibrating Noise to Sensitivity
Pls let me know f (x) xn
xn-1  x3 x2 x1
San
random coins
f (x) + Lap(Sf /)
x Dn
h(y)  e-/Sf ||y||1

19. Calibrating Noise to Sensitivity - Why it Works?
Sf = max |f(x)-f(x’)|1
Property of Lap:  x,y: h(x)/h(y)  e||x-y||1
Pr[T(x)=t] / Pr[T(x’)=t]  e / Sf ||fA(x)- fA(x’)||1  e
h(y)  e-/Sf ||y||1
dist(x,x’)=1

20. Main Result
Theorem: If a user U is limited to T adaptive queries of sensitivity Sf
then -indistinguishability if iid noise Lap(SfT/) added to query answers
Same idea works with other metrics and noise
Which useful functions are insensitive?
All useful functions should be insensitive…
Statistical conclusions should not depend on small variations in data

21. Using insensitive functions
Strategies:
Use theorem, output f(x) + Lap(Sf /)
Sf may be hard to analyze/compute
Sf high for functions considered ‘insensitive’
Express f in terms of insensitive functions
Resulting noise depends on input (in form and magnitude)

22. Example - Expressing f in terms of insensitive functions
x  {0,1}n f(x) = ( xi)2
Sf = n2 - (n-1)2 = 2n-1
af = ( xi)2 + Lap(2n/)
If f(x) << n noise dominates
However
f(x) = (g(x))2 where g(x) =  xi
Sg=1
Better to query for g
Get ag = xi + Lap(1/)
Estimate f(x) as (ag)2
Taking  constant results in stddev O( xi)
– (1/ )2

23. Useful Insensitive functions
Means, variances,…
With appropriate assumptions on data
Histograms & contingency tables
Singular value decomposition
Distance to a property
Functions with low query complexity

24. Histograms/Contingency Tables
x1,…,xn 2 D where D partitioned into d disjoint bins b1,…,bd
h(x) = (v1,…,vd) where vi=|{i : xi  bi}|
Sh = 2
Changing one value xi changes vector by · 2
Irrespective of d
Add Laplacian with std. dev. 2/ to each count
Can do that with sum queries …
b1 b2 … b4

25. Distance to a Property x P Say P = set of “good” databases
Distance to P = min # points in x that must be changed to make x in P
Always has sensitivity 1
Add Laplacian with stdev 1/
Examples:
Distance to being clusterable
Weight of minimum cut in graph P x
distance to P

26. Approximations with Low Query Complexity
Lemma:
Assume algorithm A that randomly samples n points and Pr[ A(x)  f(x) ± ] > (1+)/2
Then Sf · 2
Proof:
Consider x,x’ that differ on point i
Let Ai be A conditioned on not choosing point i
Pr[Ai(x)  f(x)±  | pt i not sampled] > 1/2
Pr[Ai(x’)  f(x’)± | pt i not sampled] > 1/2
 point p that is within dist  from both f(x), f(x’)  Sf · 2
Support of Ai(x)=Ai(x) p

27. Local Sensitivity
Median – typically insensitive, large (global) sensitivity
LSf(x) = max ||f(x)-f(x’)||1
Example:
f(x) = min(xi, 10) where xi{0,1}
LSf(x) = 1 if xi  10 and 0 otherwise
dist(x,x’)=1 10 n
 xi

28. Local Sensitivity – First Attempt
Calibrate noise to LSf(x)
Answer query f by f(x) + Lap(LSf(x)/)
If x1…x10=1 and x11…xn=0
Answer = 10 + Lap(1/)
If x1…x11=1 and x12…xn=0
Answer = 10
Noise magnitude may be disclosive! 10 n
 xi

29. How to Calibrate Noise to Local Sensitivity?
Noise magnitude at a point x depends on LS(y) for all y  Dn
N*f = max (LSf(y) e- dist(x,y))
Median 10 n
 xi

30. Talk Outline
A framework for output perturbation based on “sensitivity”
Formalize “sensitivity” and relate it to privacy definitions
Examples of sensitivity based analysis
New ideas
Basic models for privacy
Local vs. global
Noninteractive vs. Interactive

31. Models for Data Privacy
Collection
and
sanitization
You
Bob
Alice
Users
(government, researchers,
marketers, …)

32. Models for Data Privacy – Local vs. Global
Collection
and
sanitization
You
Bob
Alice
San
San
Collection
and
sanitization
You
Bob
Alice
Including
“SFE”

33. Models for Data Privacy – Interactive vs. Noninteractive
You
Bob
Alice
Collection
and
sanitization
You
Bob
Alice
Collection
and
sanitization

34. Models for Data Privacy - Summary
Local (vs. Global)
Non central trusted party
Individuals interact directly with (untrusted) user
Individuals control their own privacy
Noninteractive (vs. Interactive)
Easier distribution: web site, book, CD, …
More secure: can erase the data once it is processed
Almost all work in statistics, data mining is noninteractive!
OLD NOTES!
This talk is about database privacy. The term can mean many things but for this talk, the example to keep in mind is a government census. Individuals provide information to a trusted government agency, which processes the information and makes some sanitized version of it available for public use.
- privacy is required by law
- ethical
- pragmatic: people won’t answer unless they trust you
There are two goals: we want users to be able to extract global statistics about the population being studied. However, for legal, ethical and pragmatic reasons, we also want to protect the privacy of the individuals who participate.
And so we have a fundamental tradeoff between privacy on one hadn and utility on the other. The extremes are easy: publishing nothing at all provides complet eprivacy, but no utility, and publishing the raw data exactly provides the most utility but no privacy.
Thus the first-order goal of this paper is to plot some middle course between the extremes; that is, to find a compromise which allows users to obtain useful information while also providing a meaningful guarantee of privacy.
This problem is not new: it is often called the "statistical database" problem. I would say a second-order goal of this paper is to change the way the problem is approached and treated in the literature…
Graphically, this is what is going on. As I said, there are two goals, utility and privacy.
Utility is easy to understand, and to explain to a user. To prove that your scheme provides a particular utility, just give an algoriithm and an analysis.
Privacy is much harder to get a handle on…

35. ? Four Basic Models Global, interactive incomparable Local,
noninteractive
Global,
interactive
incomparable ?

36. Interactive vs. Noninteractive
Local,
noninteractive
Global,
interactive

37. Separating Interactive from Noninteractive
Random samples: can compute estimates for many stats
(essentially) no need to decide upon queries ahead of time
But not private (unless small domain, small sample [CM06])
Interaction: get the power of random samples
With privacy!
E.g. Sum queries f(x) = i fi(xi)
Even chosen adaptively!
Noninteractive schemes seem weaker
Intuition: privacy  cannot answer all questions ahead of time (e.g. [DN03])
Intuition: sanitization must be tailored to specific functions

38. Separating Interactive from Noninteractive
Theorem: If D={0,1}d, then for any private, noninteractive scheme, many sum queries cannot be learned, unless d = o(log n)
Weaker than Interactive
Cannot emulate random sample if data is complex

39. Local vs. Global
Local,
noninteractive
Global,
interactive

40. Separating Local from Global
D = {0,1}d for d = (log n)
View x as an nd matrix
Local: rank(x) has sensitivity 1, can release with low noise
Global: cannot distinguish whether rank(x) = k or much larger than k
For suitable choice of d,n,k

41. To sum up Defined privacy in terms of indistinguishability
Considered semantic versions of definitions
“Crypto” with non-negligible error
How to Calibrate noise to sensitivity and # of queries
Seems that useful stats should be insensitive
Some commonly used functions have low sensitivity
For others – local sensitivity?
Begun to explore the relationships between basic models

42. Questions Which useful functions are insensitive?
What would you like to compute?
Can we get stronger results using:
Local sensitivity?
Computational assumptions? [MS06]
Entropy in data?
How to deal with small databases?
Privacy in a broader context
Rationalizing privacy and privacy related decisions
Which types of privacy? How to decide upon privacy parameters? …
Handling rich data
Audio, Video, Pictures, Text, …