1. Algorithmic Transparency & Quantitative Influence
Yair Zick
National University of Singapore
Joint work with Amit Datta, Anupam Datta, Ariel D. Procaccia and Shayak Sen

2. Black-Box Decision Makers are Everywhere
Health
Insurance discounts for data
data-driven insurance policies
Education
course assignments
Streaming and course allocation
Finance
Determining loan eligibility
Credit Scores
Media
Personalized Ads
News & Social Media

3. Explaining Black-Box Decision Makers
Stakeholders have a right to explanation
… but algorithms are often “black boxes”:
intellectual property
inherently complex

4. Quantitative Feature Influence (Datta, Datta, Proccacia and Zick, IJCAI 2015; Datta, Sen and Zick, IEEE S&P 2016)
We are given a dataset (a set of labelled data points)
How important was feature 𝑖 in the decision for
An individual?
A group (e.g. ethnic minorities, gender groups)?
In general?
Numerical measures of feature importance in black-box settings: algorithmic transparency

5. Algorithmic Transparency
Privacy
Fairness
Integrity

6. Personalized Transparency Report
An individual was deemed worthy of arrest
Birth Year
1984
Drug History
None
Smoking History
Census Region
West
Race
Black
Gender
Male
Evidence of racial discrimination
Illustrates the dangers of the black-box use of machine learning

7. Axiomatic Feature Influence (Datta, Datta, Procaccia and Zick, IJCAI 2015)
How should an influence measure behave?
What properties do we want our measure to have?
A common approach in game theory: provably fair revenue division methods.
Influence as a “resource” that needs to be divided.
There exist unique influence measures satisfying certain natural properties.

8. Quantitative Input Influence (QII) (Datta, Sen and Zick, IEEE S&P 2016)
Deals with correlated inputs
a causal measure
Breaks correlations via randomized interventions.
Supports a general class of transparency queries
parametric in a quantity of interest
From individual to group influence
Computes joint and marginal influence
uses game theoretic aggregation measures

9. QII for Individual Outcomes
𝑈 𝑖
Inputs
𝑋∼ 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑖 ,…, 𝑥 𝑛
Classifier
Quantity Measured
Pr 𝑐 𝑋 =1 𝑋=𝒙]
𝑋 −𝑖 𝑈 𝑖 ∼ 𝑥 1 , 𝑥 2 ,…, 𝑢 𝑖 ,…, 𝑥 𝑛
Pr 𝑐 𝑋 −𝑖 𝑈 𝑖 =1 𝑋=𝒙] 𝑐 𝑐
Hint at problem later
Consider a classifier that takes as input a vector of features x1 .. xn, and outputs 0 or 1.
I want to measure the influence of input i on the classification of some individual x
Randomized intervention on feature 𝑖 breaks correlations: replace 𝑥 𝑖 with an independent random sample 𝑢 𝑖 .

10. Quantity of Interest Pr 𝑐 𝑋 =𝑐( 𝒙 0 ) 𝑋= 𝒙 0 ] Pr 𝑐 𝑋 =1 𝑋 is female]
Outcome of an individual
Pr 𝑐 𝑋 =1 𝑋 is female]
Outcomes of a group of individuals
Pr 𝑐 𝑋 =1 𝑋 is male] − Pr 𝑐 𝑋 =1 𝑋 is female]
Disparity between group outcomes
Talk about disparate impact.

11. QII: Definition
The Quantitative Input Influence (QII) of an input is the difference in a quantity of interest, when the input is replaced with a random value via an intervention.
QII is defined on a quantity of interest
𝜄 𝒬 𝑖 =𝒬 𝑋 −𝒬( 𝑋 −𝑖 𝑈 𝑖 )

12. Joint and Marginal Influence
Classifier
Decision
𝐴𝑐𝑐𝑡𝑠≥3
𝐴𝑔𝑒≤50
𝐼𝑛𝑐𝑜𝑚𝑒≥50𝑘
The influence of individual inputs may not be significant in itself.
For young, low income individual, age and income alone don’t influence the outcome.
However, jointly age and income can significantly influence outcome
Intersting things happen because of interactions between different features
Interventions on individual inputs commonly have no effect!

13. Randomized intervention on a set of features
Resampled features in 𝑆
Set QII
Inputs
𝑋∼ 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑖 ,…, 𝑥 𝑛
Classifier
Quantity Measured
𝒬(𝑋)
𝑋 −𝑆 𝑈 𝑆 ∼ 𝑥 1 , 𝑢 2 ,…, 𝑢 𝑖 ,…, 𝑢 𝑛
𝒬 𝑋 −𝑆 𝑈 𝑆 𝑐 𝑐
Hint at problem later
Consider a classifier that takes as input a vector of features x1 .. xn, and outputs 0 or 1.
I want to measure the influence of input i on the classification of some individual x
Randomized intervention on a set of features
𝜄 𝑆 =𝒬 𝑋 −𝒬( 𝑋 −𝑆 𝑈 𝑆 )

14. The marginal importance of 𝑖
𝑚 𝑖 𝑆 =𝜄 𝑆∪𝑖 −𝜄 𝑆 =𝒬 𝑋 −𝑆 𝑈 𝑆 −𝒬( 𝑋 −𝑆∪ 𝑖 𝑈 𝑆∪{𝑖} )
“How much impact does 𝑖 have given that we have already intervened on 𝑆?”
Measure average marginal influence of 𝑖
How should we aggregate 𝑚 𝑖 (𝑆)?
Well studied in economics!
The Shapley value, the Banzhaf index…
𝜑 𝑖 = 𝑆⊆𝑁 𝑛 𝑆 𝑚 𝑖 𝑆
The only value to satisfy certain desirable properties!

15. Personalized Transparency Report
An individual was deemed worthy of arrest
Birth Year
1984
Drug History
None
Smoking History
Census Region
West
Race
Black
Gender
Male
Evidence of racial discrimination
Illustrates the dangers of the black-box use of machine learning

16. Summary General transparency queries Correlated inputs
Parametric in a quantity of interest
Robust to broad range of queries
Correlated inputs
Define a causal measure
Average Marginal Influence
Use game-theoretic notions of average marginal influence
Axiomatically justified measures
Additional Results
Most measures can be efficiently approximated
QII measures can be made differentially private cheaply