1. Keep the Adversary Guessing: Agent Security by Policy Randomization
Praveen Paruchuri
University of Southern California

2. Motivation: The Prediction Game
Police vehicle
Patrols 4 regions
Can you predict the patrol pattern ?
Pattern 1
Pattern 2
Randomization decreases Predictability
Increases Security
Region 1
Region 2
Region 3
Region 4

3. Domains Police patrolling groups of houses
Scheduled activities at airports like security check, refueling etc
Adversary monitors activities
Randomized policies

4. Problem Definition
Problem : Security for agents in uncertain adversarial domains
Assumptions for Agent/agent-team:
Variable information about adversary
Adversary cannot be modeled (Part 1)
Action/payoff structure unavailable
Adversary is partially modeled (Part 2)
Probability distribution over adversaries
Assumptions for Adversary:
Knows agents plan/policy
Exploits the action predictability

5. Outline Security via Randomization No Adversary Model
Partial Adversary Model
Contributions: New,
Efficient Algorithms
Randomization +
Quality Constraints
MDP/Dec-POMDP
Mixed strategies:
Bayesian Stackelberg
Games

6. No Adversary Model: Solution Technique
Intentional policy randomization for security
Information Minimization Game
MDP/POMDP: Sequential decision making under uncertainty
POMDP  Partially Observable Markov Decision Process
Maintain Quality Constraints
Resource constraints (Time, Fuel etc)
Frequency constraints (Likelihood of crime, Property Value)

7. Randomization with quality constraints
Fuel used
<
Threshold

8. No Adversary Model: Contributions
Two main contributions
Single Agent Case:
Nonlinear program: Entropy based metric
Hard to solve (Exponential)
Convert to Linear Program: BRLP (Binary search for randomization)
Multi Agent Case: RDR (Rolling Down Randomization)
Randomized policies for decentralized POMDPs

9. MDP based single agent case
MDP is tuple < S, A, P, R >
S – Set of states
A – Set of actions
P – Transition function
R – Reward function
Basic terms used :
x(s,a) : Expected times action a is taken in state s
Policy (as function of MDP flows) :

10. Entropy : Measure of randomness
Randomness or information content quantified using Entropy ( Shannon 1948 )
Entropy for MDP -
Additive Entropy – Add entropies of each state
Weighted Entropy – Weigh each state by it contribution to total flow

11. Randomized Policy Generation
Non-linear Program: Max entropy, Reward above threshold
Exponential Algorithm
Linearize: Obtain Poly-time Algorithm
BRLP (Binary Search for Randomization LP)
Entropy as function of flows

12. BRLP: Efficient Randomized Policy
Inputs: and target reward
can be any high entropy policy (uniform policy)
LP for BRLP
Entropy control with

13. BRLP in Action Increasing scale of Beta = .5 = 1 = 0 Deterministic
- Max
entropy
= 0
Deterministic
Max Reward
Target
Reward
Increasing scale of

14. Results (Averaged over 10 MDPs)
For a given reward threshold,
Highest entropy : Weighted Entropy : 10% avg gain over BRLP
Fastest : BRLP : 7 fold average speedup over Expected Entropy

15. Multi Agent Case: Problem
Maximize entropy for agent teams subject to reward threshold
For agent team:
Decentralized POMDP framework
No communication between agents
For adversary:
Knows the agents policy
Exploits the action predictability

16. Policy trees : Deterministic vs Randomized
Deterministic Policy Tree A1 A2 O1 O2
Randomized Policy Tree O1 A1 A2 O2

17. RDR : Rolling Down Randomization
Input :
Best ( local or global ) deterministic policy
Percent of reward loss
d parameter – Number of turns each agent gets
Ex: d = .5 => Number of steps = 1/d = 2
Each agent gets one turn ( for 2 agent case )
Single agent MDP problem at each step

18. RDR : d = .5 Agent 1 Fix Agent 2’s policy Maximize joint entropy
Joint Reward > 90%
M = Max Reward
Agent 2
Fix Agent 1’s policy
Maximize joint entropy
Joint reward > 80%
90% of M
80% of M

19. RDR Details To derive single agent MDP:
New Transition, Observation and Belief Update rules needed
Original Belief Update Rule –
New Belief Update Rule –

20. Experimental Results : Reward Threshold vs Weighted Entropy ( Averaged 10 instances )

21. Security with Partial Adversary Modeled
Police agent patrolling a region.
Many adversaries (robbers)
Different motivations, different times and places
Model (Action & Payoff) of each adversary known
Probability distribution known over adversaries
Modeled as Bayesian Stackelberg game

22. Bayesian Game It contains: Set of agents: N (Police and robbers)
A set of types θm (Police and robber types)
Set of strategies σi for each agent i
Probability distribution over types Пj: θj  [0,1]
Utility function: Ui : θ1 * θ2 * σ1 * σ2  R

23. Stackelberg Game Agent as leader
Commits to strategy first: Patrol policy
Adversaries as followers
Optimize against leaders fixed strategy
Observe patrol patterns to leverage information
Adversary
Nash Equilibrium: <a,a> : [2,1]
Leader commits to uniform
random strategy {.5,.5}
Follower plays b: [3.5,1] a b
2,1
4,0
1,0
3,2
Agent

24. Previous work: Conitzer, Sandholm AAAI’05, EC’06
MIP-Nash (AAAI’05): Efficient best Nash procedure
Multiple LPs Method (EC’06): Given normal form game
Finds optimal leader strategy to commit to
Bayesian to Normal Form Game
Harsanyi Transformation: Exponential adversary strategies
NP-hard
For every joint pure
strategy j of adversary:
(R, C: Agent, Adversary)

25. Bayesian Stackelberg Game: Approach
Two Approaches:
Heuristic solution
ASAP: Agent Security via Approximate Policies
Exact Solution
DOBSS: Decomposed Optimal Bayesian Stackelberg Solver
Exponential savings
No Harsanyi Transformation
No exponential # of LP’s
One MILP program (Mixed Integer Linear Program)

26. ASAP vs DOBSS ASAP: Heuristic Control probability of strategy
Discrete probability space
Generates k-uniform policies
k = 3 => Probability = {0, 1/3, 2/3, 1}
Simple and easy to implement
DOBSS: Exact
Modify ASAP Algorithm
Discrete to continuous probability space
Focus of rest of talk

27. DOBSS Details Previous work: Fix adversary (joint) pure strategy
Solve LP to find best agent strategy
My approach:
For each agent mixed strategy
Find adversary best response
Advantages:
Decomposition technique
Given agent strategy
Each adversary can find Best-response independently
Mathematical technique obtains single MILP

28. Obtaining MILP
Decomposing
Substitute

29. Experiments: Domain Patrolling Domain: Security agent and robber
Security agent patrols houses
Ex: Visit house a
Observe house and its neighbor
Plan for patrol length 2
6 or 12 strategies : 3 or 4 houses
Robbers can attack any house
3 possible choices for 3 houses
Reward dependent on house and agent position
Joint space of robbers exponential
strategies: 3 houses, 10 robbers

30. Sample Patrolling Domain: 3 & 4 houses
LPs: 7 followers
DOBSS: 20
4 houses
LP’s: 6 followers
DOBSS: 12

31. Conclusion Agent cannot model adversary
Intentional randomization algorithms for MDP/Dec-POMDP
Agent has partial model of adversary
Efficient MILP solution for Bayesian Stackelberg games

32. Vision Incorporating machine learning Dynamic environments
Resource constrained agents
Constraints might be unknown in advance
Developing real world applications
Police patrolling, Airport security

33. Thank You
Any comments/questions ?