1. Computer Science 1 Systematic Structural Testing of Firewall Policies JeeHyun Hwang 1, Tao Xie 1, Fei Chen 2, and Alex Liu 2 North Carolina State University 1 Michigan State University 2 (SRDS 2008)

2. Automated Software Engineering Research Group 2 Motivation Firewall filters packets between a private network and the outside Internet –Correct firewall behavior is dependent on firewall policies and their implementations –Misconfiguration in firewall policies leads to security problems Security holes in Firewall?

3. Automated Software Engineering Research Group 3 Problem Factors for misconfiguration –Conflicts among rules –Rule-set complexity –Mistakes in handling corner cases Systematic testing of firewall policies –Exhaustive testing is impractical –Considering test effort and their effectiveness together –Complementing firewall verification How to test Firewall?

4. Automated Software Engineering Research Group 4 Outline Motivation Problem Firewall Testing Approach –Structural Coverage –Test Packet Generation –Mutation Testing Seeding a small fault in a policy and checking if our test packets detect the fault Assessing the quality of test packets Experimental Results Conclusion.

5. Automated Software Engineering Research Group 5 Firewall Testing Actual Decisions Test Packets Firewall Test Generation – Generate test packets Test Execution – Evaluate the test packets against a firewall and capture their actual decisions Expected Decision s Test Results Evaluation – Check if the decisions are consistent with our expected decisions – If decisions are not consistent, faults cannot be revealed

6. Automated Software Engineering Research Group 6 Firewall Policy Structure A Policy is expressed as a set of rules. A Rule is represented as → is a set of RuleSrcSPortDestDPortProtocolDecision r1r1 **192.168.*.***accept r2r2 1.2.*.****TCPdiscard r3r3 ***** predicat e decisio n is “accept” or “discard” Given a packet (Src, Sport, Dest, Dport, Protocol) –When is evaluated “True”, is returned –Src, Sport, Dest, Dport, Protocol –Representing Integer range Given a packet (Src, Sport, Dest, Dport, Protocol) – can be evaluated “True” or “False”.

7. Automated Software Engineering Research Group 7 Packet Evaluation Example RuleSrcSPortDestDPortProtocolDecision r1r1 **192.168.*.***accept r2r2 1.2.*.****TCPdiscard r3r3 ***** Packet P 1 (1.2.2.0, 1, 192.16.0.0, 100, TCP) –“accept” is returned T T T TT Packet P 2 (1.2.2.0, 1, 192.16.0.0, 100, TCP) –r 1 is not applicable –“discard” is returned First matching rule 193.0.0.0 F TTTTT

8. Automated Software Engineering Research Group 8 Structural Coverage Definition Rationale: when the policy part containing a fault is not evaluated (i.e., “covered”), the fault is not exposed. Rule coverage of a policy P by packets T #rules evaluated by at least one packet in T #rules in P –In practice, packets can cover rules except invalid rules, which are always overridden by other rules

9. Automated Software Engineering Research Group 9 Structural Coverage Definition (Contd...) Predicate coverage of a policy P by packets T #predicates evaluated to true or false by T at least once 2 ×#predicates in P –Each predicate to be true and false Similarly, clause coverage of a policy P by packets T #clauses evaluated to true or false by T at least once 2 ×#clauses in P –Each clause to be true and false

10. Automated Software Engineering Research Group 10 Random Packet Generation Given domain range (e.g., IP addresses [0, 2 8 -1]), random packets are generated within the domain. SrcSPortDestDPortProtocol Domain***** 162.168.12.55192.168.0.510TCP Easy to generate packets Due to its randomness, difficult to achieve high structural coverage

11. Automated Software Engineering Research Group 11 Packet Generation based on Local Constraint Solving Considering an individual rule, generates packets to evaluate constraints of clauses in a specified way –For example, every value is evaluated to true 162.168.12.55192.168.0.510TCP –For example, Dest field value is evaluated to false, and the remaining values are evaluated to true RuleSrcSPortDestDPortProtocolDecision r1r1 **192.168.*.***accept TTTTTTTTTF 162.168.12.55168.1.0.510TCP Conflicts among rules

12. Automated Software Engineering Research Group 12 Packet Generation based on Global Constraint Solving Considering preceding rules are not applicable, generates packets to evaluate constraints of certain rule’s clauses in a specified way –Packet is applicable to r 3 (considering that r 1 and r 2 are not applicable) 162.168.12.551.5.0.510TCP TTTTT Resolving conflicts among rules and require analysis time to solving such conflicts RuleSrcSPortDestDPortProtocolDecision r1r1 **192.168.*.***accept r2r2 1.2.*.****TCPdiscard r3r3 ***** F F

13. Automated Software Engineering Research Group 13 Mutation Testing Why mutation testing? –Measure the quality of a test packet set (i.e., fault detection capability) Seed a fault into a firewall policy and generate a mutant (a faulty version). Decisions Test Packets Mutant (faulty version) Expected Decision s Firewall (correct version) Compare their decisions –The fault is detected in a mutant (i.e., the mutant is “killed”).

14. Automated Software Engineering Research Group 14 11 Mutation Operators Remove RuleRMR Change Rule DecisionCRD Change Rule OrderCRO Change Range End point OperatorCREO Change Range Start point OperatorCRSO Change Range End point ValueCREV Change Range Start point ValueCRSV Rule Clause FalseRCF Rule Clause TrueRCT Rule Predicate FalseRPF Rule Predicate TrueRPT DescriptionOperator

15. Automated Software Engineering Research Group 15 Mutation Examples RuleSrcSPortDestDPortProtocolDecision r1r1 **192.168.*.***accept RPT (Rule Predicate True) r1r1 *****accept CRD (Change Rule Decision) r1r1 **192.168.*.***discard Example CREV (Change Range End point Operator) r1r1 **192.168.0.***accept

16. Automated Software Engineering Research Group 16 Fault Detection Example accept RuleSrcSPortDestDPortProtocolDecision r1r1 **192.168.*.***accept r2r2 1.2.*.****TCPdiscard r3r3 ***** Evaluating a packet p (which is only applicable to r 2 ) –“accept” versus “discard” Given a policy P (assuming correct!!) –Mutant M: CRD (Change Rule Decision) on r 2

17. Automated Software Engineering Research Group 17 Experiment Given a firewall policy (assuming correct!) –Mutants –Packet sets (for each technique) Investigating the following correlations –Packet sets and their achieved structural coverage –Structural coverage criteria and fault-detection capability –Packet sets and their reduced packet sets in terms of fault-detection capability Characteristics of each mutation operator

18. Automated Software Engineering Research Group 18 Experiment (Contd...) Notations –Rand : packet set generated by the random packet generation technique –Local : packet set generated by the packet generation technique based on local constraint solving –Global : packet set generated by the packet generation technique based on global constraint solving –R-Rand, R-Local, and R-Global are their reduced packet sets

19. Automated Software Engineering Research Group 19 Subjects We used 14 firewall policies Number of test packets : approximately 2 packets per rule # Rules : Number of rules # Mutants : Number of mutants Gen time (milliseconds) : packet generation time (particularly for Global) Global : global constraint solving

20. Automated Software Engineering Research Group 20 Measuring Rule Coverage Rand < Local ≤ Global Rand achieves the lowest rule coverage In general, Global achieves slightly higher rule coverage than Local

21. Automated Software Engineering Research Group 21 Reducing the number of packet sets Reduced packet set (e.g., R-Rand) –Maintain same level of structural coverage –R-Rand (5% of Rand), R-Local (66% of Local), and R-Global (60% of Global) –Compare their fault-detection capabilities

22. Automated Software Engineering Research Group 22 Fault detection capability by subject policies R-Rand ≤ Rand < R-Local ≤ Local < R-Global≤ Global Packet set with higher structural coverage has higher fault-detection capability

23. Automated Software Engineering Research Group 23 Fault detection capability by mutation operators Mutant killing ratios vary by mutation operators –Above 85% : RPT –30% - 40% : RPF, RMR –10 – 20% : CRSV, CRSO –0% - 10% : RCT, RCF, CREV, CREO, CRO

24. Automated Software Engineering Research Group 24 Related Work Testing of XACML access control policies [Martin et al. ICICS 2006, WWW 2007] Specification-based testing of firewalls [J¨urjens et al. PSI 2001] –State transition model between firewall and its surrounding network Defining policy criteria identified by interactions between rules [El-Atawy et al. Policy 2007]

25. Automated Software Engineering Research Group 25 Conclusion Firewall policy testing helps improve our confidence of firewall policy correctness Systematic testing of firewall policies –Structural coverage criteria –Three automated packet generation techniques Measured Coverage: Rand < Local ≤ Global Mutation testing to show the fault detection capability –Generally, a packet set with higher structural coverage has higher fault-detection capability –Worthwhile to generate test packet set for achieving high structural coverage

26. Automated Software Engineering Research Group 26 Questions?