1. Defining Security Metrics
Security Planning
Susan Lincke

2. SABSA High-Level Framework

3. Gap Analysis
The difference between where you are and where you want to be: (For example:
# malware infections/month
Rate of finding illegal software, hardware
Security awareness training averages

4. SEI/COBIT Level 4 Monitoring: Includes Metrics
Metrics inform management (and independent auditors) of the effectiveness of the security program Monitoring achievement of control objective may be more important than perfecting security procedures
control objective: ‘where we want to be’: this means setting incremental goals.
In our last slide, our control objective was to achieve level 3, (not level 5.)

5. Which metrics to use? Business-Driven Technology-Driven
Addresses specific business risks
Inherent industry risks
Tailored to organization
Measures adherence to control objectives
Addresses recent threats observed by CERT
CERT: Computer Emergency Readiness Team
Addresses recent forensic data

6. Monitoring Function: Business-Driven Metrics
Executive mgmt is interested in
risk, budget, policy.
Review every 6 months-1 year
Metrics can be classified into three areas: Strategic (mgmt-oriented or high-level), tactical (mid-level, tends to be people-oriented), and operational (low-level, tends to be technical).
Technical details:
E.g., firewall, logs, IPS,
vulnerability tests.
Review weekly.
Automate statistics.
Determine effectiveness of
security program: risk changes,
compliance, incident response tests.
Review quarterly to half-year

7. Monitoring Function: Business-Driven Metrics
Project Plan or Budget Metrics
Risk performance
Disaster Recovery Test results
Audit results
Regulatory compliance results
Metrics can be classified into three areas: Strategic (mgmt-oriented or high-level), tactical (mid-level, tends to be people-oriented), and operational (low-level, tends to be technical). This slide shows specific concerns at all three levels.
Vulnerability Scan results
Server config. standards
compliance
IDS monitoring results
Firewall log analysis
Patch mgmt status
Policy compliance metrics
Exceptions to policy/standards
Changes in process or system
affecting risk
Incident management effectiveness

8. Which metrics?
Step 1: What are the most important security areas … threats …. regulation … to monitor in your organization? Step 2: Which metrics make the most sense to collect. Can they be automated? Step 3: Consider the 3 perspectives: strategic, tactical, operational metrics, relative to 3 audiences.

9. Monitoring Function: Metrics
Risk:
The aggregate ALE
% of risk eliminated, mitigated, transferred
# of open risks due to inaction
Cost Effectiveness:
What is:
Cost of workstation security per user
Cost of spam and virus protection per mailbox
Operational Performance
Time to detect and contain incidents
% packages installed without problem
% of systems audited in last quarter
Organizational Awareness:
% of employees passing quiz, after training vs. 3 months later
% of employees taking training
Technical Security Architecture
# of malware identified and neutralized
Types of compromises, by severity & attack type
Attack attempts repelled by control devices
Volume of messages, KB processed by communications control devices
Security Process Monitoring:
Last date and type of BCP, DRP, IRP testing
Last date asset inventories were reviewed & updated
Frequency of executive mgmt review activities compared to planned
Metrics should be selected from the highest-priority risks, and are best if their collection is automated.
Sample metrics: select the ones most important to your organization.

10. Monitoring Function: Metrics cont’d
Security Management Framework:
Completeness and clarity of security documentation
Inclusion of security in each project plan
Rate of issue recurrence
Compliance:
Rate of compliance with regulation or policy
Rate of automation of compliance tests
Frequency of compliance testing
Secure Software Development:
Rate of projects passing compliance audits
Percent of development staff certified in security
Rate of teams reporting code reviews on high-risk code in past 6 months
Incident Response Metrics
# of Reported Incidents
# of Detected Incidents
Average time to respond to incident
Average time to resolve an incident
Total number of incidents successfully resolved
Total damage from reported or detected incidents
Total damage if incidents had not been contained in a timely manner

11. Workbook: Metrics Metrics Selected
What are the most important areas to monitor in your organization?
Cracking Attempt
Lunatic gunman
Major Risks:
FERPA Violation
Web Availability
Category
Metric
Calculation & Collection Method
Period of
Reporting
Strategic
Cost of security/terminal
Information Tech. Group
1 year
Cost of incidents
Incident Response totals
6 months
Tactical
% employees passing FERPA quiz
Annual requesting testing
% employees completing FERPA training
Two annual trainings with sign-in. Performance review
# Hours Web unavailable
Incident Response form
Opera-
tional
# brute force attacks
1 month
# malware infections
The top part shows the major risks for a school: lunatic gunman, FERPA violation, cracking attempt, web availability.
Once we know are risks that we want to monitor for, we can define metrics at each of the strategic, tactical, and operational level. The Calculation and Collection method defines how the statistic will be gathered. The period of reporting is how often we will review the results of the statistics. Normally we review metrics for Operational level more often than Strategic, for example.

12. Technology-Driven Metrics
SANS-Recommended
Critical Controls for Effective Cyber Defense
Technology-Driven Metrics

13. Creating a baseline configuration of network
This is the ‘normal’ network

14. Noticing inappropriate ‘additions’ to the network
New PC
New AP
New wireless

15. Checking the security configuration of network
Patched? Legal software?
Firewall on & security configured? Antivirus on and patched?
Limit USB access?
WPA2 AES,
EAP/
TLS?
Monitor
Network?
Withstands attacks?
SQL, buffer overflow,
cross-site scripting,
clickjacking, …

16. Noticing inappropriate actions
New sys admin or user acct
Transfer of confidential data or illegal
packets
Detect new network service

17. SANS: Critical Controls for Effective Cyber Defense
Typical SANS Metric:
Temporarily install unauthorized software, hardware or configuration on a device. It should be:
found within 24 hours (or best: 2 minutes)
isolated within one hour confirmed by alert/
reported every 24 hours until issue is resolved.

18. SANS Critical Control 1: Inventory of Authorized Devices
Ensure all devices (with IP address) on network are known, configured properly, and patched.
Scan network daily or use DHCP reports or passive monitoring.
Compare results with baseline configuration.
Metric: Temporarily install unauthorized device.
Summary of SANS Critical Controls

19. SANS: Critical Control 2: Inventory of Authorized Software
Ensure all software is approved and recently patched
Whitelist defines the permitted list of software.
Blacklist defines illegal software (e.g., IT tools).
Endpoint Security Suites (ESS) contain antivirus, antispyware, firewall, IDS/IPS, s/w white/blacklisting.
Metric: Temporarily install unauthorized software on a device.
Summary of SANS Critical Controls

20. SANS Critical Control 3: Secure Configurations for Hardware & Software
All devices are hardened using recommended security configurations
Illegal software list exists, includes Telnet, VNC, RDP
New software is quarantined and monitored.
Imaged software is maintained in an updated state.
Build secure images, and use configuration checking tools daily.
Metric: Temporarily attempt to change a set of random configurations.
Summary of SANS Critical Controls

21. SANS Critical Control 4: Continuous Vulnerability Assessment
Run vulnerability scans on all systems at least weekly, preferably daily. Problem fixes are verified through additional scans.
Vulnerability scanning tools (updated) for: wireless, server, endpoint, etc.
Automated patch management tools notify via when all systems have been patched.
Metric: If the scan does not complete in 24 hours, an notification occurs.
Summary of SANS Critical Controls

22. SANS Critical Control 5: Malware Defense
Antivirus/antispyware is always updated
Run against all data: shared files, server data, mobile data.
Additional controls: blocking social media, limiting external devices (USB), using web proxy gateways, network monitoring.
Endpoint security suites report tool is updated and active on all systems
Metric: For install of benign malware (e.g., security/hacking tool), antivirus prevents installation or execution or quarantines software
Sends an alert/ within one hour indicating specific device and owner
Summary of SANS Critical Controls

23. SANS Critical Control 6: Application S/W Security
New application software is tested for security vulnerabilities:
Web vulnerabilities: buffer overflow, SQL injection, cross-site scripting, cross-site request forgery, clickjacking of code, and performance during DDOS attacks.
Input validated for size, type
No system error messages reported directly to user
Automated testing includes static code analyzers and automated web scanning.
Configurations include application firewalls and hardened databases.
Metric: An attack on the software generates a log/ within 24 hours (or less).
Automated web scanning occurs weekly or daily
Summary of SANS Critical Controls

24. SANS Critical Control 7: Wireless Device Control
Wireless access points are securely configured with WPA2 protocol and AES encryption.
Extensible Authentication Protocol-Transport Layer Security (EAP/TLS) provides mutual authentication.
Only registered, security-approved devices are able to connect
Wireless networks are configured for the minimum required radio footprint.
Metrics: Wireless intrusion detection systems detect available wireless access points and deactivate rogue access points within 1 hour
Vulnerability scanners can detect unauthorized wireless access points connected to the Internet.
Summary of SANS Critical Controls

25. SANS Critical Control 8: Data Recovery Capability
Backups are maintained at least weekly and more often for critical data.
Backups are encrypted and securely stored.
Multiple staff can perform backup/recovery.
Metric: Test backups quarterly for a random sample of systems. This includes operating system, software, and data restoration.
Summary of SANS Critical Controls

26. SANS Critical Control 9: Security Skills Assessment
Security awareness training: required for end users, system owners
Security training: necessary for programmers, system, security and network administrators
Metric: Test security awareness understanding
Periodically test social engineering tests via phishing s and phone call
Employees who fail a test must attend a class
Summary of SANS Critical Controls

27. SANS Critical Control 10: Secure Network Configurations
A configuration DB tracks approved configurations in config. mgmt. for network devices: firewalls, routers, switches.
Tools perform rule set sanity checking for Access Control Lists.
Two-factor identification is used for network devices.
Metric: Any change to the configuration of a network device is reported within 24 hours
Summary of SANS Critical Controls

28. SANS Critical Controls
11. Control of Network Ports, Protocols and Services:
Default Deny packets.
Periodically review for restriction
Metric: Measure time to recognize added network service
12. Controlled Administrative Privilege:
Minimal elevated privileges
Passwords are complex, changed periodically, 2-factor
Metric: Measure time to recognize new sys admin
Summary of SANS Critical Controls

29. SANS Critical Controls
13. Boundary Defense:
Use firewall zones to filter incoming and outgoing traffic.
Blacklist & whitelist network addresses
Metric: Measure time to recognize unauthorized packets
14. Analysis of Security Audit Logs:
Server logs are write-only and archived for months.
Firewalls log all allowed and blocked traffic.
Unauthorized access attempts are logged
Metric: Measure time to recognize no log space
Summary of SANS Critical Controls

30. SANS Critical Controls
15. Need to Know Access:
Prevent exfiltration of data (e.g., to competitors)
Classify data
Use restrictive firewall configurations
Log access to confidential data
Metric: Measure time to recognize unauthorized access
16. Account Monitoring and Control:
Terminated accounts -> removed
Expired password/ disabled/ locked out accounts, -> investigated
Failed logins -> lockouts
Inactivity -> locked sessions
Unusual time access -> alert
Data exfiltration recognized by keywords.
Metric: Measure time to recognize new/ changed user accounts
Summary of SANS Critical Controls

31. SANS Critical Controls
17. Data Loss Prevention:
Prevent exfiltration of proprietary or confidential info
Encrypt mobile and USB devices
Disable USB
Metric: Measure time to recognize transfer of confidential data file
18. Incident Response:
Incident Response Plan defines who does what for various conditions
IRP includes contact information for third party contractors

32. SANS Critical Controls
19. Secure Network Engineering:
Separate zones exist: DMZ, middleware, private network
DMZ accessed through proxy firewall
DMZ DNS is in DMZ; internal DNS is in internal zone, …
Emergency config. for restricted network is ready for quick deployment.
20. Penetration Tests:
Penetration tests = vulnerability tests + attacker tests
Red Team exercises test incident response team reactions
Metric: Measure false positive, false negative, true positive rate

33. Question
The difference between where an organization performs and where they intend to perform is known as:
Gap analysis
Quality Control
Performance Measurement
Benchmarking
1 – Gap Analysis

34. Question The MOST important metrics when measuring compliance include:
Metrics most easily automated
Metrics related to intrusion detection
Those recommended by best practices
Metrics measuring conformance to policy
4 – Metrics measuring conformance to policy

35. Question
SANS recommends that an initial maximum allowable time to detect a problem in a network or server configuration is:
Two minutes
One hour
One day
One week
1 – Gap Analysis