1. Factors Impacting the Effort Required to Fix Security Vulnerabilities
Lotfi ben Othmane, Golriz Chehrazi, Eric Bodden, Petar Tsalovski, Achim Bruker, Philip Miseldine
09 September 2015

2. Vulnerabilities Fixing Process at SAP
Fixing Processes
Released software  SAP security response process
Under development software  Fixing process for security testing
Participants include
Central security teams
IMS maintenance organization
Security experts
Developers ….

3. Introduction - The Problem
Goal: Predict the time to spend on analyzing and fixing a given vulnerability?
Let t = f (x1, ……xj)
What are x1……xj ?
SAP collects data about fixing security vulnerabilities. However, the data does not capture important issues such as team setup.
This work collects information about how really security fixes are done through interviews of the experts. The results are the result of outside observers about the practices at SAP AG—they are not our own.

4. Introduction - Motivation
Cost of implementing security fixes
Vulnerabilities
Average fix time (min)
Dead Code (unused methods)
2.6
Poor logging: system output stream
2.9
XSS (stored)
9.6
Lack of authorization check
6.9
Unsafe threading
8.5
Null dereference
10.2
SQL injection
97.5
Cornell, RSA 2012
The only factor considered is vulnerability type What about the others?

5. The Study - Scope
Goal: Identify the factors that impact the fixing time
Method: Interview participants in the vulnerability fixing process
Result: The major factors that impact the fixing time

6. The Study - Conduct of the Study
Interviews were conducted from 8 to 12 Dec. 2014
Number of participants 12 (12 hours)
9 from Germany and 3 from India
Security experts, developers, coordinators, project leaders
NetWeaver experts, custom application experts, application experts
Selection of participants
Preparation of the questions
Interviews
Transcribe the interviews
Code the interviews
Consolidate the data
Analyze the results

7. The Study - Conduct of the Study – Cont.
Selection of participants
Preparation of the questions
Each interview is transcribed into about 16 pages
Identified 21 code classes from 3 sample interviews
Coded each transcript in a report of 4 pages
Each interviewee is asked to review the report of his interview
Interviews
Transcribe the interviews
Code the interviews
Consolidate the data
Analyze the results

8. The Study - Conduct of the Study - cont.
Coding examples
“Code injections are difficult to fix”
vulnerability type
Vulnerability characteristics
“If the function module is the same in all these 12 or 20 releases […] , then I just have to do one correction”
Similarity of code in the different releases
Software structure

9. Factors that Impact the Vulnerability Fix time
Factor categories
# of factors
Freq.
Vulnerabilities characteristics 6 9
Software structure 19 10
Technology diversification 3 5
Communication and collaboration 7 8
Availability and quality of information and documentation
Experience and knowledge 12 11
Code analysis tool 4
Other

10. Observed Fixing Process
Case 1: Analysis and design of global solution
Implemen-tation
Test
Release
Pre-analysis
Case 2: Analysis and design area solution
This is the actual process. It maps the simplified process from the interviews, from the reality and not from the theoretical process in documents
This is how the process is implemented in practice and observed by external analyst
We found three cases for fixing issues
Case 1: Require design of a global solution
Impacts several products
May better be addressed using API change
Addressed by several teams and involves the central security team
Case 2: Require design of a solution specific to a group of products
Apply to a specific development area—e.g., mobile applications area
The security expert of the area designs the solution with the developers
Case 3: Local solution
Apply to a specific vulnerability instance
The developer fixes the issue with potential help from colleagues
Case 3: Analysis and design local solution
Implemen-tation
Test
Release
Iterations among successive steps are performed implicitly / not marked

11. Take-Away
Vulnerability type is one among many factors (65) that impact the vulnerability fix time
The 8 factor categories reflect the main areas for improving the vulnerability fixing processes
E.g., software structure, training, etc.

12. Threats to Validity
Control of the threats to the validity of the results
The interviewees are diversified
2 researchers coded each interview and the results are consolidated
The participants validated the reports of their interviews
Weaknesses
Used one method to identify the factors—interviewing experts
Interviewed only 2 developers
External use
Diversity of product areas
Distribution of development teams

13. Lessons learned
The main interview questions shall help the interviewees to tell their own stories
“What” questions are inefficient to enumerate elements
The participants sometimes have their own messages to deliver
Vulnerability fixing processes are as many as the process participants
Do not try to base the fix effort estimate on “a process”

14. Thank you lotfi.ben.othmane@sit.fraunhofer.de