2006 Annual Research Review & Executive Forum Costing Secure System COSECMO Data Mining Danni Wu Edward Colbert {danwu, ecolbert}@cse.usc.edu 2006 Annual Research Review & Executive Forum USC Center for Software Engineering © 2006 USC-CSE 18 September 2018 © 2006 USC-CSE

2006 Annual Research Review & Executive Forum Costing Secure System Goal Of Presentation Review data mining of COCOMO & COCOMO II data sets Published Security Targets How affects COCOMO II for development of secure software systems (“COSECMO”) MetaH provides semantics & supporting tools UML provides graphic front-end © 2006 USC-CSE 18 September 2018 © 2006 USC-CSE

Outline Hidden Security Requirements in COCOMO data sets? Security Assurance & Functional Requirements Relations © 2006 USC-CSE 18 September 2018

Mining COCOMO Data Sets Questions: Do any of projects in COCOMO data set have security requirements? Nobody asked when collected Do we have any data that might support COSECMO Behavior analysis Calibration validation We have data Document size for COCOMO 81 projects Range of test data size (DATA) for COCOMO II © 2006 USC-CSE 18 September 2018

Security, Reliability, & Document Size in COCOMO 81 E – A /A Percentage estimation error of effort Doesn’t match expectation COCOMO will significantly under-estimate projects with RELY driver rating High or Very High 1 – Very Low, 2 – Low, 3 – Nominal, 4 – High, 5 – Very High * Based on COCOMO ‘81 data set (50 projects) © 2006 USC-CSE 18 September 2018

Security, Reliability, & Document Size in COCOMO 81 (cont.) 2006 Annual Research Review & Executive Forum Costing Secure System Security, Reliability, & Document Size in COCOMO 81 (cont.) PP/TKDSI: Pages of documentation per 1000 source instructions Documentation increases when RELY rating goes up Expected 3 project show “excessive” documentation compared to others with same RELY rating Variance increase with RELY rating Is discrepancy result of security or safety requirements? 1 – Very Low, 2 – Low, 3 – Nominal, 4 – High, 5 – Very High * Based on COCOMO ‘81 data set (50 projects) © 2006 USC-CSE 18 September 2018 © 2006 USC-CSE

Security, Reliability, & Document Size in COCOMO 81 (cont.) COCOMO 81 under estimates effort of the 3 projects Is documentation size factor? Project RELY PP/KDSI E-A / A (%) 1 High 291 -14 2 194 -18 3 Very High 241 -33 © 2006 USC-CSE 18 September 2018

No DOCU Driver in COCOMO 81 Security, Reliability, & Document Size in COCOMO 81 (cont.) What about DOCU Driver in COCOMO 2000? No DOCU Driver in COCOMO 81 In COCOMO 2000, only asks if size “is/is not excessive for lifecycle need” High reliability, security, & safety assurance routinely needs more documentation So Extra documentation not excessive DOCU would be set to nominal © 2006 USC-CSE 18 September 2018

Security, Reliability, & Document Size in COCOMO 81 (cont Security, Reliability, & Document Size in COCOMO 81 (cont.) RELY & DOCU in COCOMO 2000* No clear trend indicates that DOCU rating increases when RELY rating increases 1 – Very Low, 2 – Low, 3 – Nominal, 4 – High, 5 – Very High * Based on COCOMO 2000 data set (161 projects) © 2006 USC-CSE 18 September 2018

New security driver will have to address Security, Reliability, & Document Size in COCOMO 81 (cont.) Observations Some high-reliability projects in COCOMO ’81 set are under estimated Possibly due to unaccounted effect of security or safety requirement Reliability requirements accounted for At least in theory DOCU driver doesn’t help Only looks at size “is/is not excessive for lifecycle need” New security driver will have to address © 2006 USC-CSE 18 September 2018

Security Target File Analysis Goals Discover relationship between EAL & number of SFRs Size estimation by security objectives 256 files from NIST website http://www.commoncriteriaportal.org/public/consumer/index.php?menu=4 Domains include Access control devices & systems Boundary protection devices & systems Database Management Systems Operating System © 2006 USC-CSE 18 September 2018

Are Security Assurance & Functional Requirements Really Independent? Common Criteria v2 treats as independent Do more SFR’s always indicate higher EAL requirement? © 2006 USC-CSE 18 September 2018

Mapping EAL with SFRs Data from 256 ST files in NIST website Range of SFR: [3, 107] Range of EAL: [1, 7] © 2006 USC-CSE 18 September 2018

Estimating Size by Security Objectives Current COSECMO: SFR classes for size estimation Security objectives are more intuitive to developer in early phase of SW life-cycle © 2006 USC-CSE 18 September 2018

CC SFR Classes Correlations Red => High correlation between two classes © 2006 USC-CSE 18 September 2018

SFR Classes Correlations (cont.) FDP (Data Protection) & FCS (Cryptographic Support) Most relations cannot be discovered by this matrix 3 or more SFR classes may be needed to achieve 1 security objective © 2006 USC-CSE 18 September 2018

Mapping Security Objectives with SFRs * (X) indicates the corresponding SFR class is optional for a particular security objective © 2006 USC-CSE 18 September 2018

How to Choose SFRs Step 1: Define project domain Step 2: List main security objectives Step 3: Selecting SFR classes based on security objectives Step 4: Identify SFR’s that support selected SFR classes © 2006 USC-CSE 18 September 2018

Case Study: Firewall Domain: Boundary Protection Devices & Systems Primary security objectives: Authentication Accountability Intrusion Detection & Response Selected SFR classes: Authentication – FDP, FIA, FMT, FPT, FTP Accountability – FAU, FDP, FMT, FPT, FTP Intrusion Detection & Response – FAU, FMT, FPT FAU FDP FIA FMT FPT FTP © 2006 USC-CSE 18 September 2018

Regression Analysis Result Response = TOTALSFR Terms = (FAU FDP FIA FMT FPT FTP) Coefficient Estimates Label Estimate Std. Error t-value p-value Constant -0.134506 0.938609 -0.143 0.8868 FAU 0.989551 0.135819 7.286 0.0000 FDP 0.959841 0.161468 5.944 0.0000 FIA 1.03107 0.118332 8.713 0.0000 FMT 0.950064 0.0784849 12.105 0.0000 FPT 1.37014 0.170744 8.025 0.0000 FTP 2.47284 0.721565 3.427 0.0014 R Squared: 0.970174 Sigma hat: 1.68954 Number of cases: 47 © 2006 USC-CSE 18 September 2018

Case Study: Database Domain: Database Primary security objectives: Authentication Accountability Availability Integrity Recoverability Selected SFR classes: Authentication – FDP, FIA, FMT, FPT Accountability – FAU, FDP, FMT, FPT Availability – FMT, FPT, FRU Integrity – FDP, FMT, FPT Recoverability - FDP, FMT, FPT, FTA FAU FDP FIA FMT FTA FPT FRU © 2006 USC-CSE 18 September 2018

Observation from ST File Analysis Total 11 ST files in the domain Seven SFR classes are used in Database domain FAU, FDP, FIA, FMT, FPT, FRU, FTA Availability: Achieved mainly by FRU class FRU_RSA.1 Resource Allocation © 2006 USC-CSE 18 September 2018

Conclusions Potential data points for COSECMO could exist in current COCOMO data set DOCU does not help in cost estimation for secure software Trend exists that EAL increases when SFR increases Prove mapping between SFR classes & security objectives By ST files analysis © 2006 USC-CSE 18 September 2018

Future Work Further analysis on other COCOMO drivers using COCOMO 2000 data set DOCU driver may not be the only one DATA? TOOL? Size estimation using security objectives Collect expert opinions Run Delphi Get size data from ST files’ vendors © 2006 USC-CSE 18 September 2018

Appendix: Common Criteria SFR classes FAU – Security Audit FCO – Communication FCS – Cryptographic Support FDP – User Data Protection FIA – Identification and Authentication FMT – Security Management FPR – Privacy FPT – Protection of the TSF (Trusted Security Function) FRU – Resource Utilization FTA – TOE Access FTP – Trusted Path/Channels © 2006 USC-CSE 18 September 2018