1. Guidelines for the application of Data Envelopment Analysis to assess evolving software Alexander Chatzigeorgiou University of Macedonia Thessaloniki, Greece 6th International Workshop on Foundations and Techniques for OSS Certification

2. Efficiency 1,000,000 € 25,000 € Efficiency = profit #employees

3. Analogy between companies and software v1.0 v1.2 v2.0 vx.y... size properties metrics

4. Software Context CBO LCOM WMC DIT...

5. Software Context (2) version i version i+1 Complexity = 5

6. Inputs The ideal input variable would be a measure of functionality (e.g. FPs) Size metrics can be used as a substitute

7. How DEA works… v1 v2 v3 v4 v5 v6 v7 v8 v.1v.2v.3v.4v.5v.6v.7v.8 NOO (x)130150200230300320400450 fan-in (y 1 )10.611.21313.61515.115.814 cohesion (y 2 ) 912.510.5111210913 v.1v.2v.3v.4v.5v.6v.7v.8 NOO (x) 11111111 fan-in (y 1 ) 0.080.07 0.060.05 0.040.03 cohesion (y 2 ) 0.070.080.05 0.040.030.020.03

8. How DEA works…(2) Efficiency = 0A 0A’ v.4

9. Added value of DEA improving quality degrading quality improving quality

10. Added value of DEA (2) DEA offers a unified view by combining selected design metrics and by normalizing over inputs DEA is particularly useful when the trends of individual metrics are highly convoluted

11. Case Study 2 Project: JDeodorant (20 revisions, 4 years of development) Evaluation of: changeability (CBO, LCOM*, DIT **) ** Samoladas I, Gousios G, Spinellis D, Stamelos I. The SQO-OSS quality model: Measurement based open source software evaluation. 4th International Conference on Open Source Systems (OSS'2008).

12. Case Study 2 is the evolution of DEA scores depicting the truth ? If a metrics combination is successful in capturing several aspects associated with the same property (e.g. changeability), then it should be demonstrated that the combined value is highly correlated to the desired property. However, quality sub-characteristics are not directly quantifiable; that is why a number of metrics are required to assess them. To confront this problem, we employed secondary measures, which are related to some extent to the property of interest.

13. Case Study 2 - JDeodorant is the evolution of DEA scores depicting the truth ?

14. Case Study 3 Project: Industrial Telecom System (12 versions, 30 years of development) Evaluation of: quality in terms of the number of faults

15. Case Study 3 is the evolution of DEA scores depicting the truth ? there is no single property that can be related to the number of design and programming faults, the engineers suggested to assess the validity of the extracted scores by means of the number of faults having a priority beyond a certain level (trouble reports with priority A or B).

16. Case Study 3 - Telecom is the evolution of DEA scores depicting the truth ?

17. Super-efficiency Efficiency v.2 > 1

18. Super-efficiency (2) Telecom When the basic DEA model leads to multiple versions with score equal to one, the super-efficiency DEA model should be used instead.

19. Selection of outputs DEA Efficiency Score versions fan-in, fan-out, cohesion Output variables can have varying impact on the extracted DEA scores. When not sure about the necessity to consider a metric, experimentation could highlight whether its impact is significant or not.

20. No Inputs Inputs can be neglected by providing a constant input for all DMUs

21. Conclusions Main benefits from the application of DEA: the ability to provide a unified view of several metrics the normalization of the derived scores over size properties of the examined systems Eventually, DEA leads to an efficiency score for each examined project enabling the ranking of various versions and the analysis of the evolution.

22. Thank you for your attention 6th International Workshop on Foundations and Techniques for OSS Certification

23. How DEA works (math…) The efficiency score of each version (DMU), is given by maximizing the weighted ratio of all outputs over all inputs, subject to the constraints that the efficiency scores for each DMU must be less than or equal to unity: subject to