Presentation on theme: "Guidelines for the application of Data Envelopment Analysis to assess evolving software Alexander Chatzigeorgiou University of Macedonia Thessaloniki,"— Presentation transcript:
Guidelines for the application of Data Envelopment Analysis to assess evolving software Alexander Chatzigeorgiou University of Macedonia Thessaloniki, Greece 6th International Workshop on Foundations and Techniques for OSS Certification
Added value of DEA improving quality degrading quality improving quality
Added value of DEA (2) DEA offers a unified view by combining selected design metrics and by normalizing over inputs DEA is particularly useful when the trends of individual metrics are highly convoluted
Case Study 2 Project: JDeodorant (20 revisions, 4 years of development) Evaluation of: changeability (CBO, LCOM*, DIT **) ** Samoladas I, Gousios G, Spinellis D, Stamelos I. The SQO-OSS quality model: Measurement based open source software evaluation. 4th International Conference on Open Source Systems (OSS'2008).
Case Study 2 is the evolution of DEA scores depicting the truth ? If a metrics combination is successful in capturing several aspects associated with the same property (e.g. changeability), then it should be demonstrated that the combined value is highly correlated to the desired property. However, quality sub-characteristics are not directly quantifiable; that is why a number of metrics are required to assess them. To confront this problem, we employed secondary measures, which are related to some extent to the property of interest.
Case Study 2 - JDeodorant is the evolution of DEA scores depicting the truth ?
Case Study 3 Project: Industrial Telecom System (12 versions, 30 years of development) Evaluation of: quality in terms of the number of faults
Case Study 3 is the evolution of DEA scores depicting the truth ? there is no single property that can be related to the number of design and programming faults, the engineers suggested to assess the validity of the extracted scores by means of the number of faults having a priority beyond a certain level (trouble reports with priority A or B).
Case Study 3 - Telecom is the evolution of DEA scores depicting the truth ?
Super-efficiency Efficiency v.2 > 1
Super-efficiency (2) Telecom When the basic DEA model leads to multiple versions with score equal to one, the super-efficiency DEA model should be used instead.
Selection of outputs DEA Efficiency Score versions fan-in, fan-out, cohesion Output variables can have varying impact on the extracted DEA scores. When not sure about the necessity to consider a metric, experimentation could highlight whether its impact is significant or not.
No Inputs Inputs can be neglected by providing a constant input for all DMUs
Conclusions Main benefits from the application of DEA: the ability to provide a unified view of several metrics the normalization of the derived scores over size properties of the examined systems Eventually, DEA leads to an efficiency score for each examined project enabling the ranking of various versions and the analysis of the evolution.
Thank you for your attention 6th International Workshop on Foundations and Techniques for OSS Certification
How DEA works (math…) The efficiency score of each version (DMU), is given by maximizing the weighted ratio of all outputs over all inputs, subject to the constraints that the efficiency scores for each DMU must be less than or equal to unity: subject to