1. Ozan Raşit Yürüm, Özden Özcan-Top, Prof. Dr. Onur Demirörs
Software Management Research Group
Middle East Technical University
Assessing Software Processes over a New Generic Software Process Assessment Tool
Ozan Raşit Yürüm, Özden Özcan-Top, Prof. Dr. Onur Demirörs
IWSM - MENSURA 2015

2. Outline Introduction Literature Review The Tool: GSPA
Motivation
Scope of the Study
Literature Review
The Tool: GSPA
Meta-model
Functions
Application of GSPA
Case Study Design
Case Study Conduct
Results & Discussion
Conclusion
Here is my outline,
As you can see, I divided my presentation into six main parts.
In the first part, I will introduce you motivation behind this study and the scope of this study.
In the second part, I will mention about literature review related with software process assessment models.
Then, in the third part, I will explain the generic software process assessment tool with its meta-model and functions.
Then, I will continue with application of GSPA. In this part, I will explain how we designed and conducted multiple case study
After that, I will yield findings for each research question.
Finally, I will summarize the results in conclusion part.
Software Management Research Group

3. Motivation Softare Process Assessment Models
Various type of process assessment
Software Process Assessment
Too much time
Countless and complicated evidences and findings
Not easy to comprehend and analyze the outputs
Software Process Assessment Tools
Not meet the expected features
Let’s start with the motivation part.
Software organizations usually perform process assessment based on well-known process assessment frameworks such as CMMI and ISO/IEC day. In addition to these models, a number of process assessment models such as Automotive SPICE, Enterprise SPICE, Brazilian Software Improvement, and Agile Maturity Model are derived from these models and they are being updated.
However, the process assessment based on all of these models including cmmi and iso15504 takes too much time because of the fact that findings and evidences might be countless and complicated. it is not easy to comprehend and analyze the outputs emerging from the assessment while performing paper based assessment.
Therefore, software process assessment tools have been valued in order to increase especially the impact of self-assessment.
However, there is no existing tool which meets expected features such as defining model and performing basic process assessment function.
Software Management Research Group

4. Scope of the Study In this study,
we developed a generic software process assessment tool (GSPA) and analyzed GSPA with a multiple case study to answer the following research questions:
RQ 1: To what extent is the tool sufficient in meeting expected features?
RQ 2: What are the advantages of an automated generic software process assessment tool?
RQ 3: What are the weaknesses of the proposed tool?
Therefore, in this study we developed a generic software process assessment tool to support various type of process assessment models and analyzed the tool with a multiple case study to answer the following research questions:
Software Management Research Group

5. Software Process Assessment Tools
In the literature review part, I will mention about our previous case study which we conducted out in order to compare the tools in terms of meeting expected features. We found these process assessment tools and compare them.
Software Management Research Group

6. Software Process Assessment Tools
Comparison Criteria
Suitability for defining new model
Suitability for performing assessment
Reporting automatically
Guiding assessor
Evaluation of different projects
Suitability for parallel assessment
Suitability for discovery of tool features
In order to compare them, these seven comparison criteria were determined by taking opinions of process assessment experts. That is, an ideal tool has to have these features which are
Suitability for defining new model:
In order for a software process assessment tool to be flexible, it requires to be used for any kind of process assessment model. This is possible when a tool enables to define new process assessment model and to perform process assessment based on new defined model. Allowing deleting, editing, adding elements of new process assessment model and sorting the elements in a certain order are expected from a software process assessment tool.
Suitability for performing assessment: The necessary elements of process assessment model require can be rated, findings can be entered and evidences can be stored in a certain order in a fully functional software process assessment tool.
Reporting automatically: A software process assessment tool has to meet the minimum requirements
Guiding assessor: Self-assessment can be performed by people who are not experts. Namely, the guidance of software process assessment according to process assessment model for assessors will facilitate the feasibility of the tool.
Evaluation of different projects: Process assessment in organizations is performed over multiple projects which are selected via certain sample methods in order to represent the whole organization. Inasmuch as, a software process assessment tool has to allow to evaluate multiple projects in parallel order or in sequence and to amalgamate the results of different project assessments reasonably.
Suitability for parallel assessment: Process assessment in organizations can be performed by multiple teams, as well as it can be performed by a single team. Thus, a software process assessment tool has to enable multiple teams to evaluate multiple projects simultaneously.
Suitability for discovery of tool features: In terms of ease of use, tool features such as starting process assessment, reporting, saving, editing settings of tool has to be found and understood easily by users. In short, all features that a tool claims to provide with have to be presented clearly to the users. Furthermore, a tool has to provide help to facilitate the discovery of tool features and the visual design has to be user-friendly.
Software Management Research Group

7. Software Process Assessment Tools
Here are the results of our previous case study. As you see, Appraisal Assisstan got best result from our previous case study. However,
when these software process assessment tools are examined as a whole, no tool has the expected features completely.
When accessible software process assessment tools [20][21][22][27][28][30] from those listed in Table 1 were compared, Appraisal Assistant [27] got the best result with its reporting automatically, and supporting different projects properties and func-tions [24]. Although the Appraisal Assistant [27] got the highest score in terms of suitability for defining a new model, we observed challenges related to association of generic goals with capability dimension, organization of concept elements and changing the number of capability and maturity levels in the functionality of “defin-ing a process assessment model” in the tool. In addition, it does not meet the expecta-tion of parallel process assessment capability. Besides it is subject to judgement in terms of user friendly interfaces. Among other software process assessment tools, the CMM-Quest v1.3 [21] and SPICE 1-2-1[20] were also rated “fully achieved” in terms of guiding assessors, automated reporting functions, and user friendly interfaces. On the other hand, the Appraisal Wizard [28] and the Model Wizard [30] are only suita-ble for basic assessment functions such as ratings of goals and practices, and adding evidences. When these software process assessment tools were examined as a whole, no tool meets the expectations of the features completely.
Software Management Research Group

8. The Meta-Model of GSPA
Helps various process assessment model to be supported within a single process assessment tool.
Allows compensation of weak sides and emphasis of powerful sides.
Integration of two most known process assessment models
CMMI
ISO/IEC 15504
Now, I will explain our tool, GSPA.
Since a meta-model helps various process assessment model to be supported within a single process assessment tool, we constructed a meta-model by integrating two most known process assessment models which are CMMI and ISO/IEC It also allows compensation of weak sides and emphasis of powerful sides of models.
Software Management Research Group

9. The Meta-Model of GSPA 4 steps to create meta-model CMMI class diagram
ISO/IEC class diagram
Integration of CMMI and ISO/IEC 15504
Adding rating framework and Categorization
We followed 4 steps to create meta-model.
Software Management Research Group

10. Step 1: CMMI Class Diagram
Firstly, we drew class diagram of CMMI.
Software Management Research Group

11. Step 2: ISO/IEC 15504 Class Diagram
Secondly, we drew the class diagram of ISO 15504
Software Management Research Group

12. Step 3: Integration of CMMI and ISO/IEC 15504
Mapping
We mapped CMMI and ISO and renamed the concepts for our meta-model in order to integrate CMMI and ISO 15504
Software Management Research Group

13. Step 3: Integration of CMMI and ISO/IEC 15504
Here, is the integration of CMMI and ISO/IEC 15504
Software Management Research Group

14. Step 4: Adding rating framework and categorization classes
Finally, we added rating framewok and categorization classes.
Software Management Research Group

15. Functions of the GSPA
We also drew use case diagram to show functions of GSPA. There are two main functions which are to create process assessment model and to perform process assessment. We extended these functions in use case diagram.
Software Management Research Group

16. Case Study Design Model Selection Assessor Selection Case Selection
Two most common models and one model based on these common models with different concepts
Assessor Selection
Either At least one year experience or complete Software Quality Management Course with grade ‘AA’
Case Selection
Including evidences, ratings and comments or notes
Detailed report
At least two different projects
Purposiveness and availability
Process assessment via GSPA
Case study design consists of 4 steps: As a first step, process assessment models were determined. In the second step assessors were chosen according to their experience with the selected models. In the third step, cases were actually assessment examples which were performed manually before. They were selected by assessors. The examples are supposed to include evidences, ratings, comments and detailed report. Finally, assessment were performed via GSPA
Software Management Research Group

17. Case Study Conduct Step 1: Model Selection
While conducting case study, for the first step, we found these process assessment models as a result of literature review. CMMI and ISO were chosen since most of the models are derived from them. And Agility Assessment Model were also chosen since its structure is same with these models but concepts are different.
Software Management Research Group

18. Case Study Conduct Step 2: Assessor Selection Assessor 1 (ISO 15504)
Master Student
Software Quality Management Course with grade ‘AA’
Assessor 2 (CMMI)
Working intensively about process assessment related with CMMI and ISO for six years
Assessor 3 (Agility Assessment Model)
three year-experience on CMMI assessment
one of the creators of Agility Assessment Model
As a second step, 3 assessors which are familiar with the related model were chosen.
Software Management Research Group

19. Case Study Conduct Step 3: Case Selection Case for CMMI
Defense Industry
55 Employee
Case for ISO/IEC 15504
CMMI Level 3
100 Employee
Case for Agility Assessment Model
Government Organization
60 Employee
After assessor selection, one case is selected for each process assessment model by as. The case based on CMMI was performed in an organization working for defense industry with 55 employees. The other case related with ISO/IEC was performed in an organization having CMMI Level 3 certificate with 100 employees. The last one based on Agility Assessment Model was performed in a government organization developing web based applications and having 60 employees.
Software Management Research Group

20. Case Study Conduct Step 4: Automated Assessment with GSPA Tool
Final step is to perform process assessment with GSPA Tool. The assessors followed these steps. Now, I want to show you a short video in order to clarify you about what i mean with these steps.
Software Management Research Group

21. Data Collection Interview Observation and Think Aloud Structured
One question at a time
Observation and Think Aloud
Note taking
How do we collect data?
Data for each case study were obtained by making interview with each assessor about the tool. In addition, during assessment, observations and think aloud process were implemented so that the results can be more accurate.
Software Management Research Group

22. Data Analysis Content Analysis Coding Data Generating Themes
Suitability for defining new model
Suitability for performing assessment
Reporting automatically
Guiding assessor
Evaluation of different projects
Suitability for parallel assessment
Suitability for discovery of tool features
Advantages of the tool
Weaknesses of the tool
Matching Codes with Themes
Interpreting Data
For the data analysis, we used content analysis. There are four main steps in content analysis. Firstly, we code the finding and generate 9 themes. These are determined according to research questions. Then, we matched codes with themes. After that, we interpreted data to find answer research questions.
Software Management Research Group

23. Results & Discussion
RQ1: To what extent is the tool sufficient in meeting expected features?
Feature
Agility Assessment Model
ISO 15504
CMMI
Suitability for defining new model
Fully Achieved
Suitability for performing assessment
Largely Achieved
Reporting Automatically
Guiding Assessor
Evaluation of different projects
Suitability for parallel assessment
Not Achieved
Suitability for discovery of tool features
Here are the results related with research question one. During interview, the assessors were also asked for rating the features.
As it is seen from Table, all features except for the suitability of the tool for parallel assessment are rated as “Fully Achieved” by at least two assessors.
Especially, the main purpose of this study which is to create different process assessment models was met. This shows that our meta-model works enough.
In addition, the functionality of the tool is measured with the features which are suitable for performing basic assessment functions, reporting automatically, and evaluation of different projects. The results, related with these features, show that the tool supports all kind of process assessment models during assessment in terms of functionality.
Furthermore, guiding assessor during the assessment and suitability of the tool for discovery of tool features are very important in terms of usability. These results were also satisfactory
The results of this study show that the tool meets expected features almost completely for all features expect for parallel assessment.
Software Management Research Group

24. Results & Discussion
RQ 2: What are the advantages of an automated generic process assessment tool?
GSPA’s features allow definition of new model components which were derived from either ISO/EC or CMMI.
The internal consistency of derived process assessment models can be validated with this tool by matching each indicator such as practice or work product with outcome or achievement.
The compatibility of a derived process assessment model to ISO/IEC or CMMI can be identified by comparing the model’s concepts with the meta-model.
Since the information about structured process assessment model are shown systematically, assessors do not lose time within the pages of technical report by defining process assessment model.
The reporting feature helps assessor gain percent of their time during process assessment
Software Management Research Group

25. Results & Discussion
RQ 3: What are the weaknesses of the proposed tool?
Extra elements such as fallacy cannot be defined as desired with the tool.
Explanations and descriptions about determining level satisfaction point, and merging projects are not satisfactory enough.
There is no error control mechanism in the tool yet.
It does not support parallel assessment yet.
Software Management Research Group

26. Conclusion
GSPA;
fulfills the requirements of the 6 of 7 expected features of a software process assessment tool satisfactorily
has many advantages for process assessment in terms of gaining time, creating process assessment model, validating process assessment model, performing process assessment based on different process assessment models.
has weaknesses related with definition of extra concepts, understandibility of explanations and descriptions.
Software Management Research Group

27. Thank You  www.smrg.ii.metu.edu.tr Software Management Research Group