1. Predictor of Customer Perceived Software Quality By Haroon Malik

2. Introduction Predict the customer’s experience within first three months Quantifying the relative importance of various processes and product factors on customer experience Deployment issues –Usage patterns, –Software platforms & –Hardware platform.

3. Introduction Come up with a model that can be easily adapted and used at other organization with little or more tailoring.

4. Driving force Techniques already exists to predict how many faults remains in unchanging software system, changes or module will have defect and even how much effort defect repairs will require. Many researches examined the effect of software contents and development process on measure of customer perceived quality.

5. Driving force Most of which ignore the –Hardware configurations, –Software platform, –Usage patterns & –Deployment issues End users experience the software typically experience the quality of the entire “Solution”. Past researches do consider the importance of these factors but no solid work is done to validate the claim or quantify these factors.

6. The Software project Call processing software for AVYA telephony systems –Established product –Seven million lines of code mostly in C and C++ Multiple releases are in field and are being used by tens of thousands of customers. Used by clients whose business depend on the high availability of the product.

7. Capturing Interaction Four database are considered for capturing customer interaction measures –Customer issue tracking system Trouble ticket database –The equipment database –Change management Sablime database Database-1 Database-2

8. The Process Avaya uses a tiered support process. Trouble Ticket Half of the over 4 million tickets created in 2003 are related to product analyzed in this project. Equipment Database –Software release –Number of licensed ports –4 million systems listen in equipment databases.

9. Customer perceived quality models Business activity distraction ---- Negative effect To major aspect of customer perceived quality: –Impact of problem occurrence –Frequency of problem occurrence Interest –Rare high-impact problems Equipment services outages Malfunction resulting in software modification Model

10. Customer perceived quality models –Frequent low impact problems Technician dispatches Customer calls Alarm reports Model

11. Factors to predict Factors –Deployment issues –Usage patterns –Software platform –Hardware configurations Prior work examines –Software product –Development process lack empirical validation They are not good predictors (Static. do not vary for a single release)

12. Predictor Measures ? Value Predictors –System size. –Operating system –Ports –Total Deployment time –Software Upgrades Nuisance Predictors –US of international installation –Service contracts –Missing configuration information

13. System size This predictor measures –Hardware configuration factors –Software platform factors and –Usage patterns factor. Introduced “LARGE” variable indicator in the model Small and medium systems have: –Fewer customer interactions –Few settings to configure –Fewer systems to interface with –Likely not to be involved in business critical application requiring 7x24 uptime. –Less likely to experience and report issues.

14. Operating System Operating system predictor measures : –Software platform factor & –Hardware configuration factor Considered –An open Linux –proprietary –Commercial windows Very small number Os systems used NT/Win2000 Off-the-shelf operating system introduce unnecessary complexity and configuration issue.

15. Ports The port predictor measures –Usage pattern factor & –Hardware configuration factors The number of ports indicates how many licensed end points are supported by the system. Model encoded the log number of ports with log(nPort) variable

16. Total deployment time Deployment time predictor measure the deployment issues Anticipation: fewer customer interaction as the total deployment time increases. I need it on Timeeee!!!!

17. Software upgrades The software upgrades predictor measures the deployment issues factor. Encoded the existence of upgrade using an indictor variable called “Upgr”. Upgrade serves to keep the machine running properly by incorporating the latest fixes and refinements to the system Upgrades have clearly defined purpose of making the system more stable, so expect to have that effect.

18. Nuisance Factors These factors are likely to identify peculiarities of data reporting and collection process, but not necessarily differences in underlying customer perceived quality. –US of International installation –Service contracts –Missing configuration information

19. Factors Vs Predictors Predict for each customer (outputs): –Software defects –System outages –Technician dispatches –Calls –Automated alarms Using Logistic regression and Linear regression Using predictors (inputs): –Total deployment time –Operating system –System size –Ports –Software upgrades For a real world software system

20. Results First fit the models to test the relationships hypothesized previously. Use the models to predict customer interaction for the next major releases. Present full results only for two measures.

21. Software Failure Software Failure Nuisance Factors

22. Most important predictor Total Deployment time: –Customers who installed the application early may have detected malfunctions that are fixed by the time later customers install their systems –The individuals performing the installation and configuration may have acquired more experience, have access to improved documentation –The lesson from this relationship is that customers that are less tolerant of availability issues should not be the first to install a major software release “never upgrade to dot zero release.”

23. Another Important Predictor Operating system (software platform, hardware configurations) –Systems running on the proprietary OS are 3 times less likely to experience a software defect compared with systems on running the open OS (Linux) –Systems running on the commercial OS (Windows) are 3 times more likely to experience a software defect compared with systems running on the open OS (Linux)

24. Customer calls

25. Predicting customer call traffic Ports and NAPorts are not available know from talking to the customer call center that they have estimates of how many calls a call representative can handle so our prediction of the number of calls in a month can be used to plan staffing. The take away is that they the predictions are accurate and can be used to plan various activities

26. Validations Accounted for data reporting differences –Included indicator variables in the models to identify populations (e.g. US or international customers) Independently validated the data collection process –Independently extracted data and performed analyses Interviewed personnel to validate findings –Programmers –Field technicians

27. Best Contributions Identified and quantified characteristics, like time of deployment, that can affect customer perceived quality by more than an order of magnitude created models that can predict various customer interactions and found that predictors have consistent effect across interactions We learned that controlled deployment may be the key for high reliability systems