1. AILA – Artificial Intelligence Life Cycle Assurance
Jan 2019
© 2019 Cognizant

2. Template for Agenda, use as required
Content
Template for Agenda, use as required
AILA- Overview
QA for AI
AI for QA
| © 2019 Cognizant

3. Where AI fits in QA and QA fits in AI
Traditional App Lifecycle
AI/ML based App Lifecycle 1
Requirements Analysis
AI/ML Model Requirement
AI/ML Model Design & Data Analysis 2
Design 3
Development
Deployment & Maintenance 4
Testing
QA for AI 5
Maintenance
Frictionless lifecycle automation
Model Testing
Model Dev & Training
AI for QA  QI BOTs
| © 2019 Cognizant

4. How is Quality Assurance for AI different
QA for Traditional Apps
QA for AI based Apps
There is no defined Output
Output is defined
Expected behavior/result significantly consistent across dataset
Expected behavior/result significantly varies across dataset
Low functionality drift overtime
High functionality drift overtime
Very high need to regress application
Minimal regression of application
When we talk about testing or QA for traditional apps. Output or expected behavior is pre-defined. While in AI or ML based apps there is no predefined output or expected behavior. You might see low functionality drift and hence minimal regression required for traditional apps. While AI or ML based apps will have high functional drift overtime and hence requires high need to regress the application. In short we have “moving target” to test in AI/ML apps while we have “predefined target” for traditional apps. Thus QA is totally different when it comes to AI/ML apps.
| © 2019 Cognizant

5. Artificial Intelligence Lifecycle Assurance - AILA
AI for QA Quality insight Bots framework
QA for AI Data& model
“AI for QA“
Use AI to test application intelligently
Accelerator to assure Data & Model fitment
Improve Appln Quality with Analytics
QA for AI
Process Framework
QI BOTS studio customization framework
AILA
Enhancing Quality or ML/AI Process
Jumpstart Bots execution platform
“QA for AI”
Assuring Quality of AI/ML models
QA for AI
Performance & Security
iDashboard- Exec. dashboard powered by AI/ML
Assuring AI/ML business stability
Intelligent enterprise level metrics dashboard
AILA has 2 components. AI for QA is to use AI to augment and optimize testing of software applications. And QA for AI is to do testing for AI/ml apps.
AI for QA is “quality insights” and it does use data from SDLC and optimize testing of software applications. It has workbench or studio which can be used to develop/deploy ML bots to augment testing of software apps. It displays the outcome or results in dashboard for end user to consume and act upon insights to improve quality of software apps.
QA for AI has 4 main components. We will see them in detail in next slide. It broadly assures quality of data, model, process and performance/security of AI/ML apps.
| © 2019 Cognizant

6. QA for AI
Assessing AI/ML model from QA Standpoint
| © 2019 Cognizant

7. What is QA for AI? Good question. QA for AI is also a testing process.
I know software testing is a process used to identify the correctness, completeness and quality of developed computer software.
What is QA for AI?
Good question. QA for AI is also a testing process.
It will also identify correctness, completeness, reliability and quality of “data and AI/ML” application.
As you know data is the new “oil or electricity”
And AI/ML process has 2 parts:
Train ML model with data and
Test ML model with data.
And as you can see “data” forms a critical role in AI/ML process.
Hence we need to “validate” data for it’s quality too.
Interesting, why validate data specifically?
| © 2019 Cognizant

8. Provide comprehensive quality assurance for AI/ML solutions
360° QA for AI - Overview
Provide comprehensive quality assurance for AI/ML solutions
1. Assure Data Quality
3. Assess ML Process Quality
Test data/feature for correctness, completeness and quality.
This will remove any bias or insufficient data to train/test the model.
Helps the model to perform well in production as in development.
Assess ML process and frameworks from quality assurance perspective.
Validate whether industry best practices are adapted in AI/ML process.
Ensure the process is lean and efficient too.
4. Assure ML Performance, Security Quality
2. Assure ML Model Quality
Test/evaluate ML model for it’s correctness, completeness and quality.
Ensure best algorithm selected for the problem.
It is providing required outcome for data driven decisions.
Assure ML solutions on non-functional aspects including security and performance.
Ensure that model is trained and deployed within allowable batch window.
It is resilient for expected data volume spikes or poor quality of data.
360° QA for AI
This slide talks about 36 degree view when it comes to QA for AI. We are covering the entire testing life cycle of AI/ML apps.
Assure data quality – The key to any ai/ml app is data. Data fuels and defines an ai/ml app. Hence it is vital to validate data before we move forward with ai/ml app development. This first component ensures that we have all required data needed and also the correct data required. This will identify any bias in data which might influence the outcome of the ai/ml system. This assurance of data ensures that we have right data during development as we might see it in production. If there are any data gaps like missing values, not right data to build the model. Then this phase will highlight the same.
Assume ML model quality – Once we have assured data which is required to build the AI/ML model. The next step is to build the model. It is important to make sure that model performs correctly. For example if the model is to capture anomalies in data. It should not miss out any anomalies and highlight the right data as “anomaly”. It will not serve the business purpose for which the model is built. Hence the results from model has to be validated and verified before moving on to next step. This phase ensures that best algorithm is used and also data driven decisions made using this model fits the business need (validate that true positive and true negative outcome and their importance for the business).
Assess ML process framework quality – Since AI/ML is in it’s nascent stage. We need to make sure that entire process has followed industry standard best practices and also process is lean and efficient. This phase ensures the same.
Assure ML performance, security quality – While we have validated the functional aspects of AI/ML application till now. It is critical that non-functional aspects like ML performance and security are also validated. For example model should be re-trained within allowable batch window and it cannot take days to complete training. And also it should be robust enough to handle any spikes or poor quality of data. Security should be validated to ensure that ML app is not vulnerable.
*ML – Machine Learning
| © 2019 Cognizant

9. 1. Assure Data Quality 01 03 02 04 Correlation Analysis
Validate Data quality to pick and train the model for more accurate predictions 01 03
Correlation Analysis
Data Bias Analysis
Validate and filter potential Data where systems draw improper conclusions about data sets, either because of human intervention or as a result of a lack of cognitive assessment of data.
E.g. Root cause predictor trained only with values related to “coding” will be flawed when provided with other root causes like “data issues”, “environment issues”
Validate that no variable is left out which are correlated.
E.g. “Root cause” is correlated to “priority”, “severity”, “description” in TriageBot
Outliers Analysis 02 04
Trend Analysis
Identify and exclude outlier data from the data set and refine the model
E.g. Out of 100 Defect root causes one or two non standard category such as local hardware issues
Assure all necessary data trends across time windows are duly considered in the data set
E.g. Defect Seasonality prior to holiday seasons in retail industry due to application rollout and holiday freeze
Let us quickly look at the first pillar for QA for AI – assure data quality.
Correlation analysis – The variables or columns of data used to train ML model is called as features. We nee to ensure that we have selected the right features which is going to add more value to the outcome. Hence it is vital to do correlation analysis. It will correlate features to the output label and make sure that right features are selected for the model.
Outlier analysis – Usually data will have outliers and that might become a noise while training ML model and impact the quality of outcome from the model. This step will ensure that outlies are analyzed and removed from data to train the model.
Data bias analysis – Similar to outlier. Bias in data might impact the outcome. For example root cause predictor trained just with values related to “coding” as root cause might impact how well the model performs when it is given data related to other root causes like “data issues”, “environment issues” This step will ensure that we are identifying, removing any such bias in the data.
Trend analysis – In most cases the data might follow certain trend. To relate to real world example. If we have Retail floor planned ML model. It should consider trend of not only “summer data” but also “winter” and other seasons. So that it performs well during other seasons data given as input.
| © 2019 Cognizant

10. 2. Assure ML Model Quality
Validate the ML model built for prediction accuracy and right model selection 03
A/B testing / Accuracy comparison / Quality metrics 01
ML Algorithms
Support all type of ML Algorithms based on the requirement (like Classification, Regression and Clustering)
Dynamically compare model accuracy across multiple models and identify best performing model 02
Model Frameworks 04
Regression Testing
Support ML/AI models generated in leading technologies such as Python, R and Spark
Perform regression testing on models and qualify model performance. Good or bad against current production model.
This phase is to assure that model built is “correct” for the purpose and gives “right” results.
ML algorithms – There are many types of algorithms like regression, classification and clustering to name a few. QA for AI will support all type of major supervised/unsupervised ML algorithms.
Model frameworks - ML models can be built in various technologies like Python, R and Spark. QA for AI can support all technologies.
Quality metrics – It is key to validate whether model we have in hand is the best out of all other models. This will be objectively measured and selected using industry standard metrics like AUC, F1 score, Confusion matrix. This is automated using various utilities in QA for AI.
Regression testing – While quality metrics will help with model in development. We need to regress new ML model out of development and qualify that it is ready for production. Hence it is critical to do regression testing of such a model built and ready for production.
| © 2019 Cognizant

11. 3. ML Process Quality Assurance
Assess maturity of Machine Learning process, frameworks of organizations and provide comprehensive recommendations on improvement areas.
Assess ML process with regard to business value definition.
Assess algorithms and data processing pipeline usage with regard to business problems.
Process 01 04
Assess ML Frameworks on features, integration and scalability aspects.
Assess support for processing of structured and unstructured data. 02 05
Technology
Value
Assess metrics and measurement techniques of model performance.
Assess ML projects with regard to model deployment, governance, ethical assurance and robustness. 03 06
Process quality assurance will be driven by questionnaire and not utilities
ML process quality is assurance is more of consulting objectively on existing ML process.
Assess ML process with regard to business value definition. That whether required process in place to cater to the need of business like millisecond prediction can be validated or is it more of offline prediction which does not require more rigor.
Assess ML Frameworks on features, integration and scalability aspects so that required validation is performed quickly and efficiently.
Assess metrics and measurement techniques of model performance. Ensure that industry standard metrics are used for the purpose.
Assess algorithms and data processing pipeline usage with regard to business problems. Validate whether pipeline in place can solve any algorithm or data related issues. Whether it is lean and efficient.
Assess support for processing of structured and unstructured data. Data ingestion from unstructured and structured data sources has standard tools like talend, Logstash and processes.
Assess ML projects with regard to model deployment, governance, ethical assurance and robustness. Validate frequency in which model is validated and whether it is sufficient for the governance of the model.
| © 2019 Cognizant

12. 4. Assure ML Security and Performance
Assure ML solutions on Non-Functional aspects including security and performance.
Validate predictive response time of the models at varying user loads.
Validate model with regards to train and retrain time during learning phase.
Predictive Response Time
Train-Retrain Time
Validate scalability of models with regards to volume of data.
Validate ML models on its robustness with regard to CPU usage, memory consumption etc.,
Scalable Models
Model Robustness
As you can see this step is make sure that ML models security and performance validated.
1. It ensures that we have model which has predictive response time. Depending on peak user loads or transactions, the model should yield the outcome within acceptable time window.
2. Similarly train and re-train time during learning phase should remain with the limits like it should not take weeks to train a model rather it should be within a nightly batch window.
3. Similar to user loads, the model should perform with influx of huge volume of data and entire stack should be scalable to handle the spike in volume of data.
4. Model robustness can be validated against standard performance tuning aspects like CPU usage, memory consumption.
| © 2019 Cognizant

13. ML Assurance Workbench – Simplifying & Industrializing ML Assurance
“Enabling QA for AI through ML Assurance workbench”
GUI based ML assurance platform
Validate Model and Data Quality over industry defined metrics & measurements
ML model comparison dashboard across varied data set & parameters
Monitor model regression across different sectors and make informed decisions
Automated feature impact assessment to determine model stability based on feature relationship
| © 2019 Cognizant

14. AI for QA
Use AI to test application intelligently
| © 2019 Cognizant

15. Artificial Intelligence for QA activity through Quality Insights
Creating intelligent and agile systems that deliver results at a fraction of the cost DO
LEARN
Replicate repetitive human actions
Learn to understand context, adapt to users and systems
THINK
Learn to understand context, adapt to users and systems
Handle judgment-oriented tasks
| © 2019 Cognizant

16. Quality Insights – Insights powered by BOTS
Eliminate Non Productive Testing
Assure Immersive Customer experience
Greater Business Requirements Coverage
Close the feedback loop by learning from logs
Proactively Plan the Test Process
BizInsight
DevInsight
TestInsight
OpsInsight
CxInsight
Requirements
Coding & Unit Testing
System &SIT
Operations
Customer Feedback
| © 2019 Cognizant

17. BizInsight : Powered By BIZBots
Data Source
User stories
Defect logs
Test Execution logs
Defect BackLog
User Story Points
Defect predictor (biz)
Predict defects early in the cycle by application of advanced machine learning techniques on user stories and defects and Predict count of defects in upcoming release using key parameters relating to defects in past releases
TC Failure predictor (biz)
Perform risk based testing by identifying the failure test cases from user stories based on pattern analysis.
HOW DO I WORK ?
Defect backlog pruner
Prune and optimize defect backlog by identifying priority defects based on active user stories and invalidating stale defects that are no longer business relevant.
Story Point Predictor
Predict the user story point by application of clustering machine learning algorithm on user stories.

18. DevInsight: Powered By DevBots
Build verification predictor
Predict success percentage of build based on SCM & build logs
TC Failure predictor (dev)
Helps to identify impacted test cases from file changes during the release from SCM logs
HOW DO I WORK ?
Defect predictor (dev)
Predict defects based on file modification patterns from the SCM logs
Predict defect based on duplicate code analysis
Data Source
SCM logs
Code
Build logs
Defect Logs
Test Execution Logs

19. TestInsight: Powered By TestBots
Regression Test Optimizer
Identify test scripts that are similar by applying text analytics and clustering algorithms on http, https logs from client during test case execution
Duplicate Defect Identifier
Identify duplicate defects by using keyword matching techniques
HOW DO I WORK ?
Defect Resolution Time Predictor
Predict defect resolution time by analyzing historic defects and resolution time
Data Source
Defect logs
Test cases
Test execution logs
Test plans
Schedule Variance Predictor
Predict schedule variance using the historical data.
Traceability Mapper
Automatic Traceability mapper that generates traceability between Defects and Test cases using advance text analytics.
Defect AI Assist
Identifies frequent failures and to whom defect can be assigned.

20. OpsInsight: Powered By OpsBots
Testing Gap Analyzer Using App Server Logs
Identify gaps in testing process by establishing the correlation between production logs and test execution logs.
TriageBot
Predict root causes of the defects using ML based solution that learns from historic defect data.
Testing gap analyser using db foot print
Identify testing gap in test data by comparing patterns of data from production and test databases key columns.
HOW DO I WORK ?
Data Source
Application Logs
Database Logs
Server Logs
Event Logs
Error Logs
Performance Logs
Test Mgmt. data
Database records
Truck Roll Predictor
Predict whether truck roll can be avoided using predictive analytics model
Firmware update Predictor
Predict whether firmware upgrade is required using predictive analytics model
Test data distribution analyzer
Algorithm based solution that plots data distribution patterns and identifies data mining attributes such as unique combination of data.

21. CXInsight: Powered By Customer Experience Bots
Customer Feedback Analysis BOT
Helps capturing the voice of the customer and the underlying human emotions with sentiment analysis and clustering end-user feedback into actionable QA attributes like Functionality, Performance, Usability etc
HOW DO I WORK ?
Data Source
Mobile App reviews (Playstore & iTunes AppStore)
Social Media Comments/Tweets
End User/ Survey Feedback in Excel format
Conversational BOT
Help user in selecting suitable bot in a conversational way/manner.

22. BOTs in a DevOps workflow
Req.
Dev.
Test
Build
Prod.
Production Environment
Req. Mgmt
Test Design
Design
Test Plan
Develop
Unit Testing
Build
Version Control
Deployment (QA)
Test Execution
Test Results
Defect Backlog Pruner
Defect Prediction
Test Case Failure Prediction
Reduced Defect leakage
Faster QA cycles
Improved visibility
BIZ INSIGHT
ADPART
Defect Prediction
TC Failure Prediction
Improved visibility
Increased speed
Lower cost
ADPART
inDev.
DEV INSIGHT
Build Verification Predictor
DEV INSIGHT
Reduced Defect leakage
Increased speed
Continuous Feedback / Improvement
Increased speed
Quicker Defect resolution
Improved visibility
OBJECT PROBE
DEV
INSIGHT
ADPART
TEST
INSIGHT
OPS
INSIGHT
OPS
INSIGHT
Defect Prediction
TC Failure Prediction
Duplicate Defect Identifier;
Test Data Distribution Analyzer;
Regression Test Sequencer & Optimizer; Testing Gap Analyzer;
Traceability Mapper
Testing Gap Analyzer;
Triage BOT; DashBot
Survey BOT; Chat BOT
Solution Selector BOT
Customer Insight
Increased visibility
Better Quality
Lower cost
iDashboard
CX INSIGHT
BizInsight
DevInsight
TestInsight
OpsInsight
CXInsight
Cognizant IPs
| © 2019 Cognizant

23. Bots Studio –Industrializing BOTs Development and Deployment
“Simplifying AI for QA through a single platform”
* Drag & Drop enabled visual workflow designer platform for the analytics solution development
Simplifies development effort and the need for niche team composition
Pre-built catalog of ready to use BoT for common use cases
Ability to qualify and validate analytics model built with comprehensive metrics and measurement techniques
Enables faster deployment to production through one click deployment
BUILD & CONFIGURE
TRAIN & TEST
DEPLOY
| © 2019 Cognizant
* H Roadmap item

24. Thank you
Digital Assurance CoE
© 2019 Cognizant