Model-Based Black Box Testing and its Standardization Layers MaTeLo THE SOLUTION TO BOOST YOUR TEST EFFICIENCY

EMBEDDED SYSTEMS ALL4TEC Since 1998 French: 45 experts Turnover 2010: 4 M€ Expertise offer: PROCESS IMPROVEMENT SAFETY ENGINEERING SYSTEM ENGINEERING OPTIMIZED TESTING SOFTWARE RELIABILITY EMBEDDED SYSTEMS

OUR COMPETENCES OFFER Training Consulting Engineering Tooling MaTeLo EXAM Safety Architect M Elopée

AGENDA THE TEST PROBLEM THE EXPECTED SOLUTION THE MODEL-BASED TESTING SOLUTION A PRAGMATIC INDUSTRIAL USE-CASE A MORE STANDARDIZED TOOLS INTEGRATION

THE TESTING PROBLEM Test Resources Lack Test Focus Deviation Testing Effort Time Test Needs System Complexity Test Availability Usage Intensity Product Life Reliability Market Expectation Customer Validation Development Phase Reality Normal Usage

HOW TO SOLVE THIS ISSUE? before product release Test the SUT usage engineering process Use best in class automatic generation Widely use Use dedicated tools that decuple engineering productivity Use formalization and traceability

MODEL-DRIVEN ENGINEERING Functional Requirements DESIGN TEST Implement Scenarize Design Model Usage Model MaTeLo Generate Generate Model Based Testing Code Test Cases Compile Derive Test Environment EXE Test Scripts

WHAT IS MaTeLo? MaTeLo Formal Test Specification More Productive Product Usage Point of View More Productive Coverage Fully Measured Process Systematic Engineering Productivity MaTeLo Bidirectional Requirements Traceability Markov Test Logic Test Execution Compatibility

MATELO SCREENSHOT

INTEGRATED TEST WORKFLOW MaTeLo Testor Automatic Test Cases Generation PLAN MaTeLo Usage Model Editor Test Design into Usage Models ACT MaTeLo Test Campaign Analysis Confidence And Coverage Reporting EXAM Test Automation DO CHECK

MaTeLo EDITOR Usage Model Design

MATELO USAGE MODELS Probability on every Alternatives Association of Requirements Configuration of Test Operations Setup Transition as Test Step BEGINNING TRANSITION MACRO CHAIN STATE END

MODEL TRANSITION = TEST STEP Test Operations Select_Gear(5) Accelerate(100%) Check_Speed(185) Requirements Stimulations Inputs Stimulation Equivalence Classes Timing Test Oracle Outputs = f(Inputs) Verifications Expected Outputs Timing

Usage path probability MATELO PROFILE Profiles can be embedded to qualify the usage model Operational profiles Test profiles Data distribution Usage path probability 347 0-100-150-200-500 87 (+/- 25)

MaTeLo EDITOR Requirements Management

REQUIREMENTS MANAGEMENT IBM Rational Doors MaTeLo Requirements Library Import MaTeLo Usage Model Association (N,M) Requirement Update New Baseline Quick Model Update Update Requirements Change Analysis Model Impact Highlighting Available for other ReqMgt tools with import of XML or CSV files

MaTeLo TESTOR Test Cases Generation

CRITICALITY, COMPLEXITY MaTeLo TEST STRATEGY Start Close Risk based approach Custom Test profile CRITICALITY, COMPLEXITY UPDATE FOCUS Most probable approach Start Close FREQUENCY FOCUS Arcs coverage approach Close Start REQUIREMENTS COVERAGE Usage Test profile approach Start Close OPERATIONAL COVERAGE DEFINE THE TEST STRATEGY, BY CHOOSING Test Algorithm Test Profile Part of model to test Generate

MaTeLo TESTOR: HTML TEST PLAN Input Expected Result Requirement Verdict State Test Function

Example: MaTeLo FOR EXAM Association Test Repository SUT Interface Test Operations Stimulation Measurement Administration Sub Test Sequences Requirements Name UUID Description Association Automatic Test Cases Automatic call of Test Operation Automatic parameterisation Requirements association Test Case description generation Usage model respect Generate

MaTeLo TCA Test Campaign Analysis

Confidence Indicators TEST CAMPAIGN PROCESS New Test Cases Generation Metrics Quality/Version Target Quality Criteria Coverage Reliability Confidence Indicators New Package New version Bugs Operational Confidence Needs Coverage Functional Coverage

From Pragmatic Tools Integration To Its Standard Adoption

THEORETICAL MBT WORKFLOW Automation Test Framework Test Scheduler Simulator Atomic Test Action Confidence & Coverage Reports Test Scripts Usage Model Test Cases Test Cases Verdicts Test Execution Log Functional Requirements System Under Test

A 1ST INDUSTRIAL TOOLS INTEGRATION EXAM UML Python dSpace MicroNova Carts Vision CAN IOs Doors Automation Test Framework Atomic Test Action Confidence & Coverage Reports Test Scheduler Test Scripts Usage Model Test Cases Test Cases Verdicts MaTeLo Simulator Test Execution Log Functional Requirements System Under Test

A 2nd INDUSTRIAL TOOLS INTEGRATION TestStand Proprietary Opal-RT NI Clemessy Diagnostic Calibration Radio Reqtify MS Excel MaTeLo EXAM Automation Test Framework Atomic Test Action Confidence & Coverage Reports Test Scheduler UML Test Scripts Python Usage Model Test Cases Test Cases Verdicts dSpace MicroNova Simulator Carts Test Execution Log Functional Requirements Doors Vision CAN System Under Test IOs

VARIOUS TOOLS INTEGRATION EXAM TestStand Automation Test Framework Atomic Test Action Confidence & Coverage Reports PROVEtech Proprietary CANoe EXACT TTCN-3 VB C, C# CSV Java Simulink Flexray Temperature XML HP QC Test Scheduler UML Test Scripts Python Usage Model Test Cases Test Cases Verdicts Proprietary MaTeLo dSpace NI Opal-RT MicroNova Simulator Clemessy Carts Test Execution Log Functional Requirements Doors Reqtify Vision CAN System Under Test Diagnostic Calibration MS Excel IOs Radio

VARIOUS TOOLS INTEGRATION TestStand Proprietary dSpace Opal-RT MicroNova NI Clemessy Carts Vision Diagnostic Calibration CAN IOs Radio Doors Reqtify MS Excel EXAM UML Python PROVEtech CANoe EXACT TTCN-3 VB C, C# CSV Java Simulink Flexray Temperature XML HP QC Automation Test Framework Atomic Test Action Confidence & Coverage Reports Test Scheduler Test Scripts Usage Model Test Cases Test Cases Verdicts MaTeLo Simulator Test Execution Log Functional Requirements System Under Test

MBT AND ITS STANDARDS Simulator TestStand Proprietary dSpace Opal-RT MicroNova NI Clemessy Carts Vision Diagnostic Calibration CAN IOs Radio Doors Reqtify MS Excel EXAM UML Python PROVEtech CANoe EXACT TTCN-3 VB C, C# CSV Java Simulink Flexray Temperature XML HP QC Automation Test Framework Atomic Test Action Confidence & Coverage Reports Test Scheduler ATX Test Scripts ATX Usage Model Test Cases Test Cases Verdicts ATX ATX OTX HIL API MaTeLo RIF Simulator ODS Test Execution Log Functional Requirements FIBEX XCP System Under Test ODX GDI

Tools Standard Application Feeling Items Tools Standard Application Feeling Requirements Doors RIF, ReqIF 3/5 Test Specification Model MaTeLo, Word - (DSL) - Test Cases Specification Doors, Word, EXAM ATX, OTX 0/5 , 2/5 Test Campaign, Management MaTeLo, Quality Center ATX 0/5 Test Case Implementation EXAM, ECU TEST, TestStand, vb, python… HIL API, TTCN3 1/5 , 2/5 Test Execution Physical Signal NI, dSPACE, Vector… HIL API 1/5 Network Signal CAN, LIN, FlexRay… FIBEX ECU parameter CANape, INCA XCP 5/5 Diagnostic CANdela, ODX, UDS Simulator Signal Simulink, AMEsim… Test Device Signal Agilent, LeCroy… GDI Test Case Execution Log, Trace NI, EXAM, dSPACE, ETAS… ODS, ATML 2/5, 2/5 Test Case Verdict Quality Center, MaTeLo Issue BugZilla, Jira, Quality Center ISSUE ? Requirement Coverage Doors, Quality Center, MaTeLo ATX, ReqIF Confidence Indicator Quality Center, MaTeLo

Wiki & Forum & Documentation QUESTION ? www.all4tec.net Sales contact anthony.faucogney@all4tec.net +33 6 80 88 40 59 Wiki & Forum & Documentation

ABSTRACTION LAYERS MaTeLo Third Party Equipment Usage Scenario Description Markov-Chain Usage Model MaTeLo << derive >> ATX Test Case Specification UML Sequence Diagram << generate >> HIL API Test Flow Control Device Driver Python Precompiled Application << control >> ODX, FIBEX, XCP… System Under Test Hardware In the Loop Test Bench Third Party Equipment

EXAM ABSTRACTION MANAGEMENT Possible Usage Scenarios Test Case 1 Test Case 2 Test Case n Strategic Test Cases ATX HIL Plateform dSpace NovaSim Carts ASAM HIL API … Proprietary Network CAN_1 CAN_2 LIN_X FlexRay_1 FlexRay_2 Functions Manual Gear Auto Gear Hand Free ACC StartStop Test Tools CANoe CANape INCA MS Excel Diag Tool Env. Model Gasoline Diesel Turbo Hybrid Electric Available Test Configurations HIL API ODX, FIBEX, XCP… HIL API