1. Expert System Presentation On…. Software Certification for Industry - Verification and Validation Issues in Expert Systems By Anca I. Vermesan Presented by Wilford Morrison

2. Software Certification for Industry - Topics Introduction KBS Characteristics applicable to V&V –Task Related –Architecture Related Software Certification –Product vs Process Certification Components of Certification –Quality Characteristics of KBS –V&V Methods & Techniques –Methods of Certification Industrial Needs –V&V of Safety Critical Systems Conclusion V&V – Verification & Validation KBS- Knowledge- based Software

3. What is Verification and Validation? V&V techniques used What is Certification? Components of Certification Why Certification for KBS? Issues with Certification Software Certification for Industry Introduction

4. Software Certification for Industry New V&V techniques developed for KB component Two major characteristics: –Type of task performed – What are specific task being carried out within a narrow domain? –Type of Internal architecture – The similarity of architecture across all KBS Characteristics of KBS Applicable to V&V

5. Software Certification for Industry The vague nature of the requirement specification Incremental nature of the knowledge elicitation process Not amenable to formal analysis Characteristics of KBS Applicable to V&V Task-Related : Issue: In KBS what serve as input is typically not well defined at the outset and the relation of possible inputs to the output may not be clear either compared to conventional system

6. Software Certification for Industry Uses declarative style programming languages (usually Rule-based) Often cause different types of errors and anomalies (new V&V technique needed) Characteristics of KBS Applicable to V&V Architecture-Related : Issue: While some of the errors are similar to conventional software, the effects and methods of detection are rather different

7. Software Certification for Industry Why software certification? –Gives confidence, quality and dependability –Determinant factor in software competitiveness –Assuring their conformance with requirements and specifications Software Certification

8. Software Certification for Industry Composed of three types:- –Self certification- when a producer declares that their product conforms to specified standards –Buyer certification- a particular buyer may require that products be submitted for certification by a specified body –Third-party- a body independent of both the producer and the user carries out the certification process (the focus of this paper) Software Certification ….cont’d

9. Software Certification for Industry Consist of three major activities:- –Software quality evaluation- involves V&V techniques, tests, static/dynamic analysis as a measurement to determine quality of the software –Software quality assessment- the process of comparing the actual tests and measurements of the functions of interest with the specifications –Software quality certification- an independent third party gives written assurance that a software product or process conforms to specified functions Software Certification ….cont’d

10. Software Certification for Industry Software Certification ….cont’d Product vs. Process Certification The idea that process-approach gave way to product-approach Ultimate goal of certification is to assess the quality of the software product Therefore, process certification is intended to analyse the software development process

11. Software Certification for Industry Process Oriented Certification steps include:- –Certifier and manufacturer agree on criticality of the software –A software methodology is then agreed upon –Software is developed, validated and tested according to the methodology –Documented results allow the certifier to approve the process and a certificate may be granted Software Certification ….cont’d Product vs. Process Certification

12. Software Certification for Industry Product Oriented Certification steps include :- –Analysis of the actual requirement for the evaluation (This may refer to standards and regulations with which the product should comply) –Specification of the evaluation that is to comply with the requirements identified by the analysis –Design of the evaluation which shall identify evaluation techniques and plan –Performing the evaluation (consist of inspecting and testing product parts) –Reporting the result obtained from performing the above activities Software Certification ….cont’d Product vs. Process Certification

13. Software Certification for Industry The three major components are:- –Quality characteristics –V&V methods and techniques –Certification methods Components of Certification

14. Software Certification for Industry This list consist of:- –Competency- this deals with the quality of the knowledge in the system relative to human skills –Completeness- this involves ensuring all the knowledge is referenced and there is no attempts to access non-existing knowledge –Correctness- the knowledge within the KB should be 100% –Reliability- determines how often a system fails to arrive at the correct solution to a problem –Testability- system should be designed so that a test plan can be carried out –Verifiability- possible to establish logical correctness of the KB through verification Components of Certification ….cont’d Quality characteristics

15. Software Certification for Industry Clustered into two main groups:- –Non-method specific techniques- involves human analysis of a product –Method specific techniques- involves using a formal and structured approach Components of Certification ….cont’d V&V Methods and Techniques

16. Software Certification for Industry Non-method specific techniques –Involves human analysis of product, relying on individuals to use their experience to find errors –Involves the general examination of programs by using reviews, walkthroughs and inspections to identify defects and discrepancies –Inspections are aim at detecting semantically incorrect knowledge Components of Certification ….cont’d V&V Methods and Techniques Issue: Use of this technique usually lead to bias and omissions

17. Software Certification for Industry Method specific techniques –The properties of the product are well defined and can be expressed in a structured and formal way –This method may offer rigorous checks of the code functionality to mathematical proof of specification refinement –Can be further classified into:- Empirical methods Logical methods Components of Certification ….cont’d V&V Methods and Techniques

18. Software Certification for Industry Empirical Methods Involves running a prototype system on a selection of test cases and assessing the results. (Known as “black-box” validation ) Logical Methods Consist of identifying anomalies such as redundant, contradictory, circular or deficient knowledge. (Known as “white-box” validation ) Components of Certification ….cont’d V&V Methods and Techniques Issue: exhaustive testing is impossible due to the many paths through the KB

19. Software Certification for Industry Common methods used are:- –Static Analysis –Dynamic Testing –Inspections –Modeling Components of Certification ….cont’d Methods of Certification

20. Software Certification for Industry Static Analysis –This involves subjecting the program texts (and its requirements and specifications documents) to scrutiny and review in order to detect inconsistencies and omissions Static Analysis –This is a generic form for techniques of software analysis which do not involve software execution –Other forms include, dataflow diagrams, symbolic execution, requirements tracing and program proving Components of Certification ….cont’d Methods of Certification Note: Static methods are in general judged cheaper, more practical to perform, and more amenable to automatic testing than dynamic methods

21. Software Certification for Industry Dynamic Testing –Encompasses any form of software analysis, which involves software execution. –In KBS, the input space may be so large that it is infeasible to examine the behavior for all inputs –Also for KBS, an expert is usually required to judge the errors detected, and many test cases need to be generated for systems with extensive chaining between rules –Other forms include, random testing, equivalence partitioning, boundary testing and cause-effect graphs Components of Certification ….cont’d Methods of Certification Note: Dynamic testing for evaluation in the view of certification would most likely involve re-running test cases prepared during the software process development and applied during the testing phase

22. Software Certification for Industry Software Certification has strategic importance to all industries Mature software industry help to integrate other industries Improved V&V techniques along with considerable experience and expertise has lead to a transfer to industry Industrial Needs

23. Software Certification for Industry As essential components- where the KB element performs an essential function within the system and without which the system will not perform As advisory components- where they act in an advisory or supervisory capacity to enhance an independent system Industrial Needs….cont’d KBS in Safety Critical System Note: - When a KBS fails, it usually fails catastrophically rather than safely - KBS put a burden on the overall system hazard and safety analysis

24. Software Certification for Industry V&V of KBS are as much potential value to certification specialists as they are to software producers Certification of KBS cannot be sufficiently accomplished by use of static testing means V&V need objective points of reference on which to guide the certification activities Conclusion

25. Software Certification for Industry The “process culture” of traditional software engineering may at last lead to the “product culture” in knowledge engineering Increasing use of software embedded in systems that need to be certified for safety reasons will demonstrate the necessity for software certification Software certification has a strategic importance, as it is a way in which the software industry can become mature and integrate harmonically with other industries Conclusion….cont’d

26. Software Certification for Industry - Verification and Validation Issues in Expert Systems Questions?