Formalized Model Development & Test Generation: Key Role of Abstraction Bernard P. Zeigler Arizona Center for Integrative Modeling and Simulation (ACIMS) and Joint Interoperability Test Command (JITC) Fort Huachuca, AZ Foundations 04, ASU, Tempe, Oct.13-15

Outline Systems-based framework for M&S –Bernard P. Zeigler and Hessam S. Sarjoughian, “Implications of M&S Foundations for the V&V of Large Scale Complex Simulation Models” Proc. Foundations 02, Laurel, MD, November, 2002, available from Application to Systems Development – Interoperability Standards Conformance Testing Role of Abstraction –Nature of abstraction, scalability –Abstractions employed in JITC application

Simulation-based V&V as an afterthought development of the system simulation tests of the system Government Contractor

Automated test suite design Informal Behavior Requirements at one or more levels of System Specification Simulation based testing Reference Model Formalized Behavior Specification Software- Intensive System Implement- ations Formalized Model System Development Process

Informal Behavior Requirements at one or more levels of System Specification: Systems theory offers well-characterized levels at which requirements for system behavior can be stated informally. Formalized Behavior Specification: The behavior requirements are translated into a formalized model using framework for M&S methodology.. Reference Model: serves as a reference model for any implementation of the standard. This model can be analyzed and simulated with a simulation protocol to study logical and performance attributes Using model continuity, it can be executed in real-time protocol and provides a proof-of- concept prototype for an operational system. Automated test suite design: Branching in the lower path from the formalized specification, we develop a test suite consisting of test models within a test frame that can interact with a System Under Test (SUT) to test its behavior relative to the proposed standard. Simulation based testing: The test suite is implemented in a distributed simulation infrastructure and executed against the SUT. The test suite provides explicit pass/fail/unresolved results with leads as to components/rules that might be sources of failure. Optimization and Fielded Execution: The reference model provides a basis for correct implementation of the standard in a wide variety of technologies. The test suite provides a basis for testing such implementations as SUTs in a suitably extended test infrastructure. Iterative nature of development: The process is iterative allowing return to modify the master formalized model and its informal precursor requirements. Model formalization minimizes the artifacts that have to be modified as the process proceeds. Formalized Model System Development Process

How is simulation software different from other software? It represents the behavior of dynamic systems whose state changes intrinsically depend on time Properly controlling the flow of time is critical Simulation software may combine: –continuous (time-driven) and discrete (event-driven) processes –actual operating hardware and software representations –wall clock and {faster/slower} than real time advance

independent specification of the dynamic system behaves over time methods needed to compare the behavior of the system implementation and the independent specification A Formal Framework for Modeling and Simulation should provide DEVS Experimental Frames

DEVS = Discrete Event System Specification Provides formal M&S framework: specification,simulation Derived from Mathematical dynamical system theory Supports hierarchical, modular composition Object oriented implementation Supports discrete and continuous paradigms Exploits efficient parallel and distributed simulation techniques Theory of Modeling and Simulation, 2nd Edition, Academic Press, 2000, Bernard P. Zeigler, Herbert Praehofer, Tag Gon Kim DEVS Background

DEVS Hierarchical Modular Model Framework Atomic: lowest level model, contains structural dynamics -- model level modularity + coupling Coupled: composed of one or more atomic and/or coupled models hierarchical construction

Automated test suite design: DEVS Federates Standards Document Mil-Std 6016C (JSLIRS) HLA Infrastructure Simulation based testing Reference Master DEVS Model Captured As DEVS Temporal Rules DEVS Model Continuity: System Implement- ations Formalized DEVS-based System Development Process

Interoperabilty Standards Test Generaton Express the Mil-Std 6016C (JSLIRS) as a collection of state variables and transition rules Rules are written as conditions, actions, and exceptions – If D is true now then do action E at time t, unless F occurs in the interim State variables and rules characterize the dynamic behavior of the system Basis for semi-automated reasoning tools to generate test cases that target specific system behavior

Transactions consist of sets of Rules Constraints (Exception) Rules Stop Modify C2 Record for TN 123 Rule Processing Stop, Do Nothing, Alerts, Or jump to other Transaction Track Display Operator decisions Validity checking Transmit Msg Other ConsequentProcessing Jumps (stimuli) to other Transactions of specification Stimuli Transaction Level - example P.1.2 = Drop Track Transmit PreparationProcessing Time outs Periodic Msg

Abstractions: Needed For Scalability Base Specification Abstraction Analyze useful information? Preservation mapping Different abstractions serve to address different questions They may be sequenced combined, and crosschecked Abstractions may be made more concrete (If you could) inverse mapping Answers

Rules Dependency Abstraction Rule 2 depends on (can be affected by the execution of ) Rule 1 if there is a non-empty intersection between the action variables of Rule 1 and the condition variables of Rule 2 Rule 1 A B C Rule 2 C D

Dependency Abstraction and Concretization Rule Abstractions Executable Rules Dependency Graph Analysis Rule Engine Simulation Textual processing Rules Expressed In textual form (XML) Manual Concretization candidate test paths Feasibility Verification test plan

Example Dependency Graph start rule ordinary rule output rule rule text dependencies rule level Rule set: Operator action to initialize the terminal or change data

Example: Path Analysis shortest path (no cycles allowed within) rule under test test sequence for rule under test = feasible path from a start to an output that traverses the rule under test feasible = can be forced on the system under test by starting in a state (variablevalue assignments) that allows the designatied sequence of rule activiations to proceed feasibility has to be checked by rule engine Rule test sequence: 1.Initialize Terminal 2.Check input data validity 3.Transmit J2.2 PPLI message

Abstraction via Modulo Operation on Variable Names G = Rules, Variables Dependency Graph G Modulo Message types Message types Record types invocations G Modulo Record types G Modulo Direct calls Message Flow Data Store Dependency Invocation dependency + Analysis

Abstraction via Rule Set Partitioning G = Rules, Variables Dependency Graph G/group1 G/group2 G/group3 Hierarchical Modular DEVS

Employing Module Abstraction to Discover Test Paths Module abstraction provides a “bird’s eye” view of the overall interdependence of transactions. For example, we might seek a path that tests a desired sequence of transactions, and therefore, traverses a corresponding sequence of transaction modules Such abstraction can eliminate large numbers of possible paths from consideration since paths not present at a higher level cannot appear at a lower level. Moreover discovery of potential paths can be broken into two stages: –discover a path through modules –verify that internal connecting paths within the modules along this path exist Potential paths need to be tested for feasibility using rule engine For large rule sets, this multi-abstraction/multi-stage approach can significantly reduce the search necessary for path discovery.

Summary A Formalized Model System Development Process provides the rigorous basis for specification of behavior. Such specification drives integrated and concurrent development of effective processes to test system implementations. Test generation needs to exploit both micro and macro level analysis. Requirement might be for detailed local function testing or for overall sequence of behaviors Abstraction is the key to the scalability needed to develop and test large complex systems Multi-abstraction/multi-stage path discovery is a scalable methodology We will apply these methods and tools to generate actual test cases, and conduct reviews by SME’s to guide further development This formalized model, semi-automated test case development approach allows JITC to efficiently extract test cases from a standard, convert them to test procedures and apply these directly to a system for Standards Conformance and other testing efforts.