Page 1 Software Test Automation Texas Instruments Software Development Organization September 2009

Page 2 Agenda 1. Goals of Common SDO test automation effort. 2. System Architecture / Layers 3. Test Setup 4. Test Plans 5. Roadmap

Page 3 1) Goals

Page 4 Goal of SDO Automation Effort To establish a common integration and test infrastructure for development and test teams to use across SDO that leverages open source components To enable customers, partners, and community to run regression tests and contribute tests

Page 5 Goals and Solutions Goal: Share same test suites/test cases across sites – Solution: Use open source test management system (Test Link) to store test suites. Use single test database at all sites Automate the test execution of most test suites/test cases. Goal: Open architecture to support multiple execution engines – Solution: Design a system that allows using different types of execution engines to suit user needs. The execution engine type is part of the test case definition. Goal: Share hardware resources – Solution: Implement a common Resource Manager that allocates resources as needed Goal: Enable more people to run tests – Solution: Implement a web-based test management system that allows people to trigger common tests and get results at local or remote locations. Goal: Facilitate Continuous Integration & Test – Solution: Incorporate Build Execution Engines (BEE) into the test automation architecture so a Continuous-Integration system can be implemented. Code can be built and tested by developers and testers as often as desired and at any point in the development cycle. Goal: Maximize Open Source Software usage – Solution: Leverage existing open source tools as much as possible The new test architecture uses TestLink and Staf.

Page 6 2) System Architecture

Page 7 High-Level Test Architecture Test Master Controller Build Execution Engine Test Management System Test Artifacts (Mysql) Service Provider Test Execution Engine Test Execution Engine Service Provider Test Execution Engine Site A Build Execution Engine Test Management System Test Artifacts (Mysql) Service Provider Test Execution Engine Test Execution Engine Service Provider Test Execution Engine Site B IP Network Test Artifacts Databases must be synchronized on a periodic basis The 3 rd layer is the service-providers layer. TEE and BEE are just special Service Providers. It should be possible to reuse Service Providers across sites if/when desired Typically 1 TMS : 1 TMC 1 TMC : Many Service Providers Test Master Controller Create and Manage Test Cases & Organize them into Test Plans. Request Test Job Execution. Manage Resources. Dispatch & Monitor Jobs. Execute Build & Test Jobs.

Page 8 Test Architecture: 3 Functional Layers 1) Test Management System. 2) Test Master Controller. 3) Service Providers Test Projects Test Management System TI Custom Fields* Test Master Controller TestLink (GPL) + Staf Staf (EPL) Test Reports Test Requirements Test Execution Engine … Equipment Drivers * Custom fields= Test Execution Engine, Test Execution Logic, Test Parameters, HW Assets, SW Assets Staf + Vatf|Stax|iCOTS|TestRunner|Other Test Cases Web Front-End Execution Logic Resource Management Service n 1 1 Data in XML Data adapted to TEE Build Execution Engine … Execution Logic Data adapted to BEE n Dispatcher Service Monitor Service Test Plans

Page 9 Test Management Layer: Test Link Test Link is an open source test management tool – It uses a MySQL database to hold test cases / test suites / test plans – It runs on a server, and requires no proprietary tools to run – It uses a browser as the user interface – It does not natively have good support for automated test cases, but we are enhancing its automation support by integrating it with STAF

Page 10 Test Master Controller Layer: Staf Staf is an open source, multi-platform, multi-language framework designed around the idea of reusable components. – It allows Staf processes on different machines to communicate & initiate processes on each other We will use it for: – Resource management – Dispatching test requests for execution – Tracking test execution – Communication between layers of the system

Page 11 Service Providers: TEE / BEE Layer The service providers layer creates builds and runs tests Initial implementation: – Test Engine: VATF (Home-grown Test Execution Engine) – Build Engine: File BEE (provides pre-built image) Arago BEE (builds image using Arago/OE infrastructure) Staf client will run on test execution systems to interface the TEE and BEE components to the TMC layer, and to perform resource registration / status

Page 12 3) Test Setups

Page 13 High-Level Test Setup

Page 14 TEE Test Setup details (USB)

Page 15 TEE Test Setup Details (OVQ)

Page 16 4) Test Plans

Page 17 Test Plans TI will offer Test Plans for individual components (i.e. PSP) and for integrated systems/solutions (i.e. DVSDK). TI will provide Test plans for different test scopes: – L1: Level 1 Sanity (a.k.a Smoke) tests focus on verifying that a feature or component is operational. Each L1 test should take at most few minutes to complete and the whole L1 test plan should not take more than 12 hours to run. L1 test plan should be run as frequently as possible. – L2: Level 2 tests focus on complete functional validation. The L2 Test plan may take few days to run and may involved manual testing. – L3: Level 3 tests focus on measuring performance, stressing the system and exercising system use case scenarios. The L3 test plan may take few weeks to run and most likely required some manual testing.

Page 18 Test Plans cont. TI has test plans for many areas as shown in the next two slides, however they are not yet in TestLink format. TI is currently in the process of migrating existing test plans to TestLink. After migrating the test plans to TestLink, we will benefit from some of the features of the new system such as reuse same test logic at different locations, hardware resource management and continuous testing of software built using Arago. Existing automated test plans focus primarily on L1 and L2 testing. TI is working on adding more L3 tests as well as incorporating new open source test tools into the automated test plans.

Page 19 PSP Test Areas ATA Audio ALSA, OSS CIR EDMA Ethernet GPIO I2C Keyscan MMC SD, SDHC, SDIO NAND NOR RTC SPI Timer UART Uboot USB Host MSC, ISO, HID USB Slave MSC, CDC/RNDIS Video Fbdev Display Video V4L2 Display Video V4L2 Capture Video H3A Video Previewer/Resizer Video Face Detection WDT

Page 20 DVSDK Test Areas Applications: – Demos – DVTB – Gstreamer Codecs: – H.264 encoder – H.264 decoder – MPEG4 encoder – MPEG4 decoder – JPEG encoder – JPEG decoder – AAC encoder – AAC decoder – G711 encoder – G711 decoder – G729 encoder – G729 decoder – MP3 decoder

Page 21 Test Equipment & Drivers Drivers for the following Hardware has already been written and validated: – Generic Linux host driver (DM355, DM357, DM365, DM644x, DM6467, etc.) – APC Power Controller – Extron Video/Audio Switches – Extron USB Switches – Tascam DVD players – Pionner DVD players – Q-Master's Video Quality System – Video Clarity's Video Quality System – Generic PC Sound Card – Spectra Lab's Audio Quality System – Opera’s PESQ Speech Quality System. There are also some second-tier drivers that reuse HW drivers and add extra functionality – DVSDK DVTB for multiple DVSDK releases – DVSDK Demos for multiple DVSDK releases – Windows & Linux USB Hosts

Page 22 5) Roadmap

Page 23 Roadmap Beta: December 2009 – Internal and selected External users are able to setup the system and run existing Test Plans using ‘high-end’ test equipment. Open source Git Project available Contributions limited to internal and possibly a few external users. Installation Guide available. Users Guide available. GA: March 2010 – Users are able to develop new test plans on their own and contribute to the open source project. Contributions open to all Developers guide available. Developers how-to videos available. Support for less-expensive test equipment, so it is easier for anybody to run tests on their setup.

Page 24 Backup Slides

Page 25 Steps to create automated Test Plan 1. Select Test Equipment to use (if any) 2. Select BEE & TEE to use 3. Define Test Parameters 4. Write Test Equipment driver (if one doesn’t exist) 5. Write Test Script for selected TEE 6. Add new Test Cases/Test Suite to TestLink project. Each test case identifies: 1. Required Software Assets 2. Required Hardware Assets 3. Test Parameters (DUT, Test Equipment, Application) 4. Test Execution Engine type (i.e. VATF, STAX) 5. Test Script An easy way to remember the information that goes into the test case definitions is by remembering the word SHAPES

Page 26 DISPATCHER TMC’s StafProc RESMGR TMS/User Request TEE TEE Register w/ RESMGR and provides its type and capabilities upon startup. It also registers with the dispatcher and provides xslt or exec file to transform the test data. 2.Test Request send to TMC 3.DISPATCHER query RESMGR for available TEE type w/ required HW_ASSETS (capabilities) 4.RESMGR provides TEE machine name 5.DISPATCHER send test request to TEE machine gotten on step 4. 6.TEE completes job execution and notifies DIPATCHER. 7.DISPATCHER releases TEE 8.TEE unregister w/ RESMGR upon shutdown 7 8 TMC’s Resource Manager

Page 27 Test Management System’s Services TestLink Basic Definitions: – Test Project holds together the multiple test artifacts (test cases, test requirements, test plans, etc.) – Test Case describes a testing task and expected results. Test Cases are the fundamental piece of TestLink. – Test Suite is a collection of Test Cases. It organizes Test Cases to units. – Test Build is the software to be tested. It is part of the Test Plan. – Test Plan is created when you'd like to execute Test Cases. Test Plans can be made up of the Test Cases from the current Test Project. Main Services provided by the TMS: – Store all Test Artifacts – Provides ways to Add/Edit/Delete Test Artifacts:  Test Cases  Test Suites  Test Requirements  Test Builds  Test Plans (Test Cases + Build Tag) Test Results – Export/Import Test Plans to/from XML – Trigger test execution (optionally)

Page 28 Test Master Controller’s Services Dispatch Jobs to appropriate BEE & TEE Translate Test parameters between TMS and BEE/TEE Provide TMS-independent test case representation to TEE Provide a mechanism by which TEE can save test results in TMS Manage Service Providers

Page 29 Service Providers’ Services At least there will be two types of service providers: – BEE: Build Execution Engines to build the required software assets. – TEE: Test Execution Engines to run the test cases. The TEE maybe as simple as a shell script or as complex as VATF. BEE & TEE are the only service providers that the TMC Dispatcher directly request services from. Complex TEE, such as VATF, will typically perform these tasks: – Initialize HW assets (both Device Under Test and Test Equipment) – Create Test Parameters representation – Run Test Execution Logic Load Device Under Test (DUT) w/ appropriate SW assets Boot DUT Connect DUT to Test Equipment Configure DUT and Test Equipment Check conditions or measure metrics Determine Pass/Fail – Save Results & Logs