1. Meta-ivi Testing BoF Tolkien Joh Stephen Lawrence
October 12, | Status, How-to and futures
Tolkien Joh Stephen Lawrence
Senior MTS, Wind River (Yocto Baseline Maintainer) Principal Engineer, Renesas (BIT Lead)
This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0) GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries. Copyright © GENIVI Alliance 2017.

2. Agenda Status of public baseline testing
How-to on building, running and extending meta-ivi-test
Automation and next steps

3. Status / where we are

4. Yocto baseline test framework (meta-ivi-test)
History
Historically release testing took place within members
EGs tested their s/w on (0.1, 0.2) pre-releases and report result to BIT
meta-ivi-test layer added
Source, wiki
From meta-ivi 11 (Miranda) by Tolkien (WindRiver)
Release test results for recent GENIVI releases posted on wiki
Provides an image for testing
Created by baseline test build, not release build
Based on some debug/test codes of GENIVI s/w component
Mainly component unit tests

5. Yocto baseline test framework
Pros
Simple
Cons
Maintained by one person
It does not have all tests from all GENIVI components
We need help/contribution to improve coverage and add missing component unit tests
There is no generated report
It provides some “raw” script terminal output only
To integrate into CI or be used for JIRA more easily we may have to convert this to some formatted text like XML, JSON

6. Meta-ivi-test How-to

7. Meta-ivi-test source layout
Test runner
Test Set
Test case runner
Test case
/opt/tests/
test-fw.sh
audiomanager_t.inc
audiomanager/
capicxx-perf/
vsomeip/
AmControlInterfaceTest
AmRouterTest
AmTelnetServerTest
AmMapHandlerTest
AmRoutingInterfaceTest
AmRouterMapTest

8. How to use meta-ivi-test
Tests are run in the meta-ivi runtime environment
We will cover:
Building and booting a test-image
How to create tests
Create a test set
Create a test case

9. Building and booting a test image
add meta-ivi-test layers to <build dir>/conf/bblayers.conf
build test-image
Boot test-image (qemu use case)
For real h/w case deploy test-image to h/w and boot
BBLAYERS += " \
    ##your source code root##/poky/../meta-ivi/meta-ivi-test \
    "
export TEMPLATECONF=`pwd`/meta-ivi/meta-ivi/conf
source poky/oe-init-build-env build
MACHINE=qemux86-64 bitbake test-image
../meta-ivi/scripts/runqemu test-image qemux86-64 nographic

10. Test runtime commands run all tests of all s/w
list the supported components and the number of tests they provide
/opt/tests/test-fw.sh
##########
audiomanager
[ number of tests: 7 ]
[ run_1 ][ PASSED ] 19 tests.
...
/opt/tests/test-fw.sh list
audiomanager 7
audiomanagerplugins 5
capicxx-perf 1
common-api-test 1
dlt-daemon 2
...
<s/w name> <number of test cases>

11. Test runtime commands (cont)
run the test set for a specific component
run an individual test case
/opt/tests/test-fw.sh dlt-daemon
##########
dlt-daemon
[ number of tests: 2 ]
[ run_1 ] PASSED
[ run_2 ] PASSED
/opt/tests/test-fw.sh dlt-daemon 1
##########
dlt-daemon
[ run_1 ]
testFile1Run1 successful
testFile1Run2 successful
testFile2Run1 successful
...

12. Creating a test set Create directory for test set <TestSet>
Create test set runner
- filename should be /opt/tests/<TestSet>/<TestSet>_t.inc
mkdir /opt/tests/Sample
ls -F /opt/tests/
Sample/ node-health-monitor/
audiomanager/ node-startup-controller/
audiomanagerplugins/ node-state-manager/...
touch /opt/tests/Sample/Sample_t.inc
chmod +x /opt/tests/Sample/Sample_t.inc
ls -F /opt/tests/Sample/Sample_t.inc*

13. Creating a test set (cont)
#!/bin/sh
get_list() {
return 1 }
get_title() {
echo “Sample"
_run_dir=/opt/tests/Sample
run_1() {
if [ "x${1}" != "x" ]; then
${_run_dir}/hello >& /tmp/t
_c1=$?
if [ "x${_c1}" = "x0" ]; then
echo " PASSED"
else
echo " FAILED" fi
${_run_dir}/hello
Script has to provide the following three functions:
get_list() -> return the number of testcases
get_title() -> return the string of test title
actual test function. The name must be run_x() like
run_1()
run_2()
...
If argument 1 is exist, please, reduce log messages

14. Creating a test case C example Shell script example
#include <stdio.h>
#include <string.h>
int main(int argc, char *argv[]) {
if ((argc > 1) && (strncmp(argv[1], ”mute”, 4) == 0))
return 0;
printf(”hello, world\n”); }
#!/bin/bash
arg=$1
if [ ”x${arg}” = ”xmute” ]; then
exit 0 fi
echo “hello, world”

15. Support
Please post questions to meta-ivi mailing list

16. Where we want to go

17. What is next? Complete Genivi component coverage
Welcome contributions to add unit tests
Integrate with Go CI server
Automation
Fuego seems attractive
In heavy use in AGL (share test cases) and elsewhere.
Plan to integrate Fuego into CI server
Then add meta-ivi-test tests into Fuego
Other options not shut out
LAVA
Kernelci
Who can help?

18. Thank you!
Visit GENIVI at or
Contact us:
This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0) GENIVI is a registered trademark of the GENIVI Alliance in the USA and other countries. Copyright © GENIVI Alliance 2017.