1. Image processing algorithm regression testing framework Soumik Ukil

2. Testing Ground Rules  Objectives:  Testing is the process of executing a program with the intent of finding errors  A good test case is one that has a high probability of finding an error  A successful test is one that uncovers an error  However, testing cannot show the absence of defects

3.  Unit Testing :  Testing to determine that individual program modules perform to specification  Basic units of software tested in isolation  Regression Testing :  Selective retesting to detect faults introduced during modification of system  Should be performed after any changes in software like bug fixes

4. Motivation Projects like NETT and BRP Projects like NETT and BRP Complex algorithms which run on hundreds of datasets Complex algorithms which run on hundreds of datasets Source is being constantly modified to fix bugs etc Source is being constantly modified to fix bugs etc Manual testing infeasible Manual testing infeasible Need automated regression testing Need automated regression testing

5. Testing Strategies Dynamic Analysis Dynamic Analysis Exercise the software in its execution environment Exercise the software in its execution environment Black Box Black Box Treat the system as black -box, ie no knowledge of internal structure Treat the system as black -box, ie no knowledge of internal structure Only need to know legal input and expected output Only need to know legal input and expected output White Box White Box Depends on Internal structure of program Depends on Internal structure of program Test all paths through code Test all paths through code

6. Testing Strategies  Static Analysis  Code Reviews  Walkthroughs  Inspections

7. Tests on image data  Gold-standard data used as basis for correctness of output  Black box tests :  Already know 'correct' output  Generate data from algorithm  Define tests that compare the two sets of data

8. Tests on image data Generic Tests: Generic Tests: Check output is of correct datatype Check output is of correct datatype Check image dimensions Check image dimensions Specific tests: Specific tests: Depends on application being tested Depends on application being tested For airway tree validation we may want to compare distance between branchpoints For airway tree validation we may want to compare distance between branchpoints For Lung Segmentation we need to compare volumes and/or distance between contours For Lung Segmentation we need to compare volumes and/or distance between contours

9. Objectives of test framework Different applications: Different applications: Regression testing after any changes Regression testing after any changes Validation of data Validation of data Flexible: Flexible: Testers only plug in specific tests Testers only plug in specific tests Test data generation and reporting of test results taken care by framework Test data generation and reporting of test results taken care by framework

10. Implementation Built on top of PyUnit : Built on top of PyUnit : Part of standard Python library for Python 2.1 and later Part of standard Python library for Python 2.1 and later Based on JUnit, a proven testing architecture Based on JUnit, a proven testing architecture Allows creation of user defined tests, aggregation into suites, running tests in textual or GUI mode Allows creation of user defined tests, aggregation into suites, running tests in textual or GUI mode

11. Writing tests with PyUnit:  Basic building blocks called Test Cases  Created by deriving from base class unittest.TestCase  An instance of a TestCase class is an object that can completely run a single test method  Fixtures  A set-up and tear down method for each test case  Many different test cases can use the same fixture  Test Suites  Test case instances can be grouped together according to the features they test  All tests can be executed together as part of a suite  Test Runner  A class whose instances run tests and report results  Can be used in text or GUI mode

12. Example: class LungSegTestCase(unittest.TestCase): def initialize(self,fname1,fname2): def initialize(self,fname1,fname2): A = AnaFile.AnaFile() # Anafile object for reading images A = AnaFile.AnaFile() # Anafile object for reading images self.data,hdr=A.read(fname1) # segmentation result self.ref_data,ref_hdr = A.read(fname2) # reference mask def setUp(self): def setUp(self): self.labels=[20,30] # left and right lung labels self.labels=[20,30] # left and right lung labels def tearDown(self): def tearDown(self): self.data = None self.data = None self.ref_data= None self.ref_data= None def test1(self):## test method names begin 'test*' def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""" """Test to see if mask image datatype is uint8.""" self.assertEquals(self.data.typecode(),'b') self.assertEquals(self.data.typecode(),'b') def test2(self): def test2(self): """Test to check that correct labels are present in mask file.""" """Test to check that correct labels are present in mask file.""" for n in self.labels: for n in self.labels: errormsg="label" + str(n) + "missing" errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg) self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

13. Example: class LungSegTestCase(unittest.TestCase): def initialize(self,fname1,fname2): def initialize(self,fname1,fname2): A = AnaFile.AnaFile() # Anafile object for reading images A = AnaFile.AnaFile() # Anafile object for reading images self.data,hdr=A.read(fname1) # segmentation result self.ref_data,ref_hdr = A.read(fname2) # reference mask def setUp(self): def setUp(self): self.labels=[20,30] # left and right lung labels self.labels=[20,30] # left and right lung labels def tearDown(self): def tearDown(self): self.data = None self.data = None self.ref_data= None self.ref_data= None def test1(self):## test method names begin 'test*' def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""" """Test to see if mask image datatype is uint8.""" self.assertEquals(self.data.typecode(),'b') self.assertEquals(self.data.typecode(),'b') def test2(self): def test2(self): """Test to check that correct labels are present in mask file.""" """Test to check that correct labels are present in mask file.""" for n in self.labels: for n in self.labels: errormsg="label" + str(n) + "missing" errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg) self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

14. Example: class LungSegTestCase(unittest.TestCase): def initialize(self,fname1,fname2): def initialize(self,fname1,fname2): A = AnaFile.AnaFile() # Anafile object for reading images A = AnaFile.AnaFile() # Anafile object for reading images self.data,hdr=A.read(fname1) # segmentation result self.ref_data,ref_hdr = A.read(fname2) # reference mask def setUp(self): def setUp(self): self.labels=[20,30] # left and right lung labels self.labels=[20,30] # left and right lung labels def tearDown(self): def tearDown(self): self.data = None self.data = None self.ref_data= None self.ref_data= None def test1(self):## test method names begin 'test*' def test1(self):## test method names begin 'test*' """Test to see if mask image datatype is uint8.""" """Test to see if mask image datatype is uint8.""" self.assertEquals(self.data.typecode(),'b') self.assertEquals(self.data.typecode(),'b') def test2(self): def test2(self): """Test to check that correct labels are present in mask file.""" """Test to check that correct labels are present in mask file.""" for n in self.labels: for n in self.labels: errormsg="label" + str(n) + "missing" errormsg="label" + str(n) + "missing" self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg) self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)

15. Application dependent tests: For lung segmentation we use an area overlap measure to test correctness: For lung segmentation we use an area overlap measure to test correctness: Check that value is above a given threshold to 'pass' test Check that value is above a given threshold to 'pass' test Similar tests can be defined with distance measures between contours Similar tests can be defined with distance measures between contours User has to plug-in appropriate tests User has to plug-in appropriate tests

16. Flow diagram:

17. Sample configuration file: [DEFAULT] # Directory of gold-standard data Refdir =../ValidationData #Directory of data generated by algorithm Maskdir =../NewData # Output file extension MaskExt = mask.img # Name of logfile directory Logdir =../logfiles [ALGORITHM] # Dir. of raw image data to run algorithm on Inputdir = /home/xyz/lung/Data # Path to executable for algorithm Execpath =../lung # Input file extension InputExt = img

18. Reporting test results: Textual output directed to log files Textual output directed to log filesSuccess:--------------------------------------------------------------------- Ran 6 tests in 88.337s OKFailure: FAIL: Left lung pixel count test (slice by slice). ---------------------------------------------------------------------- Traceback (most recent call last): File "lungsegtests.py", line 101, in test6 File "lungsegtests.py", line 101, in test6 self.failIf(fail==1,errormsg) self.failIf(fail==1,errormsg) File "unittest.py", line 264, in failIf File "unittest.py", line 264, in failIf if expr: raise self.failureException, msg if expr: raise self.failureException, msg AssertionError: significant segmentation mismatch for left lung on slices: [458, 462]

19. Status: Testing for Lung segmentation/smoothing has been implemented with the following tests: Testing for Lung segmentation/smoothing has been implemented with the following tests: Type checking of output masks Type checking of output masks Checking that all expected labels are present Checking that all expected labels are present Slice by slice area comparison for both lungs Slice by slice area comparison for both lungs Can be used by segmentation algorithms on ANALYZE images which produce labeled masks Can be used by segmentation algorithms on ANALYZE images which produce labeled masks Framework can be extended to handle other formats like airway tree definitions etc Framework can be extended to handle other formats like airway tree definitions etc

20. References: References: 1. 1. http://pyunit.sourceforge.net 2. http://www.xprogramming.com/testfram.htm Documentation and source code: CVS repository: pulmonary/TestFrameWork