Robustness Initiative Jeff Kern NRAO

CASA Robustness and Reliability Number one priority from the CASA User Survey was reliability. – Survey did not follow up on this topic Increased attention recommended by CUC in 2014 report. – “… experience of CUC members is that CASA throws frequent segmentation faults or other errors…” – “… no less important than preventing segmentation faults is to avoid implementation errors.” And from the NRAO User Committee: – “…the UC was pleased to note that there seems to an increasing awareness among the NRAO CASA team that their CASA experience and the experience of the typical users are different…” 2015 CASA Users Committee Face-to-Face2

CASA Types of Issues Documentation – Sequence of commands to accomplish goal does is unclear – Code does not do what user expects Implementation Defects – Code implementation that does not behave as it should. Algorithmic Errors – Algorithm does not yield correct results. Algorithmic Shortcomings – Algorithm applied to problem outside of intended solution space CASA Users Committee Face-to-Face3

CASA Addressing the Issues: Documentation See previous talk 2015 CASA Users Committee Face-to-Face4

CASA Addressing the Issues: Implementation Defects Removing obsolete or unused interfaces and implementations – Decreased paths through codebase. Continued refactoring to pay off technical debt 2015 CASA Users Committee Face-to-Face5 Improving test coverage – Currently underway, shift in focus from system test to module tests. – Automation, and on every supported OS. – Calibration – Clean – Data Access (VI/VB2) – Filler – Flagging – Image Analysis

CASA Addressing the Issues: Algorithmic Errors Often the result of unexpected “features” in input data. – More focused regression testing will prevent relapse – Stricter interpretation of standards (and checking) Improved internal interfaces and design: – Part of repaying technical debt More modular testing: – Decrease the parameter space to be tested – Smaller synthetic datasets for comparison Improved formal process – Stricter requirement definition – Final validation before delivery to community 2015 CASA Users Committee Face-to-Face6

CASA Addressing the Issues: Algorithmic Shortcomings Example: Cygnus in the third sidelobe of a VLA observation. – Reported error: Gaincal fails to give good solution. User may not even recognize that the problem is outside the domain space. – Education can help, but very time consuming. – Improved documentation Segmentation faults in these cases are unacceptable! – Creating useful and informative error messages may be the best we can do CASA Users Committee Face-to-Face7

CASA Planning Currently working on: – Removing obsolete code. – Improving test coverage and building infrastructure to run this routinely. – Addressing known technical debt on small scale Larger issues are deferred, for now. – Assessing and implementing tool chain changes to enforce standards. Identifying metrics that will show improvement. Suggestions? Next steps: – Addressing the larger scale technical debt – Improving error [and logging] messages – Increased “self protection” for non-compliant data sets CASA Users Committee Face-to-Face8

CASA Software Quality The fundamental issue is software quality, and choosing higher quality over more functionality. One recommendation of recent consultancy is to institute the role of Quality Assurance in the CASA development team. – Ensure that internal processes are followed. – Dedicated effort that will not be devoted to adding functionality or usability to CASA package CASA Users Committee Face-to-Face9

CASA A bit of good news 2015 CASA Users Committee Face-to-Face10