8th Sakai Conference4-7 December 2007 Newport Beach What does Quality Assurance Mean to Sakai? Alan Berg Megan May Seth Theriault

Topics What does a QA Director do? QA Today –Who is in the WG –Types of testing currently done and release activities –Strides forward Resources available –QA Network 2.5 Release Activities Update QA Tomorrow Questions

Role of QA Director Coordination! –Help bring parties interested in the same testing efforts together –Foster initiatives that support quality throughout the community –Assist in bridging the gap between test and design & development teams –Ensure release activities are transparent

QA Today Development Testing & Software Release Requirement Gathering and Design Focus on Functional Testing

Who is the QA WG? People 37 Institutions 11 Countries People 10 Institutions 4 Countries People 17 Institutions 5 Countries People 17 Institutions 4 Countries... How do they get involved?

Strides Forward Deployment testing on wider variety of platforms Integration of Accessibility WG activities in general QA cycle Expanding/updating library of test scripts Reflection on Release process –

QA Network Amsterdam Boston U Cambridge Cape Town Columbia Indiana rSmart Georgia Tech Coming soon servers in Japan and Australia

Collab How to be in the know!

Jira

Confluence

Perceived Quality Level

Quality, like beauty, is clearly in the eyes of the beholder.

Perceived Quality Poll

QA Tomorrow Sakai Community Concerns –Several prominent campuses experience serious performance problems with Release Update –Further refinement of process –Goals for 2.5

Paths Forward Common set of testing vocabulary –Unit Testing –Integration testing –System Testing –Performance Load testing ( ) Integration testing Development Testing & Software Release Requirement Gathering and Design

Roadmap for Tomorrow

Metrics to Describe Quality – 85% Verification – 85% Verification (1,345 JIRA Tickets) (611 JIRA Tickets) (504 JIRA Tickets) – 82% Verification – Past bug verification

Metrics to Describe Quality

Static code Review Search for bug patterns What to process: Source code, Binary Two types of bug pattern –Generic Classic (buffer overflow, input validation) –Contextual Language specific Framework specific Configuration specific Proximity effect

Hinting nightly

Random Example

Duplicate Code

Do we wish to use the reports more aggressively? Are the developers taking note? –Bugs reported are not always correct –Bugs found not always important –But duplicate code, unit test coverage, and failing to deal properly with exceptions hint strongly at quality. –Not enough Contextual searching for Sakai and spring configuration specific issues –Reports do not explain enough –Answer probably not, but thankfully trends in the marketplace will improve the quality of results overtime.

A year on Update prototype –Less noise more explanation –Graphical tracking of change Recommend Eclipse Plugins Look at emerging products See as part of a series of QA processes. Agree on metrics to act on and then act.

A good book to read Secure Programming with Static Analysis –Brian Chess, Jacob West Addison Wesley ISBN: Includes CD with software and examples.

What you can do Pledge Resources –What happens in the community has a direct local impact –Easy point of entry into Sakai –All skill sets are needed Discussion –Why hasn’t this been a priority? –What are obstacles in the way and how do we remove them?

Questions

Other Sessions of Interest 2.5 QA Workshop (Thursday, 13: :30 in Salon 5) The Automated Testing Conundrum in Sakai (Thursday, 15: :10 in Salon 5)