1. Quality Control in the dCache team.
Owen Synge
dCache Team

2. Overview
Quality Control in the dCache team.
Test Driven development (Our Tests).
Unit Tests and Code Analysis.
Developer Code Flow.
Code Review.
Automated build and tests.
Upload to Web Site.
Recommended Releases.
What can be improved?

3. Test Driven development (Our Tests)
Static Code analysis.
FindBugs.
Unit Tests.
With jUnit framework.
Functional tests.
DcacheTestSuite.
S2 SRM test Suite.
G2 dcacheTestSuite.
Port of dCache test suite to jUnit reporting for Hudson.

4. Allows reporting on many coding errors.
Hudson Driven System
Findbugs.
Allows reporting on many coding errors.
Very good for coding style suggestions.
Very low impact on code produced by developers.
Unit tests via jUnit.
Graphs generated by Hudson are very nice.
Main reason G2 dCache test suite ported to jUnit.
Winner 2008 Dukes Choice Java Development tool.
Score Board.
Not so useful (though often right).
Points go up and down to much.
Owen Synge

5. Unit Tests and Code Analysis
Hudson Showing jUnit and FindBugs Output

6. FindBugs Provides good results. Few fixes have come from FindBugs.
Code Analysis.
FindBugs Provides good results.
Few fixes have come from FindBugs.
Owen Synge

7. Developers check code against Unit tests.
Developer Code Flow
Developers check code against Unit tests.
Patch against appropriate branches.
Submitted to Review Board.
Forcing details of Patch details to be documented.
Often use Mercurial to manage patches.
Patch Reviewed.
Can cause friction (Usually Positive).
Approved Patches are applied to SVN.
Post commit SVN hook triggers Hudson.
SVN comment taken from Review Board Patch
Also includes Review Board.
Owen Synge

8. Code Review – Surprisingly works
Review Board : My Dashboard of Pending Patches
Owen Synge

9. Code Review : Annotating Code
Reviewer can annotate the code.
And provide links to details of comments.
Owen Synge

10. Detailed Comments. Comments on Comments. No commit till reviewed.
Details of Code Review
Detailed Comments. Comments on Comments.
No commit till reviewed.
Owen Synge

11. Shows all errors and warnings during build.
Hudson Driven Build
Shows all errors and warnings during build.
Owen Synge

12. Hudson „Artefacts“ are the dCache released code.
Release to Customers
Hudson „Artefacts“ are the dCache released code.
Uploaded to Public Web Site for release.
Detailed change log generated from Review Board.
Sites customers pick their version.
Upgrading dCache is work.
Stateful service, downtime announced and planned.
Based upon recommendations and change log.
Driven by features required.
Tier 1's have complex requirements forcing upgrade.
DESY HERA version is 5 Years old.
But may upgrade soon for Chimera.
Owen Synge

13. Hudson Deployment Testing
Hudson „Projects“ run on remote hosts.
Triggered by „Upstream Projects“ Scripts or time.
Owen Synge

14. Complex Multi Node Deployment.
Hudsons Agents can launch Scripts on multiple hosts.
These scripts run tests and virtual machines (SL4).
Leads to fresh install testing, via dCacheConfigure.sh* and fresh install testing.
Hudson
Xen Manager
dCache Node
dCache Node
dCache Node
Hudson Agent
Hudson Agent
Hudson Agent
Test
Suites
WorkerNode Node
Hudson Agent
Hudson Agent
Hudson Agent
*YAIM now uses dCacheConfigure to configure dCache.
Owen Synge

15. Hudson Multi Node Deployment
Graph display of test results on deployment.
Owen Synge

16. Certification Testbed is 4 single nodes.
Release to CERN Glite
Certification Testbed is 4 single nodes.
LCG tools tested against them.
Test suites run against nodes.
Proves critical in finding integration bugs.
Information system.
LCG tools.
Confidence that will work on Grid.
Takes a lot of effort.
Due to complexity of configuring SRM Spaces.
Impossible to Automate, far to multidimensional
Owen Synge

17. Recommended dCache Versions.
Recommended dCache release.
Passes upgrade tests.
Many Tier 1's upgrade to this.
Some Tier 2's upgrade to this.
My repository release. (Yum/Apt repository)
Passed SA3 Certification Test Bed tests.
Announce on user mailing list.
Most Tier 2's use this release.
Glite repository release.
Cautious Tier 2's use this release.
Owen Synge

18. Knowing if releases get faster or slower. Archive of old releases.
What Could Be Better.
Performance Tests.
Knowing if releases get faster or slower.
Archive of old releases.
Some times people clean up releases!
Configuration Management.
Flexibility of dCache is too great.
Cant check all features. (As configuration is code).
Less variety of dCache versions in production.
Better build scripts (Ant and Make have limits)
Better dependency handling.
Owen Synge

19. Not Having the central SA3 gang. Critical for bugs in clients.
What Could Be Worse.
Not Having the central SA3 gang.
Critical for bugs in clients.
Having an outside Party to test your deployment.
SRM Space management is important here.
Comparing with DPM.
Finding / Offloading communication to third parties.
More feature creep in requirements, clients.
Dependency issues.
We Statically link and include all dependencies
But we don't manage them well (yet).
Political Releases. (non functioning but required)
Owen Synge

20. Extending Code Review process. Golden Releases.
Changes to Our Process
Design Review.
Extending Code Review process.
Golden Releases.
6-12 Months of Support.
Others will have short term support.
Time Based Releases.
We tried it before.
We intend to be more strict about this.
Owen Synge

21. Test Driven Development Using available tools.
Summary
Test Driven Development
Using available tools.
SVN, Hudson, Review Board are all good tools.
Sites pick version. (And if they run your software).
Features demanded by customers (Tier 1's).
Based on Change log (Of critical importance).
Which is based on Review Board. (Very Good Tool).
Trust of recommendations (Honesty and Work).
SA3 helped find Grid Integration Bugs. (Which client?)
And streamlined fixing them and communication
Their own upgrade tests. (Only trust us so much).
Tier 2's don't have manpower. (So have to be cautious).
Owen Synge