1. Dave Barney, CERN

2. Tracking Tools: Introduction  Main Purpose: to keep track of all incidents & interventions at point 5  Interlocks – reasons & follow-up  Hardware changes (due to problems or enhancements)  Firmware changes (bug fixes & enhancements)  Software changes (bug fixes & enhancements) – mainly for XDAQ, DQM, PVSS etc.  Procedural changes  Secondary purpose: perhaps extend functionality to help track CMSSW software changes? Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 2

3. Tracking Tools: Requirements  Summary of planned intervention  Reason, actions to be taken  Decision-making process  Approval/rejection of intervention  Scheduling of the intervention  Follow-up of the intervention  Incident reporting (particularly interlocks)  Timeline tracking, e.g.  Grouping of interventions for technical stops etc. Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 3

4. Example Use Cases  CASE 1: bug fix of DCC (=FED) firmware  REQUIREMENTS: Version tracking (history) with details of the bug and how it has been fixed; scheduling of implementation; definition & scheduling of testing/verification; list of people involved  CLOSURE: after verification (possibly involving offline analyses)  CASE 2: replacement of faulty power supply  REQUIREMENTS: tracking of hardware items (replacement and faulty one); scheduling of replacement; radiation measurements; scheduling of testing (if appropriate); follow-up of faulty module through electronics pool or company; history of similar faults (i.e. is the real problem higher-up the chain?); list of people involved  CLOSURE: two parts: after verification of new module; after diagnosis/repair of faulty module Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 4

5. Example Use Cases (cont.)  CASE 3: upgrade of safety system (e.g. additional temperature sensors)  REQUIREMENTS: version tracking (history); summary of new functionalities with decision making procedure; scheduling of hardware/firmware(PLC)/software work; scheduling of testing/verification; list of people involved  CLOSURE: if accepted, after verification of new functionality Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 5

6. Tracking Tools Examined  Savannah (thanks to Diogo di Calafiori, ECAL)  CMS Tracker twiki-based work-packages etc. (thanks to Karl Gill)  TRAC (thanks to Andre David, ECAL)  JIRA (thanks to Alick MacPherson (idea) and Niall Stapley (implementation), LHC Operations)  DocDB (thanks to Eric Vaandering)  EDMS Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 6

7. Selected Tool - JIRA  JIRA – commercial product used by LHC operations & software development  Supported at CERN (site-wide license) and uses CERN single sign-on for authentication  Fully user configurable via web interface  Plug-in modules for e.g. interface to SVN  Use of e-groups for permissions, sign-offs etc.  Easily group interventions for technical stops etc. Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 7

8. JIRA use within ECAL  Instance set-up for ECAL http://issues.cern.ch (project: CMS ECAL) (only works within CERN network or via tunneling, for security purposes – but your own instance could be on your own server etc.) http://issues.cern.ch  Relatively easy custom-designed workflow seems to work for all ECAL interventions (so far)  Full notification scheme included: all task assignees, ECAL OM group (== Tracker C&O group) etc.  Whenever an issue is created  Whenever an issue is resolved (or cannot be resolved)  If an issue has not had any work done and is urgent  Etc. Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 8

9. JIRA Workflow for ECAL Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 9

10. Example of email notifications Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 10

11. JIRA use within ECAL (cont.)  “versions” reflect when we want the issue to be solved (e.g. “next period without beams” or “technical stop December 2010”)  “statuses” reflect the progress in solving the issue (e.g. “Open”, “In progress”, “monitoring”, Closed”)  “Components” are the individual elements within ECAL (e.g. “EB HV supply”, “Laser monitoring hardware”, “DCC”) and each component has an assigned responsible person Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 11

12. JIRA – more information  https://cms-docdb.cern.ch/cgi- bin/IntDocDB/ShowDocument?docid=2852 https://cms-docdb.cern.ch/cgi- bin/IntDocDB/ShowDocument?docid=2852  (tutorial, workflow examples etc.) Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 12

13. Backup – other tools examined Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 13

14. Savannah (e.g. https://savannah.cern.ch/projects/cmsecaldcs)https://savannah.cern.ch/projects/cmsecaldcs  Pros  Fully supported at CERN  Used already (successfully) CERN-wide, including ECAL DCS/ESS  Ticketing system to follow status of work (including threads)  Activities can be grouped  Timeline possible (not currently used by ECAL DCS)  Cons  No “decision making” functionality  No versioning (for firmware/software) Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 14

15. CMS Tracker – twiki etc. (http://twiki.cern.ch/twiki/bin/view/CMS/TrackerOperations)http://twiki.cern.ch/twiki/bin/view/CMS/TrackerOperations  Pros  Very well thought-out set of procedures  Decision-making procedures built-in to work-package documents etc.  Encourages people to think carefully about any planned interventions  Managers must read/acknowledge proposals  Grouping of activities through work-packages  Twiki used for discussion threads etc.  Also use Savannah for DCS/DSS  Cons  No central support (apart from twiki) – fully manual solution  No ticketing system  No timeline  No versioning  No search/ordering facility Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 15

16. DocDB (https://cms-docdb.cern.ch/cgi-bin/DocDB/DocumentDatabase)https://cms-docdb.cern.ch/cgi-bin/DocDB/DocumentDatabase  Pros  CMS-wide solution for document storage (including for ECAL!)  Approval process built-in  Grouping of activities (documents) possible  Versioning built-in  Powerful search facility  Cons  No ticketing system & no threads  No timeline  Poor user-interface for document retrieval Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 16

17. EDMS (http://edms.cern.ch/nav/CMS-0000000001)http://edms.cern.ch/nav/CMS-0000000001  Pros  Fully supported at CERN  Used extensively by CMS for engineering documents  Versioning built-in  Approval procedures built-in  Cons  No ticketing system (no threads)  No timeline  Slow! Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 17

18. TRAC (https://svnweb.cern.ch/trac/cmsecalp5)https://svnweb.cern.ch/trac/cmsecalp5  Pros  Fully supported at CERN (in fact every SVN repository has an associated TRAC!)  Ticketing system to follow status of work (including threads)  Versioning built-in  Timeline built-in  Grouping of activities  Approval procedure built-in (need to define workflow)  Powerful search  Easily extendable to use for CMSSW  RSS feed built-in  Twiki built-in  Cons  Not (yet) used by other CMS groups  Additional functionalities possible but needs central CERN IT intervention Tracking Tools Evaluation, Dave Barney, ECAL OM Meeting, 12th May 2010 18