1. CMS Centres for Control, Monitoring, Offline Operations and Analysis
Lucas Taylor, Northeastern University, Boston
27 November 2007
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

2. Online (data quality) monitoring
CMS Centres
CMS Control Room
Slow control, safety
Operates detectors
Calibrations
Data-quality monitoring
Data acquisition
Data transfer to Tier0
CMS Centre and
Link to Control room
Mirrors displays
Communications
Computing
Operations
Tier-0 production
Data storage / transfer
Sub detectors
Data quality monitoring (also post Tier-0)
Calibration
Good/bad runs
Software fixes
Express analysis
CMS Control
Room (Online)
at Cessy
Control of CMS
Detector Operations
CMS
Online (data quality) monitoring
Offline / Computing (data quality) monitoring
Other
Centres (?)
CMS Centre
(offline) at CERN
Control of CMS
Offline / Computing Operations
CERN
Tier-0
IN2P3
FNAL
CNAF
PIC
ASGC
GridKA T2
RAL T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

3. CMS Centre at CERN, LHC@FNAL, …
Large rooms to co-locate offline experts
Physicists working on DQM, calibration, rapid data analysis
Expert computing and offline operations teams
IT facilities include
Monitoring displays, IT systems
Web-based applications
Give all CMS collaborators live access to latest information
Communications systems connecting
CMS Control Room
CMS Centre, others such centres (DESY?)
CMS institutes
Tier-1 and Tier-2 centres
Grid operations centres (?)
CERN Control Centre (CCC) for the LHC
CMS Centre will be similar
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

4. CMS Centre at CERN Located in the former PS Main Control Room
Main Room:
300 sq. m
To be refurbished
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

5. CMS Centre at CERN Main room: 25-50 people, 22 consoles, 110 screens
Computing Operations Room ~ 15 people (done)
New offices: ~250 people
Video-conferencing (auditorium and ~ 6 meeting rooms)
Outreach room, visitors room, lockers, rest area, kitchen…
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

6. CMS Centre at CERN Will be in the former PS Main Control Room (below)
Machine controls now in the Prevessin “CERN Control Centre” (CCC)
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

7. CMS Centre at CERN Main Room Outreach Room Meeting Rooms Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

8. Lower row of screens for working
CMS Centre at CERN
Lower row of screens for working
Top row of screens
for monitoring
(e.g. displays from
Control Room)
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

9. Relocation of existing personnel / removal of everything in room
CMS Centre at CERN
June 2007
Relocation of existing personnel / removal of everything in room
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

10. CMS Centre at CERN July 2007 Major upgrade of electrical power
40 kW iced-water cooling system with 22 fan-coil heat exchangers
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

11. Opened window from conference room
CMS Centre at CERN
August 2007
Opened window from conference room
100 fixed network connections (Cat-6 cable) on GPN with 2*10 Gbps uplinks
25 analogue
phone lines
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

12. CMS Centre at CERN Sept-Oct 2007 Ventillation / heating in ceiling
Concrete: coring for services and closure of fire-breaks
Acoustic insulation
in ceiling
Repairs / levelling of false floor
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

13. Carpet tiling (permits re-lifting of false floor)
CMS Centre at CERN
Nov 2007
Painting / repairs
throughout
Fire safety systems
Video-conferencing
Carpet tiling (permits re-lifting of false floor)
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

14. CMS Centre at CERN Nov 2007 New door (fire exit) Lighting repairs
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

15. CMS Centre at CERN Nov 2007 Construction of consoles Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

16. CMS Centre at CERN Nov 2007 Construction of consoles Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

17. CMS Centre at CERN Nov 2007 (today) Construction of consoles
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

18. CMS Centre Technical Installations
Past: PS Main Control Room
Present: Installation of CMS Centre
Future (Mar 2008): CMS Centre
Similar to this: LHC Control Room
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

19. LHC@FNAL Completed: Feb. 8, 2007 Open House: Feb. 12, 2007
Started operations this year
Joint centre for CMS & LHC
Builds on FNAL Remote Operations Centre used in CMS Cosmic challenge
features
4 CERN-style consoles (8 workstations)
Videoconferencing for 2 consoles
Webcams for remote viewing of room
Secure access, secure network
High Definition (HD) Systems
Videoconferencing, Webcams…
Role Based Access for LHC controls
Screen Snapshot Service (SSS)
Completed: Feb. 8, 2007
Open House: Feb. 12, 2007
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

20. LHC@FNAL Tier-1 operations team Detector groups, e.g. Si tracker
Responsible for FNAL Tier-1
Weekday, business hour shifts
Central support for seven Tier-2 centers (Universities)
Detector groups, e.g. Si tracker
Remote shifts, Feb–Jun 2007
Worked with CERN tracker on
Data Quality Monitoring
Data transfer/bookkeeping
Event display, DCS,....
Outreach
Visits and outreach video
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

21. CMS Operations CMS “Data Operations” teams
Main team at CERN (15 people)
Second main team at
Ensures CMS workflows work ~24 / 7
Will also use CMS Centre when ready
Computing
operations
Building 8 1st Floor
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

22. CMS Operations
(Obvious) strategy for monitoring and control: Web-based GUIs
Computing centres and CMS collaborators spread globally
Location and platform independence
CMS “WebTools” project
Builds on Yahoo (YUI) toolkit
Currently in use for
Data production request system
Data transfers / management
Phedex, DBS, SiteDB …
More applications coming (DQM)
CMS CSA06:
~1 PB in ~1 month
>12 PB in
~6 months
Related CHEP Talks
No Computing Operations at CMS Facilities
No. 290 CMS Tier0 - design, implementation and first experiences
No. 369 CMS Experiences with Computing Software and Analysis Challenges
No. 370 Development of the Tier-1 Facility at Fermilab
No. 266 CMS Offline Web Tools
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

23. CMS Operations Screen Snapshot Service (SSS)
Runs on arbitrary machine (e.g. shifter in control room or operator console)
Takes snapshots of screens and exports them to Web
Remote people can watch monitoring screens with a Web browser
Used by CDF, CMS, maybe LHC
Web
Browser(s)
requests
snapshots
Snapshot
Service
snapshots
Monitored
Application(s)
Related CHEP Talks
No Computing Operations at CMS Facilities
No. 290 CMS Tier0 - design, implementation and first experiences
No. 369 CMS Experiences with Computing Software and Analysis Challenges
No. 370 Development of the Tier-1 Facility at Fermilab
No. 266 CMS Offline Web Tools
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

24. CMS Operations: Data Quality Monitoring
Web-based Graphical User Interface
Building on CMS WebTools
Top-level (shift) views
Reference plots
Repository of expert plots
Universal uniform access
Control Room, offline centres, institutes, Web café …
Related CHEP Talks
No. 221 CMS Online Web Based Monitoring
No. 279 The Run Control and Monitoring System of the CMS Experiment
No. 253 CMS Event Display and Data Quality Monitoring for LHC Startup
No. 266 CMS Offline Web Tools
No. 434 Data Quality Monitoring and Visualization for the CMS Silicon Strip Tracker
No. 268 Data Quality Monitoring for the CMS Electromagnetic Calorimeter
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

25. CMS Operations: Data Quality Monitoring
Deployed and refined in monthly commissioning runs
Will gradually migrate non-Web applications to Web
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

26. CMS Operations: Data Quality Monitoring (Iguana)
Iguana widely used for
Magnet test & cosmic challenge
Online commissioning
Offline debugging / analysis
Nice pictures, press, outreach
Web-based
Iguana prototype
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

27. Communications Strategic importance for 2500 remote physicists
Displays
DQM web clients (location independent)
Local systems (e.g. trouble-shooting)
SSS snapshots (mirror remote screens)
Phone
1 headset + speaker-phone per console
Conference phones
Videoconferencing
Main room: open connection to Control Room, CMS Centre, …
Also adjacent meeting room(s)
WebCams (on big screens)
Maintain sense of proximity
Use with phone for point to point “video conferencing”
iMac - under consideration
Computer, screen, camera, microphone
iChat for IM and few-person video-conferencing
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

28. Excellent CERN / IT support !
Communications
Robert Aymar
CERN Director General
Jim Virdee
CMS Spokesperson
HD video communications Recent demonstration of effectiveness CERN <-> USA
inaugurated with live video link to CMS and LHC
Crucial for remote physicists, especially as we start to take data and do physics analysis
Excellent CERN / IT support !
Ray Orbach, US Under Secretary of Science at Fermilab
CMS surface hall
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007

29. Summary CMS Centres are currently being established Functions
Main “CMS Centre” at CERN (Meyrin)
Remote centres: probably DESY, maybe others (?)…
Functions
Offline data quality monitoring, calibrations, express analysis
CMS Computing / Offline Operations … liaison with Tier0, 1, 2 sites
Emphasize communication and access to information
Enable all 3000 CMS collaborators to play their part in commissioning … and hence …
Increase CMS competitiveness when we start taking LHC data in July 2008 at a new energy frontier
Lucas Taylor
WLCG Meeting, CERN, 27 Nov 2007