1. CMS Centres for Control, Monitoring, Offline Operations and Analysis
Lucas Taylor, Northeastern University, Boston
Patricia McBride, Kaori Maeshima, Erik Gottschalk Fermilab
CHEP’07, Victoria
2-7 September 2007
Abstract
The CMS experiment is about to embark on its first physics run at the LHC. To maximize the effectiveness of physicists and technical experts at CERN and worldwide and to facilitate their communications, CMS has established several dedicated and inter-connected operations and monitoring centers. These include: a traditional Control Room at the CMS site in France; a CMS Centre for up to 50 people on the CERN main site in Switzerland, and remote operations centers, such as the center at Fermilab.
We describe how this system of centers coherently supports the following activities: CMS data quality monitoring, prompt sub-detector calibrations, and time-critical data analysis of express-line and calibration streams; and operation of the CMS computing systems for processing, storage and distribution of real CMS data and simulated data, both at CERN and at offsite centers.
We describe the physical infrastructure that has been established, the computing and software systems, the operations model, and the communications systems that are necessary to make such a distributed system coherent and effective.
Lucas Taylor

2. LHC & CMS startup LHC CMS First collisions at 14 TeV
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
LHC & CMS startup
Completion of magnet cryostating and tests
Descent of last magnet
Sector 7-8 cooled to 1.9 K along 3 km
All technical systems commissioned
to 7 TeV operation & machine closed
LHC
Beam commissioning starts
First collisions at 14 TeV
2010
2008
2006
2007
2009
2005
(0.1 – 1) fb-1
(1 – 10s) fb-1
(10s - 100) fb-1
CMS commissioning, data quality
monitoring, calibration, alignment,
Offline computing operations …
CMS
Complete CMS closed and ready
for extended physics run
CSA’07 – final Computing Challenge
Cosmic challenge on surface
Lucas Taylor

3. LHC & CMS startup - human implications
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
LHC & CMS startup - human implications
Jim Virdee described the challenges of the new LHC machine, new CMS detector and new physics …
Fabiola Gianotti (Interlaken) identified some of the corresponding human difficulties we will face
“O(10**3) physicists in panic-mode using and modifying the software and accessing the database, GRID …”
“…at the beginning they will be confronted with most atypical (and stressful) situations, for which a lot of flexibility will be needed”
I’ll describe how CMS Centres address this
By co-locating sub-detector offline experts (CERN, FNAL…)
By hosting CMS Computing operations teams
By giving all 3000 collaborators live access to monitoring information
By ensuring communications systems are effective
Lucas Taylor

4. CMS Centers CMS Control Room Control Room Operates CMS Slow control
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Centers
CMS Control
Room
Traditional
CMS
Control of CMS
Detector Operations
Control Room
Operates CMS
Slow control
Data acquisition
Data transfer to Tier-0
Sub detectors:
Operate detector
Calibrations
Data-quality monitoring
Constants to HLT
Online monitoring
CERN
Tier-0
IN2P3
FNAL
CNAF
PIC
ASGC
GridKA
RAL T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2
Lucas Taylor

5. CMS Centers CMS Control Room CMS Centre and LHC@FNAL CMS Centre
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Centers
CMS Control
Room
CMS Centre and
Link to Control room
Mirror displays
Communications
Computing
Operations
Tier-0 production
Data storage / transfer
Sub detectors
Data quality monitoring (also post Tier-0)
Calibration
Good/bad runs
Software fixes
Express analysis
Novel
CMS
Control of CMS
Detector Operations
Online monitoring
CMS Centre
(at CERN)
CERN
Tier-0
Control of CMS
Computing Operations
IN2P3
Offline monitoring
FNAL
CNAF
PIC
ASGC
GridKA T2
RAL T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2 T2
Other
centres (?)
Lucas Taylor

6. Completed: Feb. 8, 2007 Open House: Feb. 12, 2007 LHC@FNAL
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
Started operations this year
Joint centre for CMS & LHC
Builds on FNAL Remote Operations Centre used in CMS Cosmic challenge
features
4 CERN-style consoles (8 workstations)
Videoconferencing for 2 consoles
Webcams for remote viewing of room
Secure access, secure network
High Definition (HD) Systems
Videoconferencing, Webcams…
Role Based Access for LHC controls
Screen Snapshot Service (SSS)
Completed: Feb. 8, 2007
Open House: Feb. 12, 2007
Lucas Taylor

7. LHC@FNAL Tier-1 operations team Detector groups, e.g. Si tracker
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
Tier-1 operations team
Responsible for FNAL Tier-1
Weekday, business hour shifts
Central support for seven Tier-2 centers (Universities)
Detector groups, e.g. Si tracker
Remote shifts, Feb–Jun 2007
Worked with CERN tracker on
Data Quality Monitoring
Data transfer/bookkeeping
Event display, DCS,....
Outreach
Visits and 12 minute outreach video and photos from CERN
Lucas Taylor

8. Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Centre at CERN
Mid-way through construction; when complete will include
New office space for ~250 people
Main Monitoring & Operations Room (300 sq. m)
22 consoles with ~5 screens each
CMS Computing Operations Rooms
Already in use by ~15 operators
Meeting Rooms
Auditorium plus ~ 6 smaller rooms
All with Polycom conf. phone, some with Tandberg VC equipment
Miscellaneous rooms: outreach, WiFi visitors room / training centre, rest area with kitchen, locker room…
Lucas Taylor

9. CMS Centre at CERN Main Room Outreach Room Meeting Rooms Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Centre at CERN
Main Room
Outreach Room
Meeting Rooms
Lucas Taylor

10. Lower row of screens for working
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Centre at CERN
Lower row of screens for working
Top row of screens
for monitoring
(e.g. displays from
Control Room)
Lucas Taylor

11. CMS Centre at CERN Will be in the former PS Main Control Room (below)
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Centre at CERN
Will be in the former PS Main Control Room (below)
Machine controls now in the Prevessin “CERN Control Centre” (CCC)
Lucas Taylor

12. CMS Centre at CERN … today
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Centre at CERN … today
Detailed requirements study
Technical studies completed
Ergonomics, acoustics, fire, radiation, lighting, etc.
Green light for construction
Lucas Taylor

13. CMS Centre at CERN … today
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Centre at CERN … today
Technical installations underway
22 desks, ~110 screens
Network (2 * 10Gbps uplinks)
Lots of wireless
40 kW electrical power
Air-conditioned
under-floor iced water to 20 peripheral fan coil units
Acoustic insulation
Carpet, fibre-glass in ceiling and consoles
Lighting
Natural, strips, spots
Dimmable and zoned
Outreach displays
Lucas Taylor

14. Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Centre at CERN
When completed (March 2008) the CMS Centre will look very similar to the CERN-LHC Control Centre shown below
Lucas Taylor

15. CMS Computing Operations
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
CMS Computing Operations
Related CHEP Talks
No Computing Operations at CMS Facilities
No. 290 CMS Tier0 - design, implementation and first experiences
No. 369 CMS Experiences with Computing Software and Analysis Challenges
No. 370 Development of the Tier-1 Facility at Fermilab
No. 266 CMS Offline Web Tools
CMS “Data Operations” teams
Main team at CMS Centre, CERN (15 people)
Second main team at
Ensures CMS workflows function ~24 / 7
Control and monitoring systems
Increasing use of “WebTools”
Initiate transfers, data placement and removal, production systems…
Communications in-person and by telephone and IM with shared desktops
Web, some VNC, maybe Webex/EVO (?)
Screen Snapshot Service (SSS) for exporting arbitrary displays to the Web
Used by CDF, CMS, maybe LHC
Web
Browser(s)
snapshots
requests
Snapshot
Service
snapshots
Monitored
Application(s)
Lucas Taylor

16. Data Quality Monitoring
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
Data Quality Monitoring
Web-based DQM GUI gives access to information in Control Room, offline centres, institutes, Web café, iPhone…
Live shifter views - canned with reference histograms
Expert exploration of repository of 1000s of detailed histograms
Both real-time and offline data
DQM systems being refined with practical experience from monthly commissioning runs
Related CHEP Talks
No. 221 CMS Online Web Based Monitoring
No. 279 The Run Control and Monitoring System of the CMS Experiment
No. 253 CMS Event Display and Data Quality Monitoring for LHC Startup
No. 266 CMS Offline Web Tools
No. 434 Data Quality Monitoring and Visualization for the CMS Silicon Strip Tracker
No. 268 Data Quality Monitoring for the CMS Electromagnetic Calorimeter
Lucas Taylor

17. Communications WWW Displays Phone Videoconferencing WebCams
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
WWW
WebTools,
DQM, SSS, …
Communications
Displays
Local applications, DQM web clients, SSS mirrors
Phone
Headset per console
Conference phones
Videoconferencing
Adjacent meeting room(s)
Possibly in main room
WebCams
Maintain sense of proximity
Use with phone for point to point “video conferencing”
iMac - under consideration
Computer, screen, camera, microphone and
iChat for IM and few-person phone-/video-conferencing
CMS Control
Room
mirror
mirror ?
CMS Centre
at CERN
mirrors
Related CHEP talks
No. 35 Collaborative Tools and the LHC: An Update
No 408 Shaping Collaboration 2006: Action Items for the LHC
Lucas Taylor

18. Summary CMS Centres are currently being established
Lucas Taylor
CHEP'07, Victoria
2-7 Sept 2007
Summary
CMS Centres are currently being established
“CMS Centre” at CERN, possibly others …
These centres will enhance communication and access to information, thereby helping all 3000 CMS collaborators to play their part in commissioning
Offline data quality monitoring, calibrations, express analysis
CMS Computing Operations
…which will, in turn, increase CMS competitiveness when we start taking LHC data in July 2008 at a new energy frontier
Lucas Taylor