1. KEKB Accelerator Control System KEKB Controls Group 2001-08-29 by T. Katoh

2. Control System l Control Computer System l Timing Control System l Beam Gate Control System l Personnel Safety Control System l Communication System l Control Room and Operators Consoles

3. Contents l Construction History of KEKB Accelerator Control System l KEKB Control System Design l System Hardware Configuration l System Software Configuration l Central Control Room

4. Construction History 1994.4Started System Design for KEKB 1995.2Controls Systems Evaluation 1995.5Committee Recommendation to adopt EPICS 1995.8First Version of Computer System Specifications 1996.1Started to Dismantle TRISTAN Main-Ring 1996.3Final Version of Specifications 1996.6Opened the Bids 1997.3abco1 and 15 IOCs were Installed 1997.4Hardware Specifications were Fixed 1997.10BT Lines were Commissioned 1998.3The Rest of IOCs were Installed

5. Construction History(contd.) 1998.4Started Software Installation with Link- Persons and Engineers from the Company 1998.12Commissioning of KEKB Accelerators 1999.3Commissioning with the BELLE Detector 1999.4Started Designing PF-AR Control System 2000.10Installation of abco2 and 10 IOCs for PF-AR 2001.2Started to Dismantle PF-AR Devices 2001.7Started to Install or Re-install Equipment 2002.1Commissioning of PF-AR

6. KEKB Control System Design l System Requirements l Constraints l Basic Concepts l System Architecture

7. System Requirements l Sources –Equipment Groups Magnet and Power Supply RF Beam Monitor Vacuum Beam Transport Feedback, Facilities, Physics, etc. –Operations Group –Accelerator Physicists or Commissioning Group –Controls Group

8. System Requirements(contd.) l Requirements –All the data that are possible to take should be taken. –All the data that are taken should be saved for later analyses. –All the operation should be recorded for later inspection. –All the machine parameters and information about the machine components should be stored in the database.

9. System Requirements(contd.) l Requirements(contd.) –The Man-Machine Interface should be Operator- Friendly. –The Programming Environment should be Programmer-Friendly. –The Overall Response Time to an Operator’s Request should be less than a Second unless the Progress of the Process is Indicated.

10. Constraints l Use CAMAC as an Equipment Interface l Schedule –Control Systems Final Design January, 1995 –Removal of MR equipment January, 1996 –Control Systems Installation September, 1996 –Completed Hardware Installation March, 1997 –KEKB CommissioningDecember, 1998

11. Constraints(contd.) l Limited Man-Power –9 KEKB Controls Group Members 3 Physicists and 6 Engineers –12 Link-Persons from Other Groups 2 from Magnet and Power Supplies 2 from RF 2 from Vacuum 2 from Beam Transport 2 from Beam Monitor and Feedback 1 from Linac, 1 from Physics

12. Basic Concepts l Standard Model Architecture l International Standards –CAMAC, VME, VXI, GPIB, FDDI, etc. l Existing Software Environment : EPICS l Separate Computer-bus and Field-buses l “Link-person” System : Applications l “Out-Sourcing” : 5 People from Companies

13. Standard Model Architecture l Presentation Layer (Server) –Man-Machine Interface(OPI) –Logging, Analyses, Alarm Displays, Database, Global Feedbacks l Equipment Control Layer (IOCs) –Data Acquisition, Sequence Control, Local Feedbacks l Device Interface Layer (Field-buses) –Hardware Interfaces

14. The most Important Points l Presentation Layer –Software Development Tools User/Programmer Friendly Tools –Ergonomic Consoles LCDs, Macintosh’s, PCs, PDPs, Flat Desks l Equipment Control Layer –VMEbus : High Reliability, Flexibility l Device Interface Layer –CAMAC : High Reliability, Well-known

15. Presentation Layer l Operator’s Consoles : X-Terminals l Database Management : ORACLE 7 l Alarm Generation/Recording l Data Logging l Data Display : CATV Network l Simulation : SAD Program l High-Speed Network –FDDI, Distributed Shared-Memory Network, –100 Base/TX and 10 Base/T l Gateway to KEK Laboratory Network : acsad

16. Equipment Control Layer l Provides Standard Interfaces –CAMAC Serial Highway Drivers(Hytec) –MXI-bus Drivers for VXI Main Frames(HP) –ARCNET Drivers, GPIB Drivers(NI), etc. l Computers : VMEbus based IOCs –FORCE CPU-40(MC68040) –FORCE CPU-60(MC68060) –FORCE PowerCore6603(PPC 603e) –FORCE PowerCore6750(PPC 750) l Operating System : VxWorks

17. Device Interface Layer l Standard Interfaces –CAMAC : RF and Vacuum –VXI : Beam Position Monitors –GPIB : RF, Vacuum, Magnet Readouts, etc. –ARCNET : Magnet Power Supplies TV Signal Switches –MODBUS+ : PLCs –RS232: Vacuum Measuring Instruments

18. System Architecture(1) l Functional Configuration Database Console Simulation Logging Alarm Gateway Display Laboratory Network Console CAMAC VME CAMAC VME CAMAC Presentatikon Layer Equipment Control Layer Device Interface Layer High-Speed Network CATV CAMAC

19. System Architecture(2) l Actual Configuration Gateway Display Laboratory Network CAMAC VME CAMAC VME CAMAC Presentation Layer Equipment Control Layer Device Interface Layer CATV CAMAC FDDI Network Switch X-Term Alarm, Logging, Database, Console, Simulation Consoles F/E

20. System Configuration Power Supply Unit ARCNET Driver Module GPIB Controller Module CAMAC Serial Highway Driver Module CPU Module Various Equipment Magnet Power Supplies FDDI Switch Server Workstation Consoles & Peripherals Measuring Instruments VXI Main Frame RS232 Module Central Control Building Local Control Building VME-MXIbus Driver Module CAMAC Crates

21. Central Control Room Cisco Catalyst 1200 Bridge Phaser 550JX2 QMS2425 Server abco1 ME/RK 460 Cisco C1400 Concentrator FDDI GIGA Switch 26 Local Control Rooms Lattis System 810M X Terminals for Consoles VME IOC Timing, etc.

22. abco1 Server Workstation l PA-RISC 7200 Architecture –120MHz CPU Clock –4 CPUs –2GB of Main Memory –4GB Hard Disk Drives –FDDI Interface –20GB RAID Disk

23. acsad Server Workstations l Compaq Alpha Server –4 Alpha CPUs of 440MHz –6 Alpha CPUs of 330MHz –1 GB Memory –50 GB RAID –FDDI Network Interface –True 64 Unix Operating System

24. Former KEKB Control System abco1 acsad Laboratory Network KEKB Control Network 94 IOCs PCs KEKB Accelerator Operation EPICS Software Development Relational Database KEKB Accelerator Operation Beam Optics Simulation Operators’ Consoles Real-time Control

25. KEKB/PF-AR Control System abco2 acsad Laboratory Network KEKB Control Network 104 IOCs PCs KEKB & PF-AR Accelerator Operation EPICS Software Development Relational Database KEKB Accelerator Operation Beam Optics Simulation Operators’ Consoles Real-time Control abco1

26. abco2 Server Workstation l PA-RISC 8500 Architecture –440MHz CPU Clock –2 CPUs –1GB of Main Memory –36GB Hard Disk Drives –FDDI Interface –140GB RAID Disk

27. IOC Configuration Power Supply Module ARCNET Driver GPIB Controller （ NI GPIB 1014 ） CAMAC SD （ HYTEC VSD 2992 ） CPU Module （ PowerCore 6750 ） CAMAC Crates Measuring Instruments Magnet Power Supplies FDDI ｰ Ethernet Bridge Cisco Catalyst 1200 Ethernet HUB Lattis 810M Measuring Instruments VXI Main Frame RS-232C Module MXIbus Interface Module Terminal Server Cisco 2509/2511 System Monitor Module X Terminals RS-232C Ethernet From CCR

28. VME IOC System l Subrack –Schroff 16 Slots Subrack l Power Supply Module –259 Watts, 5V 35A, 12V 5A, -12V 2A –14 HP Wide, Schroff MPS8-7746 l System Monitor Module –Mitsubishi DRSJ-01 Remote System Reset DC Power Lines Monitor

29. VME Modules Used l CPUs –FORCE CPU-40MC68040 33MHz –FORCE CPU-60MC68060 66MHz –FORCE PowerCore 6603e PPC 603e –FORCE PowerCore 6750 PPC-750 266MHz –FORCE PowerCore 6750 PPC-750 400MHz l CAMAC Serial Highway Driver –HYTEC VSD 2992

30. VME Modules Used (contd.) l GPIB Controller –National Instruments GPIB 1014 l ARCNET Driver –Advanet ARCNET-4 l VME-MXI Driver –Hewlett Packard VME-MXI –National Instruments VME-MXI II l MODBUS+ Interface Controller

31. Number of IOCs and VME Modules MGRFMOVABTMVENTIPHLITotal IOC 8 8 20 12 4 20 2 1 1 96 CAMAC - 8 - 12 2 - - 2 - 1 25 GPIB 8 - - 12 6 20 - - - 66 RS-232C 1 - - 12 - - - - - - 13 ARCNET 96 - - - 4 - - - - - 100 MXI - - 20 - 1 - - - - - 21 Total 105 8 20 36 1320 2 0 1 225

32. Equipment Connected l 2,517 Magnet Power Supplies –176 ARCNET Segments l 800 Beam Position Monitors

33. EPICS Records on IOCs l 242,597 EPICS Records on 94 IOCs l 25,147 EPICS Records on IOCMGD06 l 2,788.5 EPICS Records in Average l 48,149 kB Max. Allocated Memory l 1,324 kB Min. Allocated Memory l 12,142 kB Ave. Allocated Memory

34. Hardware Summary l Use Standard Buses –Reliability: VME, Compact PCI, CAMAC Not ISA, EISA, PCI, Proprietary Buses –Separate CPU Bus from Field Buses Isolation, Analog Signal Handling l Choose Components Carefully –Power Supplies –Capacitors, Fans, Connectors l Use Ergonomic Equipment

35. Control Consoles

36. TRISTAN Consoles

37. KEKB and PF-AR Consoles

38. KEKB Consoles l Voice Generator using Power Macintosh l Multiple Screen Display for Consoles –DELL Optiplex NX1 –Intel Pentium II 400MHz, 1GHz –Colorgraphic Evolution 4 –Macintosh with Multiple Video Cards l X-terminal using IBM Network Station –Power PC 406(133MHz) –48MB Memory –Disk-less / Fan-less System

39. Control Consoles l 18” TFT LCDs(1280x1024 pixels) l 16” TFT LCDs(1280x1024 pixels) l 14” TFT LCDs(1024x800 pixels) l Book-type Personal Computers –DELL Optiplex NX1 and IBM Network Station –Multi-Screen Display Controllers –Wireless Keyboard/Mouse l Power Macintosh with 2/4 Video Controllers l Six 40” Plasma Displays as TV Monitors l Easy to change Configuration

40. Software

41. l Two Language Architecture –TRISTAN Experiences : NODAL and PCL –Interpretive Language for Applications SAD and python Short Turn-around Time Safe Debuging Environment Not Dedicated Application Software Programmer –Compiler Language for Low-Level Software C or C++ for Low-Level Software Quick Response Time Runs Fast Realizes Everything You Want

42. Software (contd.) l Relational Database Software –From the Designing Stage –Put All the Accelerator Information Cable Connection Lists Equipment Parameters –Physical Parameters : Sizes, Weight, … –Calibration Constants, Fitting Curve Parameters, … Histories –Installation Dates, Repair Histories, Costs, …. Property Numbers and more … –Generates EPICS Database Automatically

43. Software (contd.) l EPICS Software Toolkit –medm Generated from Relational Database –Ah : Alarm –Ar : Archiving –SNL : State Notation Language

44. EPICS Software Used l medm l SAD l python

45. medm Applications

46. python Applications

47. SAD Applications

48. EPICS Databse Files

49. Linac Portable CA Server Linac Console Linac Main Controls Linac Sub-Controls Linac Equipment KEKB Equipment KEKB IOCs KEKB CA Clients Linac CA Server

50. Linac Portable Channel Access

51. Application Software in Use l SAD141 l medm 74 l python 42 l Miscellaneous 6 l Total263

52. EPICS Records on IOCs l 242,597 EPICS Records on 94 IOCs l 25,147 EPICS Records on IOCMGD06 l 2,788.5 EPICS Records in Average l 48,149 kB Max. Allocated Memory l 1,324 kB Min. Allocated Memory l 12,142 kB Ave. Allocated Memory

53. Software Summary l EPICS R3.13 Toolkit on HP-UX 10.2 l medm for Man-Machine Interface l X-Window Server Software on the Terminals l python for Application Programming l SAD for Accelerator Simulation / Operation l VxWorks on IOCs l Tornado Cross Software Development Sys. l CAPFAST for EPICS Runtime Database Dev. l ORACLE 7 for Relational Database l Portable Channel Access Server for Linac Controls

54. VME CPU Board Benchmarks

55. CPU Board Benchmark Test FORCE Pcore 6604 FORCE CPU-40 CPU Power PC 604e MC 68040 Clock 200 MHz 25 MHz Memory 16 MB16 MB L2 Cache 512 kB --

56. CPU Load Arising from Scanning Database PCore 6604CPU-40 1.0 sec0.60 % 5.8 % 0.5 sec1.2 % 12 % 0.2 sec3.0 % 27 % 0.1 sec6.1 % 56 %

57. Transactions Time Required for Channel Access PCore 6604CPU-40 ca_search 953 us 1,189 us ca_put 23 us 109 us ca_get 58 us 118 us ca_put&ca_get 76 us 246 us

58. VME-MXI Problems

59. VME-MXI Problem l HP VME-MXI Driver Module : National Instruments Products l FORCE PowerCore 6603 or 6750 CPU Modules : Power PC l Tandra Universe or Universe II Chips : PCI-VME Bus Bridge Chips l CPU Module Halts and Never Restarts Responding to the Reset Signal without Power is switched off

60. Improvements in ARCNET l A HUB Box contains 3 ARCNET HUBs l 33 HUB Boxes were Installed for 97 ARCNET Lines l Decreases 10 2 /Day/Line ARCNET “Reconfiguration” Errors down to Zero

61. Configuration of the Test Bench CPU module VME-MXI module System Monitor module VXI-MXI Controller MPX module RF Voltmeter CPU Modules tested FORCE CPU40: MC68040, 33 MHz FORCE CPU64: MC68060, 66 MHz FORCE PowerCore 6603: PPC 603, Universe Chip FORCE PowerCore 6750: PPC 750, 266 MHz, Univ. II FORCE PowerCore 6750: PPC 750, 400 MHz, Univ. IIB

62. Normal and Abnormal Bus Cycles Normal Bus Cycle Abnormal Bus Cycle

63. Abnormal Bus Cycle

64. Normal and Abnormal DTACK* Signals PowerCore 6603, with Universe Chip Normal Abnormal

65. Abnormal DTACK* Signal PowerCore 6750, 266 MHz, with Universe II Normal Abnormal

66. Realtime Linux

67. Linux l Disk I/O causes 20 - 30 ms of Latency, possibly Latency can be around 100 ms or more. l Cause: –Non-preemptive Kernel –Scheduling Algorithm –Interrupt Disabling

68. Non-Preemptive Kernel Interrupt Low Priority High Priority Kernel Latency

69. Preemptive Kernel Low Priority High Priority Interrupt Kernel

70. EPICS iocCore Network Channel Access Run-time Database Device Access Layer I/O Buses (VME/PCI/ISA)

71. EPICS under L4-Linux L4 real-time micro-kernel X Linux Server MEDMiocCore

72. Linux System Calls Linux Server Process Interrupt L4 Kernel ( IPC ) Linux Kernel Process Interrupt Standard LinuxL4-Linux

73. RT-thread Preempts Linux Linux Server RT- thread Interrupt L4 Kernel Process

74. RT-thread Calls Linux Linux Server RT- thread Interrupt L4 Kernel Process

75. What Happens if RT-thread Calls Preempted Linux? Linux Server RT- thread Interrupt L4 Kernel Process Latency