1. Process and Data Flow Control in KLOE E. Pasqualucci (INFN - Roma) enrico.pasqualucci@roma1.infn.it

2. Outline System overview Process structure and local communication SNMP and remote communication Process control Data Flow Control system DFC monitor

3. FDDI Switch VICVIC AUXMAUXM A D C 16... ADC1ADC1 ROCKROCK VICVIC AUXMAUXM A D C 16... ADC1ADC1 ROCKROCK VICVIC AUXMAUXM A D C 16... ADC1ADC1 ROCKROCK VICVIC AUXMAUXM A D C 16... ADC1ADC1 ROCKROCK CPUCPU FDDIFDDI VICVIC VICVIC ROCKMROCKM CPUCPU FDDIFDDI VICVIC VICVIC ROCKMROCKM CPUCPU FDDIFDDI VICVIC CPUCPU FDDIFDDI VICVIC ROCKMROCKM CPU server Storage system Run Control Monitor System... Trigger chain DFC system VIC CBUS Level-2crates FDDI ~ 23000 FEE channels @ 2.5 kHz f + bckg (~10 kHz) Bandwidth: ~ 50 Mbytes/s (5 Kbyte/ev.) Storage: 200 Tbyte/y Tested with peak rates of 10 kHz in multibunches mode. Tested at maximum required throughput using no zero suppressed calorimeter data DAQ system architecture

4. DAQ software organization Spy dump Recorder Builder Circ Circ (Ybos) To Disk/Tape dmap Receiver Farm FDDI switch GeoVme map Collector Circ Sender Level 2 Farm status Chain tools simulation VME Level 1 chain Data Map data Messages Traps SpyD Monitor system Didone SpyBuff RSpyD SlowCtl system DFC system CmdSrv RunCtl

5. Process structure Initialization –Msg Q creation –Shmem subscription –Shmem space allocation for variables Main Loop –Process Event –Process Command –Idle time Interrupt Handler –Extract command from Msg Q. Id Contents Mapping Processes Process number Pointer to 1st process Pointer to 2nd process Process name Process id Message queue id Process status Last command Last command status Number of variables Variable 1 Variable 2 ….. Pointer to 3rd process ….. Header Proc. 1 Proc. 2 All

6. Id Contents Mapping Processes Header Proc. 1 Proc. 2 All Process number Pointer to 1st process Pointer to 2nd process My process My process id My Q id Process status Last command Last command status Number of variables Variable 1 Variable 2 ….. Pointer to 3rd process ….. Process number Pointer to 1st process Pointer to 2nd process My process My process id My Q id Process status Last command Last command status Number of variables Variable 1 Variable 2 ….. Pointer to 3rd process ….. Reads the Q Local communication Process number Pointer to 1st process Pointer to 2nd process Process name Process id Message queue id Process status Last command Last command status Number of variables Variable 1 Variable 2 ….. Pointer to 3rd process ….. Getting a variable: Process number Pointer to 1st process Pointer to 2nd process My process Process id Message queue id Process status Last command Last command status Number of variables Variable 1 Variable 2 ….. Pointer to 3rd process ….. Process number Pointer to 1st process Pointer to 2nd process My process Process id Message queue id Process status Last command Last command status Number of variables Variable 1 My variable = value ….. Pointer to 3rd process ….. Locate process Locate variable Process number Pointer to 1st process Pointer to 2nd process Process name Process id Message queue id Process status Last command Last command status Number of variables Variable 1 Variable 2 ….. Pointer to 3rd process ….. Sending a command: –The sender: Locates the process Gets its id and message Q Process number Pointer to 1st process Pointer to 2nd process My process My process id My Q id Process status Last command Last command status Number of variables Variable 1 Variable 2 ….. Pointer to 3rd process ….. My process Q Puts command to Q Sends an interrupt signal Stop ! Process number Pointer to 1st process Pointer to 2nd process My process My process id My Q id Process status Stop ! Executing Number of variables Variable 1 Variable 2 ….. Pointer to 3rd process ….. Polls on command status Process number Pointer to 1st process Pointer to 2nd process My process My process id My Q id Process status Stop ! Success Number of variables Variable 1 Variable 2 ….. Pointer to 3rd process ….. Stop ! –The receiver: Writes the command and status and executes it Writes the command status (acknowledgement)

7. Managing the DAQ network SNMP (Simple Network Management Protocol) Largely used to manage network devices Defined as a standard by the IETP (Internet Engineering Task Force) Implemented using a reliable UDP protocol Used to retrieve and/or set information about : –network configuration –traffic –faults –accounting Managed objects defined in a Manager Information Base (MIB) defined by IETP Private extensions of the standard MIB are allowed Public domain software, allows the implementation of : –dedicated agents –utilities for remote access

8. SNMP client-server policy MIB –Variables organized as a tree Primitives: –get, get-next, set Each device runs a daemon able to: –Understand MIB requests –Obtain required information –Execute required actions Trap mechanism KLOE uses SNMP to: –Control DAQ devices and network –Implement message distribution –Implement process control –Implement Data Flow Control (DFC)

9. The command server and the KLOE MIB sub-tree iso.org.dod.internet.mgmt.mib-2 system(1) KLOE(13) sysDescr(1) sysObjectID(2) sysUpTime(3) sysContact(4) sysName(5) sysLocation(6) sysServices(7) kprocesses(1) …. kprocNumber(1) kprocTable(2) kprocEntry(1) kprocVarTable(3) kprocVarEntry(1) kprocVarProcIndex(n,1) kProcVarIndex(n,2) kprocVarName(n,3) kprocVarSize(n,4) kprocVarType(n,5) kprocVarValue(n,6) kprocIndex(1) kprocName(2) kprocId(3) kprocMsgQId(4) kprocStatus(5) kprocLastCommand(6) kprocLastCommandStatus(7) kprocVarNumber(8)

10. Message system implementation Msg Q put command INT get command write last command and status executing execute command write command status (success, fault) Node A Node B Command Server Shared Memory Run Control DAQ Process locate process send command SNMP ack get process variables first ack req second ack req first ack second ack

11. Remarks and performance Command server –DAQ process receives commands and shares variables –Command distributor Run and process control tools –tcl/tk commands implemented get variable, send message –Fortran interface for old fashioned software –Portable AIX, OSF1, HP-UX, Solaris, Linux, LynxOS supported Optimized library –Parallel message distribution implemented Performance Local command ~1.2 ms Remote variable reading ~1.2 ms Remote command completion ~4 ms

12. Production process control cmdsrv OffCtl locpc Proc_1 pcd Proc_2 Shmem (variables) command command + start trap signal check Production node Controlnode

13. FDDI Switch VICVIC AUXMAUXM A D C 16... ADC1ADC1 ROCKROCK VICVIC AUXMAUXM A D C 16... ADC1ADC1 ROCKROCK VICVIC AUXMAUXM A D C 16... ADC1ADC1 ROCKROCK VICVIC AUXMAUXM A D C 16... ADC1ADC1 ROCKROCK CPUCPU FDDIFDDI VICVIC VICVIC ROCKMROCKM CPUCPU FDDIFDDI VICVIC VICVIC ROCKMROCKM CPUCPU FDDIFDDI VICVIC CPUCPU FDDIFDDI VICVIC ROCKMROCKM CPU server Storage system Run Control Monitor System... Trigger chain DFC system VIC CBUS Level-2crates FDDI DAQ system architecture

14. Network and trigger stat The DFC System Changes the packet distribution sequence –Avoids slow-down in data transmission and blocking timeouts Keeps latency under control latmon DFCd DFC Flow table data Performance stat Statistics Commands Traps Flow table VIC bus Collector shmem RunCtl Receiver DFC status TS

15. Receiver protocol Receives event sub-packets through the GigaSwitch Put packets into multiple circular buffer Implements DFC and LatMon farm interface Dynamic thresholds Select and copy sub-event packets Get max occupancy Send trap “full” Send trap “empty” To DFC system Send LatMon trap (#) If last # arrived To LatMon TCP/IP on FDDI 0.5 MB/s EVB (1) EVB (n) If “empty” after “full” If “full”...... 0.5 MB/s

16. ... Flow tables DFC Protocol Initialization: –Builds Network Map –Builds DFC map (ordered list of RECV IP addresses) –Creates the first table with Infinity Trigger number validity Main Loop: –Wait for “trap” –On trap (full/empty): Reads the last trigger number from Trigger Supervisor Creates next table Modifies the validity of the previous table –Sends auto-test traps DFC map Max number of tables N. of RECV nodes IP addresses DFC data in VME shared memory Flags 111111…1111 0 Validity trigger Flags 111101…1111 0 0

17. DFC algorithm and performance Validity: –v = t 0 + (t tr + (t dfc + k  dfc ))*(n + k  ) +  k = 5 –autotest DFCd reaction time (trap): –1.2 ms DFC reaction time: –t local ~ 1.2 ms –trigger interaction ~6-7 ms –t dfc ~ O(10 -2 ) ms –total 10 ms DFC-L2 interaction rate: –~ 1 table / 50 ms (sustained) DFC “dead time” implemented

18. The DFC status monitor

19. Packet latency Latency measurements: –SNMP traps sent to LatMon: Collector trap when the packet # is released for sender Receiver trap when all the sub-packets # arrived Test for receiver’s buffers

20. Summary A fast and reliable message system has been implemented using standard UNIX mechanisms and the SNMP protocol Very simple to use –process template + command definition –fortran and tcl/tk interface Allows full process control A Data Flow Control system has been developed using message system and SNMP traps It allows to redirect network traffic taking into account the dynamics of the whole system Dynamic redefinition of thresholds It successfully ran during KLOE data acquisition