DCM Embedded Software Infrastructure, Build Environment and Kernel Modules A.Norman (U.Virginia) 1 July '09 NOvA Collaboration Mtg.

What is theDCM Embedded System? The OS & specialized drivers/software required to talk to and extract information from the hardware. Specifically: – Configuration/Boot Method (flash memory + bootloader) – Limited Operating System Includes Kernel & specialized device drivers A memory based filesystem Configuration parameters Limited set of command utilities Small enough to fit entirely in memory – Application code Designed (and optimized) specifically for the hardware Compiled and run strictly out of device memory (small footprint) Designed for “diskless” I/O A.Norman (U.Virginia)2July '09 NOvA Collaboration Mtg.

DCM Embedded System 3 DAQ Server ConfigurationSandbox DCM Can now run application code and access hardware buffers from user space

DCM Boot Example: Behold a working DCM: (switch to example) Boot/Configure time: – Average 38sec – Includes ssh initialization for diagnostics – Does NOT include startup of user applications – No calls to external DBs etc… This is on NOvA test stand setup in Feynman – Quiet network with dedicated server A.Norman (U.Virginia)July '09 NOvA Collaboration Mtg.4

DCM Kernel Driver Currently 1 module (dcm.ko) Contains correct microcode and register map for current DCM prototypes Provides access to the control registers Provides data readout in single word reads Single Word Transfer A.Norman (U.Virginia)July '09 NOvA Collaboration Mtg.5

Kernel: Not Implemented No DMA transfer – Partially implimented – Does provide block transfers – BUT….results in data corruption and repeated data blocks Status block is defined but not implemented NOvA Event View in the works for Diagnostics DMA Block Transfer A.Norman (U.Virginia)July '09 NOvA Collaboration Mtg.6 Data Corruption & Repeated Events Empty Buffer

FEB Integration With the current kernel module: – Can load the control registers – Can read the control registers This means we can technically push data to a DCM which is tagged to be transmitted to the FEB. – This interface is going to change slightly, but the format is just a memory mapped I/O 7 DCM CommandFEB AddressData Word for FEB Command processed by DCM (i.e. WRITE_TO_FEB or READ_FROM_FEB) Address in FEB of FEB command register Data required for given FEB command FEB/DCM Integration Test FEB/DCM Integration Test Scheduled for Next Week We’ll know at this point how this works in practice FEB/DCM Integration Test FEB/DCM Integration Test Scheduled for Next Week We’ll know at this point how this works in practice

Designing for 180+ Systems Design: – Want lightweight identical embedded systems – Need 180+ simultaneous copies running Problem: “You are a unique DCM, just like everybody else” – Each DCM system needs to be identical, but each needs to have a unique identity “You are a unique DCM, just like everybody else” – Means having a method of passing identity information at startup and embedding it in the system A.Norman (U.Virginia)July '09 NOvA Collaboration Mtg.8 Having a DCM “Know who it is” simplifies Run Control and Resource Manager Operations

DCM Idenity 9 DCHP Table  Keys off device MAC  Static Assignment of IP address at boot  Assigns config images Configuration Sandbox DCM MAC: 00-FF-FF….FF DHCP Request DHCP Response IP: x.x TFTP Config Assignment TFTP Config Request Config Load (Kernel/RAMDISK) Database o Universal DCM ID o DCM MAC o Install Position This info comes from installer Encode a 16bit word with the DCM position 0x0Bxx0Ayy Kernels RamDisk Images Kernels RamDisk Images Kernels RamDisk Images Kernels RamDisk Images Kernels RamDisk Images Kernels RamDisk Images 180+ configurations Every DCM is forced to “Know who it is” order to take data. This is independent from Run Control or a Resource Manager

Dev. & Build Environment DCM is a PowerPC based system – Requires cross-compilation of all applications – Cross-compilers are in place and working Both a kernel compiler and app compiler are working Simple build system works for custom applications – Need to bootstrap newer compilers to accommodate external software packages chosen by the DAQ system (example: Ganglia package for DAQMon) – Build examples exist for correct cross compiles of most applications But this needs to be formalized to match the rest of the DAQ build model 10 Build Machine (Intel i686 Arch) code Compiler (cross Arch) Obj Code (PPC form) Linker (cross Arch) Foreign Libraries Foreign Libraries Executable (Foreign) Target Machine (Motorola PPC 603 Arch) Copy across network Executable (PPC) PPC Libraries (Local Copy) PPC Libraries (Local Copy) Simple Cross Compilation Flow Example

In the works Working Bootloader (via debug trig) DCM base OS and config. DCM Kernel Module – Control Reg – Data Buffer (single word mode) – User space hooks One catch all User app. – dcm_control Read/write FPGA loading Etc… Still Required Boot on power up OS Optimizations DCM Kernel Module – Data Buffer (DMA block mode) – Status Buffer – FEB programming FEB Kernel Module – Simplify FEB configurations User apps – Need diagnostics Memory views, register views, etc… Expert applications for debugging A.Norman (U.Virginia)July '09 NOvA Collaboration Mtg.11

End A.Norman (U.Virginia)July '09 NOvA Collaboration Mtg.12

Single Word Transfer DMA Block Transfer Data Corruption & Repeated Events A.Norman (U.Virginia)July '09 NOvA Collaboration Mtg.13

Build Machine (Intel i686 Arch) code Compiler (cross Arch) Obj Code (PPC form) Linker (cross Arch) Foreign Libraries Foreign Libraries Executable (Foreign) Target Machine (Motorola PPC 603 Arch) Copy across network Executable (PPC) PPC Libraries (Local Copy) PPC Libraries (Local Copy) Simple Cross Compilation Flow Example A.Norman (U.Virginia)July '09 NOvA Collaboration Mtg.14

Outline of what to cover Overview Hardware/Software diagram Hardware configuration and loading Initial boots RAM disks A.Norman (U.Virginia)July '09 NOvA Collaboration Mtg.15