1. Sundance Multiprocessor Technology SMT702 + SMT712

2. Sundance Multiprocessor Technology Organization:  Introduction to SMT702 High-speed ADC  Introduction to SMT712 High-speed DAC  FPGA Configuration (SMT702/SMT712)  Sundance Firmware Model  SMT702 Firmware Overview  SMT712 Firmware Overview  SMT702 Clocks and Resets  SMT712 Clocks and Resets  The DDR2 ‘FIFO’ Controller  Host PC Software: SMT7026  Practical Time: Questions, Help, Suggestions...

3. Sundance Multiprocessor Technology SMT702 High-speed ADC  PXIe, 3U format (32bit PXI an option) 4- lane, up to 8-lane connected Most controllers do not support 8...  Dual 8bit ADC’s, up to 3Gsps (DDR mode) AC-coupled Serial configuration Calibration feature Typical BER = x10 to the -18 CLKIN range = 500MHz – 1500MHz  Therefore (DDR) Fs = 1GHz – 3GHz Can be synchronized to same sampling time  PLL +VCO with Ref. lock Clock circuit External clock/Internal clock  3GHz(1.5GHz) or 1.5GHz(750MHz) when internal (x1 or x1/2) Reference clock in (software select)  External  10MHz backplane  100MHz backplane Reference clock out

4. Sundance Multiprocessor Technology SMT702 High-speed ADC  Trigger Internal (PXIe backplane) External (front panel)  Flash + CPLD configuration  1GB DDR2 memory x2 banks 64bit wide, 128M deep Max, 333MHz (250MHz default)  SHB x2 LVTTL 32bit + 3control +1Clock  SATA I x2  RSL x1 4 data channels @ 250MB/s  Xilinx parallel JTAG  Xilinx FPGA FFG1136-3 (fastest) FX70T, LX110T (possibility for SX50T, SX95T)  All SMA connectors on front-panel

5. Sundance Multiprocessor Technology System Architecture

6. Sundance Multiprocessor Technology SMT712 High-speed DAC  PXIe, 3U format (32bit PXI an option) 4- lane, up to 8-lane connected Most controllers do not support 8...  Dual 12bit DAC’s, up to 2.3Gsps AC-coupled Serial configuration Typical SFDR figures are close to 70dBs Stable frequency synthesis without the need for an external clock input DCM limitation of 120MHz, so minimum sampling frequency is 960Msps Can be synchronized to same sampling time  PLL +VCO with Ref. lock Clock circuit External clock/Internal clock  Default sampling frequency = 2.3GHz Reference clock in (software select)  External  10MHz backplane  100MHz backplane Reference clock out

7. Sundance Multiprocessor Technology SMT712 High-speed DAC  Trigger Internal (PXIe backplane) External (front panel)  Flash + CPLD configuration  1GB DDR2 memory x2 banks 64bit wide, 128M deep Max, 333MHz (250MHz default)  SHB x2 LVTTL 32bit + 3control +1Clock  SATA I x2  RSL x1 4 data channels @ 250MB/s  Xilinx parallel JTAG  Xilinx FPGA FFG1136-3 (fastest) FX70T, LX110T (possibility for SX50T, SX95T)  All SMA connectors on front-panel

8. Sundance Multiprocessor Technology System Architecture

9. Sundance Multiprocessor Technology Questions?

10. Sundance Multiprocessor Technology SMT702 + SMT712 FPGA Configuration

11. Sundance Multiprocessor Technology CPLD + Flash Configuration  The FPGA in both the SMT702 and SMT712 are configured at power up from a selectable address in on-board flash  A CPLD manages accessing the flash chip for configuration, and reading/writing from the Host – transparent from user  Flash size for both systems = 256Mbit

12. Sundance Multiprocessor Technology SMT6002  The SMT6002 is the provided Host application used to access the flash chip for loading and deleting bitstreams via a connection block in the FPGA to the CPLD.  The tool will automatically detect the type of board and read the contents of the flash when launched.  The default firmware (at address 0x0) is read-only and so protected from deletion. This is so that even after custom bitstream booting, there is always a recoverable link to the flash for correction.

13. Sundance Multiprocessor Technology Changing the Flash Contents  Deleting Select the bitstream to remove and press ‘Delete’. There is a ‘Safe-erase’ option which will remove every bitstream in flash except for the default bitstream at address 0x0. There is an option to remove the entire flash, but requires a modification to the Windows Registry.  Adding After a bitstream path has been chosen, select a position in flash to load the bitstream. With the ‘Basic’ addressing mode chosen, the offset address will be adjusted automatically for the board accessed. Comments and Version information may be optionally loaded as well if desired.

14. Sundance Multiprocessor Technology Bitstream Configuration  The position offset address in flash can be chosen by switch selection (SW1). SW1 has four, on/off switches integrated, but only toggle 1 and 2 are used.  The address to be booted can be counted in binary.  Position 0, (offset 0x0 default) is SW1(1 = off, 2 = off).  Position 1, (offset 0x800000) is SW1(1 = off, 2 = on).  Etc.  The FPGA can also be reconfigured after power up by selecting the bitstream, then ticking ‘Reconfigure FPGA from Flash’. Press ‘Commit’.  The approximate time to configure the FPGA from flash and be ‘alive’ to the Host is approximately 140ms.

15. Sundance Multiprocessor Technology Questions? The SMT6002 Help file can be found at: C:\Program Files\Sundance\SMT6002\SMT6002.chm

16. Sundance Multiprocessor Technology Sundance Firmware Sundance uses a FPGA to implement the communication resources. Sundance has developed various communication resources such as PCI/PCIe, 32-bit SHB, RSL…

17. Sundance Multiprocessor Technology Model Overview  Sundance uses modularity in its designs to improve reusability and maintainability. This approach creates a number of elementary blocks which are used to create the firmware.

18. Sundance Multiprocessor Technology Standard case of figure  The firmware is designed according to this model.  The FPGA is: used to implement the communication interfaces, connected to several peripherals. The role of the firmware is to allow these peripherals to exchange data, connected to the peripherals via physical wires. The peripherals may be zero or more processors, connectors or other components on the board. The following diagram shows the FPGA connected to one processor, and two connectors. The firmware comprises processor blocks, interface blocks, and connector blocks. All these blocks are interconnected with pre-defined signals. The connector and processor blocks are connected to the physical pins of the FPGA. This is a standard case of figure.

19. Sundance Multiprocessor Technology Processor block  A processor block connects a processor to one or more interface blocks, allowing a processor to exchange data with interface blocks by accessing the resources inside the interface block.  The processor block translates the access made by the processor into a format that is understood by interface blocks.  The second main purpose of a processor block is to drive the event lines of the processor connected to it. Processor block uses triggers generated by interface blocks connected to it to generate events.

20. Sundance Multiprocessor Technology Interface block  The interface block implements a collection of addressable resources (registers, and memory space mainly) that can be read or written by a processor block. Interface blocks usually drive connector blocks.  By accessing these resources, a processor block can:  obtain status information about the interface block and the connector block attached to it,  control the state of the interface block and the connector block,  transfer data to and from the connector block.  Each interface block can be connected to one processor block, one connector block, and zero or more interface blocks.

21. Sundance Multiprocessor Technology Interface block  Processors might want to know when some condition is met inside the interface block. For example when the interface block has collected a certain number of words ready to be read. When such an event occurs, the interface block can generate a trigger. These triggers may be passed to other interface blocks and to processor blocks.  Some interface blocks, called event blocks, are dedicated to gather triggers and status information from other interface blocks, and to present them to a processor block. For example gathering the status and trigger information for all the XLINK interfaces in the system. Typically, these interface blocks allow the user to select which trigger conditions should be passed to the processor block.  To indicate to a processor that a trigger condition has been met, the processor block uses event lines to the processor. The processor block may drive these event lines when the interface blocks connected to it present a trigger condition.

22. Sundance Multiprocessor Technology Connector block  The connector block forms the interface to the pins of the FPGA that are going to the connectors via physical wires.  It is usually responsible for transmitting and receiving data to and from physical wires.  The connector block has an interface that allows you to:  Send data  Receive data  Control the connector block  Obtain status All connector blocks work in a similar way but may drive the physical wires differently. They work at different speeds, and have different signals and characteristics. Sundance provides a number of standard connector blocks, such as SHB and RSL. Connector blocks are connected to interface blocks that drive them.

23. Sundance Multiprocessor Technology Implementation  This section describes the implementation of the model.  Each type of block (PB, IB, CB) has a standard interface. This interface is described in the files:  Smt_pkg.vhd. (PB  IB, PB  IB)  Chan_pkg.vhd.(IB  CB, IB  CB)

24. Sundance Multiprocessor Technology Processor block  Typically a processor connects to a peripheral using an address bus, a data bus, some control signals such as read, write and a few event lines.  In the case of Host processor via PCI/PCIe, the processor block decodes the address and the type of access made by the processor. It also generates address and control signals (read, write) used to drive the interface block. In return the interface block will provide data and triggers to the processor block. Triggers will be used by the processor block to generate events on the event lines of the processor.  The processor block forms the gateway between the processor and the firmware. The connection between a processor and a processor block is not defined. It depends on the details of the processor. The connection between a processor block and an interface block are defined by two specific types described below. The signals provided by processor block to interface block are grouped in the type t_PB_IB_X. The type t_PB_IB_X gathers: a data bus an address bus some control signals such as read and write a clock signal a reset signal

25. Sundance Multiprocessor Technology Interface block  The interface block is connected to a processor block. Each interface block has a region of the processor block’s memory space associated with it. When an access is made in this memory range, the interface block is responsible for dealing with the access. The allocation of the range of memory of an interface block can be done by specifying the properties of this interface block.  The properties allow to specify the start and the end address of the interface block. This is defined in the type t_IBProperties. This is the way the memory mapping of interface blocks is done.  Interface blocks can implement functionality that is accessed by means of resources inside the memory space associated with it. These resources can be used to control the interface block, to obtain status and to communicate data with other blocks further down in the design.  The pre-defined collection of addressable resources implemented by an interface block is called an “interface” from a software point of view. The software interfaces are described in the SMT6400 package, and in the SMT6025 for the XLINK interface block and the IB_reg.  Interface blocks can be used to drive a connector block.  Interface block and connector block are connected via a “data type” connection.

26. Sundance Multiprocessor Technology Interface block  An interface block can also connect to another interface block to exchange trigger information, gathered under the type t_Trigger. This type gathers the following information:  Trigger: when the condition of the trigger is met a pulse is generated on this line.  Trigger_latch: this line becomes high when the condition is met. It stays high until it has been reset  Flag: the flag is usually the condition that generated the trigger.  Some interface blocks (called event blocks) implement functionality dedicated to the processing of these triggers. They gather status and triggers of other blocks in the design.

27. Sundance Multiprocessor Technology

28. Another implementation of an Interface block could be an XLINK Event Block.

29. Sundance Multiprocessor Technology Memory mapping  The Processor block provides an address bus to interface blocks. The size of the address bus defines a memory space inside the FPGA addressable by the processor.  32-bit addressing supported only.  The Interface blocks decode the address bus to know if they are being accessed by the processor block. Because this address bus can be “big”, it may be expensive in terms of logic to decode all of the bits that constitute it. To simplify the address decoding, the total memory space defined by the address bus is divided into compartments of a certain size.  Interface block will react only when an access is made within the range of compartment assigned to it. The size of the compartments is fixed to 1K addresses (2^10) in the Sundance’s firmware.  The start and the end compartment of an interface block are specified by configuring its properties.  The type t_IBProperties allows to specify the following parameters: SHIFT_RIGHT: The bottom bits of the address bus provided by the processor block to be ignored when decoding the address. SHIFT_LEFT : The top bits of the address bus provided by the processor block to be ignored when decoding the address. STARTC: The number of the first compartment. ENDC: The number of the last compartment.

30. Sundance Multiprocessor Technology Memory mapping  For example IB_0 requests two compartments, and the XLINK interface requests one compartment. The following diagram shows a possible memory arrangement for two imaginary interfaces (IB_0 and IB_3) and one XLINK interface. (31-SHIFT_LEFT) - SHIFT_RIGHT= bits decoded 2^ SHIFT_RIGHT = size of a compartment. 2^(number of BITS DECODED) = number of compartments. (END – START) + 1 = the number of compartments assigned to the Interface block. For example the XLINK interface requests one compartment.

31. Sundance Multiprocessor Technology Connector block Connection between connector block and interface block is of types X_chan_t and Y_chan_t. NOTE: Sundance adopted the Diamond FPGA channel format to be able to port applications developed in Diamond FPGA into ISE projects and vice versa. Connector blocks may also exchange information with the top level of the FPGA, such as clock or reset. These scalar connections are not defined as they are specific to each connector block. For the connector blocks that have them, they should all be following the same naming convention to facilitate reusability and block interchangeability. A connector block forms the link to the physical pins of the FPGA. A connector block is connected to an interface block. Connection between connector block and the physical pins of the FPGA are not defined as they are specific to each connector block.

32. Sundance Multiprocessor Technology Summary of FPGA Channels defined in the 3L Diamond Help file  Channels are used to connect blocks to other blocks: Interface blocks Connector Blocks Custom Blocks.  A channel's unidirectional data transfer is achieved using two buses, X and Y, which carry signals in opposite directions between the communicating blocks. Each block must make sure to see the signals of the channels it can access as being synchronous to its own clock.

33. Sundance Multiprocessor Technology The Types  The following table describes the signals constituting each of the types used to implement the firmware. Signal Length (bits) Description t_PB_IB_X signals going from the Processor Block to the Interface Block CLK1 Clock signal. The clock provided by the processor block to the interface blocks. The signals of the type t_PB_IB_X are synchronous to this clock. RST1Reset signal used to reset the interface blocks. ADDR32Address bus. DATA32Data bus. READ1Read command signal. WRITE1Write command signal.

34. Sundance Multiprocessor Technology Signal Length (bits) Description t_PB_IB_Y signals going from the interface block to the processor block CLK1 Clock signal. The clock provided by the interface blocks to the processor block. The signals of the type t_PB_IB_Y are synchronous to this clock. Usually this is the same clock than the clock provided by the processor block to the interface blocks. RST1Reset signal used by the interface block to reset the processor block. ADDR32Address bus. DATA32Data bus. READ1Read enable signal. The interface block drives this signal to read data from the processor block. WRITE1Write enable signal. The interface block drives this signal to write data to the processor block. TRIGGERS32 Triggers comings from the interface blocks. The interface blocks can output triggers directly to the processor via this bus. There are 32 triggers at the most. Usually only the event blocks drive this bus. The other interface blocks tri-state it. t_IBProperties properties common to all interface blocks Interface blocks may also have custom properties SHIFTRIGHT Integer The bottom bits of the address bus provided by the processor block to be ignored when decoding the address. SHIFTLEFT Integer The top bits of the address bus provided by the processor block to be ignored when decoding the address. STARTC IntegerThe number of the first compartment ENDC IntegerThe number of the last compartment

35. Sundance Multiprocessor Technology Signal Length (bits) Description X_chan_t signals flowing between the interface block and the connector block This type is used for signals going in both directions WRITE1Write enable signal. DATA 32/64 Data bus. 64-bit data. This is considered to be two 32-bit words; one or both may be valid depending on the state of ValidWords (see below). VALIDWORDS128 describes which of the 32-bit data words are valid. ValidWords=01 2 means that only the low-order 32-bits of 'Data' contain meaningful information. ValidWords=11 2 means that both 32-bit words of 'Data' contain meaningful information. Other values of ValidWords are undefined and must not be used. Y_chan_t signals flowing between the interface block and the connector block This type is used for signals going in both directions READY 1asserted high to indicate that the transmitter may send data. t_Trigger the trigger signals going from the interface blocks to the event blocks TRIGGER1The event indicating when the condition is met (pulse) TRIGGER_LATCH1Keeps memory of the event until it has been reset FLAG1Usually the condition that generated the trigger

36. Sundance Multiprocessor Technology SMT702 FIRMWARE  We assume you read and understood the documents given previously.  Processor block: Interface to the Pcie bus. XILINX_PCI_EXP_EP  Interface blocks :XLINKs, IB_Reg, IB_TOC  Properties: IB_XLINK PROPERTIES in Smt7xx_XLink_Init_pkg. IBToc PROPERTIES in IB_pkg. IB_Reg PROPERTIES in IBReg_SMT7xx.  Connector blocks: Register blocks: CB90_SMT7xx.vhd, CB_SMT702.vhd, CB_TOC_SMT7xx Memory interface: DataPathDDR2FIFO.vhd. serial interfaces.

37. Sundance Multiprocessor Technology

38. SMT702 Firmware Memory mapping  The SMT702 memory mapping follows the table below.

39. Sundance Multiprocessor Technology SMT702 Firmware block: TOC  Number of XLINKs supported:4 max.  Address of Table of content (TOC):offset 1000 in hex from BAR1.  In the TOC, All values are in quad words (64 bits)  All offsets relative to the start of the FPGA are in bytes.  Addresses of TOC, TOD and devices are in no relationship with each other.

40. Sundance Multiprocessor Technology  The register access is specified by a BAR address and offset values. software functions: Write32(uint nBAR, uint nOffset, DWORD dword). Read32((uint nBAR, uint nOffset).  Number of register Blocks: 2 sets.  1. Register block SMT7xx: Address at offset 0 from BAR1. Implements status and control register for the board. Firmware Reset control via software:  Reset function: ResetTIMs. (from SMT6026)  Reset Pulse used as the FPGA reconfiguration enable pulse.  The board is reconfigured with a bitstream from Flash hence, the board is removed from the PCIe devices list in the PC. Interrupt status register, transparent to the user. SMT702 Firmware block: Register Blocks

41. Sundance Multiprocessor Technology  2. Register Block SMT702: Address at offset 400 in hex from BAR1. Implements specific functionalities proper to the SMT702, see a description in the SMT702 User Guide. Reset:  Reset Pulse with a delay to give time to the Hardware to propagate the reset and clear all the IB FIFOs.  Does not reset the PCIe core to prevent loosing the board on the bus.

42. Sundance Multiprocessor Technology  The X-Link is a generic 32-bit interface that can be used with any communication resource to send data to and receive data from external devices.  See XLINK specifications document:  D000051S-spec.pdf  Supports DMA transfers to from the Host. SMT702 Firmware block: XLINKs

43. Sundance Multiprocessor Technology  XLINK0: reserved to access the flash. Used bi- directionally.  XLINK1: Used bi-directionally. ADCA data path. Phase Shift Configuration for ADCA and ADCB  XLINK2: Used uni-directionally. ADCB data path.

44. Sundance Multiprocessor Technology SMT702 Firmware Blocks: Connector Blocks  Serrxtx : Provides Read/Write accesses to the flash.  DataPathDDR2FIFO: Provides access to: the FIFO that collects both ADCA and ADCB data. The SHB connector It recaptures the data and places it in the FPGA clock domain via a FIFO.  CB90_SMT7xx: Provides the FPGA reconfiguration bitstream number and command.  CB90_SMT702: Provides the path to set up the ADCs and PLL. An interface to the system monitor feature of the Xilinx Virtex-5 FPGAs. Also used to reset the XLINKs.

45. Sundance Multiprocessor Technology SMT712 FIRMWARE  We assume you read and understood the documents given previously.  Processor block: Interface to the Pcie bus. XILINX_PCI_EXP_EP  Interface blocks :XLINKs, IB_Reg, IB_TOC  Properties: IB_XLINK PROPERTIES in Smt7xx_XLink_Init_pkg. IBToc PROPERTIES in IB_pkg. IB_Reg PROPERTIES in IBReg_SMT7xx.  Connector blocks: Register blocks: CB90_SMT7xx.vhd, CB_SMT712.vhd, CB_TOC_SMT7xx Memory interface: DACDDR2DataPath.vhd. serial interfaces.

46. Sundance Multiprocessor Technology

47. SMT712 Firmware Memory mapping  The SMT712 memory mapping follows the table below.

48. Sundance Multiprocessor Technology SMT712 Firmware block: TOC  Number of XLINKs supported:4 max.  Address of Table of content (TOC):offset 1000 in hex from BAR1.  In the TOC, All values are in quad words (64 bits)  All offsets relative to the start of the FPGA are in bytes.  Addresses of TOC, TOD and devices are in no relationship with each other.

49. Sundance Multiprocessor Technology  The register access is specified by a BAR address and offset values. software functions: Write32(uint nBAR, uint nOffset, DWORD dword). Read32((uint nBAR, uint nOffset).  Number of register Blocks: 2 sets.  1. Register block SMT7xx: Address at offset 0 from BAR1. Implements status and control register for the board. Firmware Reset control via software:  Reset function: ResetTIMs. (from SMT6026)  Reset Pulse used as the FPGA reconfiguration enable pulse.  The board is reconfigured with a bitstream from Flash hence, the board is removed from the PCIe devices list in the PC. Interrupt status register, transparent to the user. SMT712 Firmware block: Register Blocks

50. Sundance Multiprocessor Technology  2. Register Block SMT712: Address at offset 400 in hex from BAR1. Implements specific functionalities proper to the SMT712, see a description in the SMT712 User Guide. Reset:  Reset Pulse with a delay to give time to the Hardware to propagate the reset and clear all the IB FIFOs.  Does not reset the PCIe core to prevent loosing the board on the bus.

51. Sundance Multiprocessor Technology  The X-Link is a generic 32-bit interface that can be used with any communication resource to send data to and receive data from external devices.  See XLINK specifications document:  D000051S-spec.pdf  Supports DMA transfers to from the Host. SMT712 Firmware block: XLINKs

52. Sundance Multiprocessor Technology  XLINK0: reserved to access the flash. Used bi- directionally.  XLINK1: Used to send the data pattern to DAC A.  XLINK2: Used to send the data pattern to DAC B.

53. Sundance Multiprocessor Technology SMT712 Firmware Blocks: Connector Blocks  Serrxtx : Provides Read/Write accesses to the flash.  DACDDR2DataPath : Provides access to: the FIFO that contains both DACA and DACB data pattern. The SHB connector. The DDS.  CB90_SMT7xx: Provides the FPGA reconfiguration bitstream number and command.  CB90_SMT712: Provides the path to set up the DACs. An interface to the system monitor feature of the Xilinx Virtex-5 FPGAs. Also used to reset the XLINKs.

54. Sundance Multiprocessor Technology Questions?

55. Sundance Multiprocessor Technology SMT702 and SMT712 Clocks and Reset Synchronous Data Acquisition and Output

56. Sundance Multiprocessor Technology SMT702 Operation  Clock Circuitry PLL+VCO ensures stable fixed sampling frequency, maxed to 1.5GHz (Fs=3GHz)  Possible to output Clock/2 = 750MHz (Fs=1.5GHz) No External clock needed this way, but can be chosen through software Reference clock can be external or 10MHz/100MHz from backplane (software selected) Clock period from ADC is Fs/4 (1.5GHz/4 = 375MHz)

57. Sundance Multiprocessor Technology ADC Sampling Phase Relationship  Counters begin in an unknown state at reset, so may not be synchronous  Resets are applied at same time, but relative phase must be verified  ADC sampling clock is split into FPGA with one input being phase shifted  Phase shift occurs in Fs T/255 increments, providing a good image of clock edge  Latched clock shape is stored in the SMT702/SMT712 configuration and status registers to be read by Host  Host issues reset if clocks are out of tolerance in phase to each other It is sometimes important to sample the two channels in phase -

58. Sundance Multiprocessor Technology

59. Host Controlled Steps to Synchronous Reset  ADC reset is issued from Host FSM issues a reset to ADC’s, DCM’s, and all logic depending on these clocks  After reset from Host to ADC is released, DCM’s and logic are held in reset until clock is available from ADC’s  When DCM’s lock onto ADC clock, all logic resets are released

60. Sundance Multiprocessor Technology

61.  DCM A is phase shifted 255 times from Host The clock provided by each ADC through a BUFG is latched with this clock, providing a good image of the clock edge of one full period of each clock  The Host reads these 255 values (‘1’ or ‘0’) and counts (XOR’s) the values between each sampled clock to find out how far off the phase relationship is between each Fs  If it is within tolerance the loop quits, otherwise another reset is issued and the clock is scanned again (phase shifted)

62. Sundance Multiprocessor Technology Final Phase-shift is for Data Capture  Once the ADC’s are aligned to each other, the clock needs to be phase shifted again The clock needs to be aligned correctly to the capture window of the data Because they are both in sync, the same clock can be used for both ADC’s  External clock requirements The default value in the software works with the onboard VCO+PLL circuitry A different frequency needs a different phase shift Trial and error recapture

63. Sundance Multiprocessor Technology SMT712 Operation  Clock Circuitry PLL+VCO ensures a stable fixed sampling frequency, maxed to 2.3GHz An additional clock out of clock circuitry is provided for system synchronization (e.g. SMT702 synchronization) No external clock needed this way, but can be chosen through software Reference clock can be external or 10MHz/100MHz from backplane (software selected) Clock period from DAC is Fs/8 (2.3GHz/8 = 287.5MHz)

64. Sundance Multiprocessor Technology DAC Clock Phase Relationship  Special attention is required for sending data in phase  There is no reset to the DAC’s An internal clock counter/divider provides the 1/8 clock to the FPGA No reset means counters start in an unknown state  Counters are matched by removing clock cycles from the internal counter A high speed ‘AND’ gate provides this feature

65. Sundance Multiprocessor Technology

66. Synchronization Pulse  An FSM manages pulses to the DAC clock counters The FSM is clocked by a DCM clock adjusted to 8/7 of the DAC clock period (328.6MHz) When a request is sent for synchronization, a pulse equal to one clock period is sent to the ‘AND’ gate input of the DAC. This equates to 7 clock pulses to the internal counter, essentially stopping the counter for 7 increments. Clock edge is sampled again and loop quits if clock correction is complete

67. Sundance Multiprocessor Technology Clocked DAC A and B Phase  Clock edge is sampled the same way as the SMT702 Sampling clock for each DAC is split into a BUFG and a DCM The DCM is phase shifted over a whole period 255 times Latched edge is stored to SMT702/SMT712 configuration and staus registers to be read by Host A software ‘XOR’ of these samples determines relative phase Because there is no reset, the Host instead issues a force synchronization pulse

68. Sundance Multiprocessor Technology

70. Final Phase-shift for Data Alignment  Once the DAC’s clocks are aligned to each other, the clocks are then phase shifted again The clocks need to be corrected to properly latch the data out of to the DAC Because they are both in sync the same clock can be used for both DAC’s  External Clock Requirements The default value for data alignment in the software works with the onboard VCO+PLL circuitry A different sampling frequency would need a different phase shift

71. Sundance Multiprocessor Technology Questions?

72. Sundance Multiprocessor Technology SMT702 and SMT712 DDR2 Memory Interface smtxxx_ddr2_interface.ngc FIFO Wrapper

73. Sundance Multiprocessor Technology smtXXX_ddr2_interface.ngc  The provided netlist is a wrapper modelling a FIFO, making memory transactions transparent It provides an interface to the Xilinx CoreGenerator MIG 2.0  The best place for information on the MIG is the Xilinx provided datasheets The provided mechanism for writing and reading from memory behaves as a FIFO would, with DDR data The usual expected FIFO flags and lines are available to the top of the component  Write clock, Read clock  Reset  Data in, Data out  Empty, Full  Write enable, Read enable  Etc.  The difference in behaviour from a FIFO is reading and writing at the same time  The wrapper has small FIFO’s inside the netlist that allow for some leeway in this respect, but to avoid complications, the interface should be viewed as a ‘duplex’ system. (i.e. read or write at one time, but not both)

74. Sundance Multiprocessor Technology

75. Questions?

76. Sundance Multiprocessor Technology SMT702 and SMT712 Host Application Environment SMT7026

77. Sundance Multiprocessor Technology Board Discovery  SmtGetBoardInfo(UINT nIndex, SMTBI& info);  SmtGetBoardIndex(UINT nBaseAddress); struct SMTBI { SMTBoardTypeType;// Define the board type char cszType[32];// String type of board UINTnBase;// Base address of the board UINT nRange;// IO range used by the board SMTHWStatusHwStatus; SMTLockLockStatus; SMTOpen OpenRes; };

78. Sundance Multiprocessor Technology Board Discovery These functions and structures recover the base address (BAR 1) of the first SMT board discovered on the PXIe bus. The application scans the PXIe bus and steps the pointer until the SMT702 or SMT712 is discovered, then the device is opened. For the SMT702, If there are multiple boards of the same type discovered on the PXIe bus, the pointer will store each of the recovered base addresses and open the subsequent boards each in turn. The BAR1 address of the PXIe device is the base address of user configurable space on the board, where our design lies. The TOC for the XLink interface blocks lie at offset 0x1000 from BAR1. Most of the addresses/functions used in the systems are read or written to the lower addresses of the BAR1 base address, starting at offset 0x400. (e.g.cb_smt702_reg.vhd or cb_smt712_reg.vhd) The offset requirements are handled by the specific SMT7x2 libraries, but if it is desired to add additional functionality it may be important to know these address offsets.

79. Sundance Multiprocessor Technology The definitions for these registers (cb_smt702_reg.vhd and cb_smt712_reg.vhd) can be found in the User Guide for each system and contain status and configuration settings for the various components and clock settings. Smt7xx.lib Functions The configuration and operation of both the SMT702 and SMT712 depends on the library Smt7xx.lib for all its basic functions. (If the graph is desired, the additional SmtGraph.lib is needed) On top of these functions are wrappers defined in two more libraries (sources are built in the provided workspace), one for the SMT702, and one for the SMT712 The Smt7xx.lib library is actually a wrapper itself for our more general SMT functions defined in the default SMT library.

80. Sundance Multiprocessor Technology XLink Functions Available in Library HXDEV XOpen (DEV_CLASS DevClass, UINT DevNo, void *buf, UINT bytes)= 0; Void XRead (HXDEV h, void *buf, UINT nBytes) = 0; Void XWrite (HXDEV h, void *buf, UINT nBytes) = 0; Void XRead (HXDEV h, void *buf, UINT bytes, double &MBps) = 0; Void XWrite (HXDEV h, void *buf, UINT bytes, double &MBps) = 0; Void XClose (HXDEV h) = 0; Void XCancel (DEV_CLASS DevClass, UINT DevNo) = 0; Memory Functions void * MemLock (UINT nBytes, void *pBuf, MDLEntry * pEntries, UINT &nEntCnt) = 0; void MemUnlock (void *pMem) = 0;

81. Sundance Multiprocessor Technology Read and Write Functions to Control and Status Registers void Read (unsigned int nReg, unsigned int &nRd) = 0; void ReadBuf (unsigned int nReg, DWORD *pBuf, unsigned int nSize) = 0; void Write (unsigned int nReg, unsigned int nCmd) = 0;

82. Sundance Multiprocessor Technology Smt702Config The board specific register information and settings for the SMT702 can be found in the Smt702_reg.h and their descriptions are defined in the User Guide. Nearly all of these software defined registers relate directly to the cb_smt702_reg.vhd registers in the firmware. Steps to Data Acquisition with GUI The steps to making a valid data capture can be summed as: Set Power Supplies Apply Resets Serial Interface Programming DCM Adjustments DDR2 Memory Storage Playback

83. Sundance Multiprocessor Technology Setting the Power Supplies ADC A ADC B Clock Circuitry OnButtonPwrApply() On pressing ‘Apply’, the GUI first issues a reset to all the components using the ‘Reset()’ function The system monitor which gathers information on the die temperature, voltage, etc. is then initialize using ‘SetSysMonitor()’ ‘SetPower()’ then sets the enable bit in the command register to turn on the power supply for each of the components (clock circuitry and DAC’s are in power off state at start up)

84. Sundance Multiprocessor Technology Issuing Resets ADCs DCM’s DDR2 Banks A and B SHB 1 SHB 2 OnButtonRstApply() On pressing ‘Apply’, the GUI issues a reset to all the components above which are checked using the ‘Reset()’ function. These are not cleared automatically.

85. Sundance Multiprocessor Technology Programming the Serial Interfaces Reference Source Clock Source ADC A Configuration Register Programming ADC B Configuration Register Programming ADC’s Calibration OnButtonIntApply() First the reference source bit is set using SetRefClk() The default clock settings are then sent to the clock registers in the FPGA. Once these values have been verified in a read back, the values are then ‘updated’, meaning they are sent to the clock chip over a serial link. This is all accomplished with SetClkSrc() Poll() is used to monitor the completion bit of this serial programming The default ADC A settings are then sent to the ADC A registers in the FPGA. Once these values have been verified in a read back, the values are then updated and serial programmed also. This is all accomplished with SetADC() Poll() is used to monitor the completion bit of this serial programming

86. Sundance Multiprocessor Technology Programming the Serial Interfaces The default ADC B settings are then sent to the ADC B registers in the FPGA. Once these values have been verified in a read back, the values are then updated and serial programmed also. This is all accomplished with SetADC() Poll() is used to monitor the completion bit of this serial programming The next step is a calibration setting for the both of the ADC’s issued with SetCalibration() Poll() is used to monitor the completion bit of this calibration The final step is a reset issued to ADC A and B with Reset()

87. Sundance Multiprocessor Technology DCM Shift Adjustments ADC A Shift Adjustment(value) ADC B shift Adjustment(value) OnButtonShiftedadjApply() The first SetShiftAdj() is issued to initialize the DCM’s to a phase offset of 0. The completion of this is monitored using Poll() A loop is then begun until the phase relationship between ADC A sampling clock and ADC B sampling clock are close enough in phase to be considered synchronized First a reset is sent to the ADC’s using Reset() SetShiftAdj() is applied 128 times to cover a half period of the clock, ensuring the clock edge recovery. Read() recovers the clock edge of each sampled phaseshift and XOR’s the result If the result is high, a counter is incremented. If the counter is too high (phase is too misaligned) the loop repeats Once the clock phases are close enough, a final SetShiftAdj() is issued to align the clock edge to data window correctly for data capture. The default value for a 3Gsps sampling clock is set already.

88. Sundance Multiprocessor Technology Filling DDR2 Memory Fill up DDR banks OnButtonDDR2MemApply() First the DDR2 is issued a reset using Write(), then cleared using Write() again. Poll() monitors all the initialization steps in the DDR of both banks to monitor when the memory is available for use The DDR2 capture enable bit is set using Write() A loop until is begun The FIFO A and FIFO B full flags are Read() until the memory is full, then the loop exits The capture enable bit for the DDR2 is then cleared with Write()

89. Sundance Multiprocessor Technology Acquisitions Save Capture Data to XX files per channel OnButtonAcqStart() A thread is begun to read the memory contents (m_hStart) A 2K buffer of memory is locked on the host using MemLock() The number of bits stored in FIFO A is Read() The number of bits stored in FIFO B is Read() The buffer is then filled with 2K samples from FIFO A using CpRead(). These samples are masked and bit shifted to represent the separate samples (they arrive concatenated). The temporary buffer is then stored to file (if selected) by using GetDatatoFile() first to store ADC A’s samples, then again to append the 2K samples to the last read 2K samples. Samples from ADC A are then sent to the graphing functions

90. Sundance Multiprocessor Technology Acquisitions The buffer is then filled with 2K samples from FIFO B using CpRead(). These samples are masked and bit shifted to represent the separate samples (they arrive concatenated). The temporary buffer is then stored to file (if selected) by using GetDatatoFile() first to store ADC B’s samples, then again to append the 2K samples to the last read 2K samples. Samples from ADC B are then sent to the graphing functions The thread continues until ‘Stop’ is pressed (m_hStop) or the memory is emptied

91. Sundance Multiprocessor Technology Smt712Config The board specific register information and settings for the SMT712 can be found in the Smt712_reg.h and their descriptions are defined in the User Guide. Nearly all of these software defined registers relate directly to the cb_smt712_reg.vhd registers in the firmware. Steps to Data Transmission with GUI The steps to making a valid data transmission can be summed as: Set Power Supplies Apply Appropriate Resets Serial Interfaces DCM Adjustments Data Source Selection

92. Sundance Multiprocessor Technology Setting the Power Supplies ADC A ADC B Clock Circuitry OnButtonPwrApply() On pressing ‘Apply’, the GUI first issues a reset to all the components using the ‘Reset()’ function The synchronization enable, reset, and synch start bit are then cleared with Write() Power to the clock circuitry and DAC’s are then cycled off then on using PwrOff() and PwrOn(), if the options are checked.

93. Sundance Multiprocessor Technology Issuing Resets DACs Clock Circuitry DDR2 Banks A and B System Monitor SHB 1 SHB 2 OnButtonRstApply() On pressing ‘Apply’, the GUI enables the synchronization (does not send a reset) with SetDacSync(). A reset is then applied to all the check options with Reset() The data source is selected for transmitting using the SetDataSrc() function

94. Sundance Multiprocessor Technology Programming the Serial Interfaces Reference Source Clock Source DAC A Configuration Register Programming DAC B Configuration Register Programming Synchronize DAC’s OnButtonIntApply() First the reference source bit is set using SetRefClk() The reset bit for the DAC’s is then cleared using Write() The default DAC A settings are then sent to the DAC A registers in the FPGA. Once these values have been verified in a read back, the values are then updated and serial programmed. This is all accomplished with SetDAC() The default DAC B settings are then sent to the DAC B registers in the FPGA. Once these values have been verified in a read back, the values are then updated and serial programmed also. This is all accomplished with SetDAC() The reset for the clock circuitry is cleared using Write()

95. Sundance Multiprocessor Technology Programming the Serial Interfaces The default clock settings are then sent to the clock registers in the FPGA and verified in a read back. This is all accomplished with SetClkSrc() The clock register settings are then serial programmed to the clock chip with an update to the FPGA registers using UpdateClkReg() After the clock chip is initialized, a calibration sequence setting for the VCO is sent to the FPGA with CalibrateVCO(). UpdateClkReg() is again used to update and send these serial configuration words to the clock chip. Read() is used to determine when this calibration cycle is complete. Synch() sends a low, then a high to the synchronization bit. A reset is then sent to DCM A with Rst_DCMDAC(), then to DCM B

96. Sundance Multiprocessor Technology Programming the Serial Interfaces If a synchronization of the DAC’s is checked – The synch reset bit will be cleared with Write() The first SetShiftAdj() is issued to initialize the DCM’s to a phase offset of 0. A loop then begins – Write() sets the synchronization enable bit, then clears it SetShiftAdj() is applied 255 times to cover a half period of the clock, ensuring the clock edge recovery. The function returns the XOR’ed result of the clock edge of each sampled phaseshift If the result is high, a counter is incremented. If the counter is too high (phase is too misaligned) the loop repeats

97. Sundance Multiprocessor Technology DCM Shift Adjustments DAC A Shift Adjustment(value) DAC B shift Adjustment(value) OnButtonShiftedadjApply() SetShiftAdj() is issued to align the clock to the correct window time for accurate latching of the data to the DAC. The default value is set to correctly work with the onboard clock circuitry

98. Sundance Multiprocessor Technology Data Source Selection File (DDR2 Pattern Generator, both DAC A and B DDR2 Pattern Generator (specify size of pattern, number of periods) FPGA embedded DDS (specify phase increment) SHB If User File to DDR2 is Source~ The specified file for A is scanned for the number of samples it holds with GetFileSize(), then the same is performed for channel B. Write() sets the DDR2 reset, then clears the reset The DDR2 is chosen as the source for DAC output with SetDataSrc() The size of the pattern for DAC A is written with Write(), then the size of the pattern for B is written with Write() The actual pattern in the file for channel A is scanned and written to the DDR2 A offset with SendPattern(). The same is again done for channel B. First – Write() clears the start/stop bit for DDR2 enable

99. Sundance Multiprocessor Technology Data Source Selection If DDR2 Pattern Generator is Source~ First Write() sets the DDR2 reset, then clears the reset The DDR2 is chosen as the source for DAC output with SetDataSrc() The size of the pattern for DAC A is written with Write(), then the size of the pattern for B is written with Write() A sine wave is then generated in the Host with the size/number of periods specified. The pattern is sent to channel A DDR2 over XLink using CpWrite(), then channel B DDR2.

100. Sundance Multiprocessor Technology Data Source Selection If embedded DDS is Source~ The DDS is chosen as the source for DAC output with SetDataSrc() The phase increment variable is sent to the DDS for DAC A with Write(), then the phase increment is sent to the DDS for DAC B If SHB is Source~ No special configuration required - SHB input is simply chosen as the source for DAC output with SetDataSrc()

101. Sundance Multiprocessor Technology Questions?