VIP Training NAB 2009.

VIP Training NAB 2009

Building a Broadcast Level Reference Design using the VIP Framework
Training Agenda 11:00 AM 11:30 AM VIP Framework History and Value Proposition 12:15 PM VIP Framework NAB Reference Designs : V2, SDI to HDMI Building a Broadcast Level Reference Design using the VIP Framework 12.15 PM 12.30 PM Training Setup – Design Tools, Equipment 12:30 PM 1:00 PM VIP Framework Development Flow 1:30 PM Quartus II Top Level HDL Template Explained 2:00 PM Lab 1: SOPC Builder, Video Output 2.15 PM Break 2:15 PM 3.30 PM Lab 2: Video Input, Nios instantiation and SW APIs 3:30 PM 4:30 PM Lab 3: Frame Sync 4.30 PM 5.30 PM Debug 5.45 PM 5:45 PM 6:30 PM Lab 4: Video Processing 7:15 PM Lab 5: Custom IP Simulation 8:00 PM Q&A, Common Issues and FAQ

VIP Framework NAB Reference Designs
Two VIP Framework reference designs will be demonstrated at NAB 2009 V2 Reference Design (V-Series) Stratix II GX A/V Development Kit Excellent demonstrator of Video framework methodology Run-time configurability Out of the Box ASSP-like video processing functionality Efficient external memory access Simple Format Conversion (Triple Rate SDI to HDMI) Stratix IV GX PCIe Development Kit Demonstrates 400 MHz 64 Bit Half Rate DDR3 Memory

VIP Framework NAB Reference Designs
Before we look at V2, a quick refresh on the V-Series V-Series Objectives Target typical broadcast applications Switcher, multi-viewer, converter, server Showcase VIP Framework and Altera silicon capabilities Highly software and hardware configurable solution, enabling rapid system configuration and design Provide great starting point for further broadcast systems Provide “Out of the Box” functionality to replace existing ASSP solutions Deliver high-quality up, down, and cross conversion of standard definition (SD), high definition (HD) and 3G video streams in interlaced or progressive format

V-Series Reference Designs
V-Series (V1-V3) of reference designs IBC NAB IBC 2008 2009 Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June July Aug Sept V1 V2 V3 2 Triple Rate SDI inputs, 1 DVI output Support for SD/HD/3G interlaced/progressive Input SII GX A/V Kit 2 Triple Rate SDI inputs - Interlaced/progressive 2 Triple Rate SDI outputs -Interlaced/progressive Motion Adaptive 4:2:2 Deinterlacer on 2-channels Genlock SIIGX A/V Kit 2 x SDI/DVI I/O - Software configurable Support for SD/HD/3G interlaced/progressive I/O Dithering, Gamma Correction OSD Multi-Port Front End Stratix IV GX / Arria II GX A/V Kit

V-Series Reference Designs – V1

Stratix II GX 90 A/V Board 2 x SD/HD/3G SDI inputs, 1 x DVI Output Video processing 2 video processing channels that support progressive/interlaced, SD/HD formats up to 1080p/1080i 1 high quality channel (MA Deinterlacer, 12x12 tap Scaler) 1 lower quality channel (Weave Deinterlacer, nearest neighbour Scaler) Highly software configurable Scaling ratio, scaler coefficient generation/loading, location offset Mode switching Thumbnail or full screen Output resolution switching 480p60, 720p60 or 1080p60

V-Series Reference Designs - V2
2 x SD/HD/3G SDI inputs and outputs Output 720p, 1080i Video processing 2 fully independent high quality SDI channels supporting any resolution, interlaced or progressive, any frame rate up to 1080p60 2 high quality channels (Motion Adaptive Deinterlacer, Multi-tap Scaler) Motion adaptive deinterlacer 4:2:2 support Reduces memory bandwidth requirements New Interlacer IP to enable 1080i output Efficient external DDR2 memory access New Genlock support Highly software configurable Scaling ratio, scaler coefficient generation/loading No hardware recompile needed when changing input or output format, resolution or frame rate Cost reduction 7K total LE reduction in frame buffers/deinterlacers compared with 9.0

Motion Adaptive Deinterlacer Enhancements >25% Reduction in Memory Bandwidth Requirement Buffer data in YCbCr 4:2:2 format Buffer 20 bit data rather than 30 bit YCbCr 4:4:4 or 24 bit RGB (V1) Single motion value calculated for each pixel 2500 LE reduction per Motion Adaptive Deinterlacer 500 LE reduction in each in Avalon-MM Read/Write Master Minor algorithm improvement In 9.0, the motion value was calculated for each color plane Reduced shimmering for panning shots

Efficient External DDR2 Memory Access Micron MT9HTF6472AY-53EB3 DDR2 SDRAM provides a maximum theoretical bandwidth of: 266.7 MHz x 64 bits x 2 (both clock edges used) = Gbit/s Local interface width = 256 bits (Half rate mode) In V2 there are 14 Avalon-MM Masters accessing one DDR2 64bit memory Total bandwidth requirement = GBit/s V2 design achieves bandwidth requirement Memory access efficiency = / = 77.4% Haven’t reached “failure point”, so achievable access efficiency with V2 topology >= 77.4% V2 memory access efficiency is >= / = 77.4%

Data Packing Video Data The data width is 256 bits wide The Avalon-MM Master packs 12 x 20 bit pixel (sample) per transfer = 240 bits = 16 wasted bits per transfer 16 bits / 12 pixels = extra bits per 20 bit sample Motion Data The Avalon-MM Master packs 25 x 10 bit motion values (sample) per transfer = 250 bits = 6 wasted bits per transfer 6 bits / 12 pixels = extra bits per 10 bit sample Motion Adaptive Deinterlacer Input format: 1080i60 1920 x 1080 x bits x 60/2s = Gbit/s Output format: 1080p60 1920 x 1080 x bits x 60/s = Gbit/s Motion format: 1 value per pixel 1920 x 1080 x bits x 60/2s = Gbit/s Memory access: 1x write at input rate: Gbit/s 1x write at motion rate: Gbit/s 1x read at motion rate: Gbit/s 2x read at output rate: Gbit/s Total Bandwidth (per Deinterlacer): Gbit/s Frame Buffers Input format: 1080p60 1920 x 1080 x bits x 60/s = Gbit/s 1x write at input rate: Gbit/s 1x read at output rate: Gbit/s Total Bandwidth (per Frame Buffer): Gbit/s Total System Bandwidth MA Deinterlacer 1: Gbit/s MA Deinterlacer 2: Gbit/s Frame Buffer 1: Gbit/s Frame Buffer 2: Gbit/s Total: Gbit/s A breakdown of the external memory bandwidth calculation

New Interlacer IP Converts video in progressive format to interlaced format Outputs a video field for each progressive frame input Frame n+1 Frame n Field n+1 (even lines) Field n (odd lines) Interlacer Av-ST Video Sink Av-ST Video Source Output field rate = input frame rate No buffering Pass through when input is interlaced

System control and configuration in software (Nios) Class API for each run-time controllable VIP Suite function Initialise and start function execution Scaler co-efficient generation, re-load, output resolution Clipped region Peripheral control (eg. Buttons/LEDs) Event manager to handle interrupts Visibility of input resolution and input/output FIFO overflow and underflow Example class API for video input bridge Example initialization control code Software view of hardware system through API

Logic Utilization On-Chip Memory Multipliers (18x18 bit) V2 94% (S2GX90) 408 M4Ks + 4 M-RAMs* 88 S2GX90 90,960 408 M4Ks M M-RAMs 192 Arria II GX 93,675 612 M9Ks + 1171 KBits (MLABs) 448 * Total block memory bits 1,816,656 / 4,520,448 ( 40 % ) ; Total block memory implementation bits 4,278,528 / 4,520,448 ( 95 % )

Stream 2 (from Source 2) Stream 1 (from Source 1) SDI_IN0 SDI_OUT_P3 SDI_IN1 SDI_OUT_P2 SD/HD/3G SDI Video Source 1 SD/HD/3G SDI Video Source 2 DVI cable SDI cable (BNC) SDI

Toggle output format of Stream 1 between 1080i60 and 720p60 Toggle output format of Stream 2 between 1080i60 and 720p60

SIV GX PCIe Kit – Single channel Video
Triple rate SDI input SD, HD or 3G, interlaced or progressive, any resolution and frame rate up to 1080p60 HDMI output Motion adaptive deinterlacer Pass through progressive input No scaler 400MHz 64bit DDR3 memory

Hardware Design in SOPC Builder Modular, parameterizable, re-usable IP components

SIV GX PCIe Kit – Format Conversion
External DDR3 Memory Large external memory bandwidth available DDR3 SDRAM provides a maximum theoretical bandwidth of: On C3 speed grade, 400 MHz x 64 bits x 2 (both clock edges used) = 51.2 Gbit/s On C2 speed grade, 533 MHz x 64 bits x 2 (both clock edges used) = 68.2 Gbit/s Local interface width = 256 bits (Half rate mode) Single channel format conversion example only uses 10.45/51.2 = 20% of available bandwidth 1 X Motion Adaptive Deinterlacer (4:4:4 mode)

External DDR3 Memory A 4 video processing channel design, containing 28 masters, will require 52.9 Gbit/s 4 Motion Adaptive Deinterlacers (4 x 5 = 20 masters) 4 Frame Buffers (4 x 2 =8 masters) Memory Access Efficiency requirement = 52.9 / 68.2 = 77.6% 533 MHz DDR3 Memory Achievable today with SOPC Video Framework!!

SIV GX PCIe Kit – Video Format Conversion
Logic Utilization (LEs) On-Chip Memory Multipliers (18x18 bit) Video Format Conversion 41K 195 M9Ks 15 S4GX230 228,000 1235 M9K + 22 M144Ks Kbits (MLABs) 1288 A2GX95 93,675 612 M9Ks + 1171 KBits (MLABs) 448

In the booth we will show the V2 design …... HDMI->DVI Cable SDI_IN0 SDI_OUT_P3 SDI_IN1 SDI_OUT_P2 SDI cable (BNC) SD/HD/3G SDI Video Source 1 SD/HD/3G SDI Video Source 2 DVI cable SDI cable (BNC)

… and replace a converter box with the Stratix IV GX PCIe design HDMI->DVI Cable SDI_IN0 SDI_OUT_P3 SDI_IN1 SDI_OUT_P2 SDI cable (BNC) SD/HD/3G SDI Video Source 1 SD/HD/3G SDI Video Source 2 DVI cable SDI cable (BNC)

VIP Framework NAB RDs – Demo Steps
3 Steps to a successful demo Step 1: Position the Video Framework Step 2: Show the reference design running AND the design flow Show the design block diagram Show the design running (Video on the screen) Show SOPC Builder (Rapid hardware system capture) Show Nios II IDE (Software configurable) Step 3: Focus on the customer’s needs Either solution enabler (Video Framework) OR Solution provider (ASSP Replacement) SOPC Builder provides abstracted view and application specific switch fabric I/O, algorithmic, component, framework

Step 1 – Position Video Framework
The Video Framework .. provides the infrastructure to make it easier to build Broadcast Level video processing systems enables customers to focus on their key value add and apply video expertise Lead customers may have their own high quality algorithmic IP V-Series … provides ASSP-level capability “Out of the Box” AND a great design starting point Ability to quickly customise both hardware and software demonstrates how to solve key video system and system design challenges on Altera’s FPGAs Eg. Efficient external memory access, IP re-use

System level design tools and methodology SOPC Builder, Nios II IDE, Quartus II Open interface standard to mix-and-match custom or off-the shelf IP Avalon Memory Mapped and Avalon-Streaming Video protocol Open Source Verilog HDL Algorithm Template to facilitate integration of custom IP into framework Building block IP cores to speed up development Interfaces (Triple rate SDI, DVI, etc), Algorithmic (Scaler, Motion Adaptive Deinterlacer, etc), Framework IP (Frame Buffer, Clocked Video Input/Output), High Performance Memory Controllers (eg. DDR3) 1080p reference designs that provide ASSP-level capability and a great design starting point V-Series, M1-M5, Cyclone III based, Partner designs (Omnitek, Bitec, Microtronix) Development boards/kits for rapid design prototyping Stratix II GX, Stratix IV GX PCIe, Cyclone III Arria II GX Coming Soon

Multi-layered solution enables rapid system design at highest level of abstraction AND ability to differentiate at different layers Devices Hardware Compilation Tools Application Specific IP Application Specific SW Application Design Application Specific Protocol Target Application 1 Target Application 2 Target Application 3 Application Targeted Content System Level Tools Processor(s), OS, toolchain Framework is a Solution Enabler. Enable customer to differentiate by adding their own IP V-Series Reference Designs are also a Solution Provider. Out of the box ASSP functionality Building Blocks and Standards IP block

Get Hands On 3 Steps to a successful demo Step 1: Position the Video Framework Step 2: Show the reference design running AND the design flow Show the design block diagram Show the design running (Video on the screen) Show SOPC Builder (Rapid hardware system capture) Show Nios II IDE (Software configurable) Step 3: Focus on the customer’s needs Either solution enabler (Video Framework) OR Solution provider (ASSP Replacement) SOPC Builder provides abstracted view and application specific switch fabric I/O, algorithmic, component, framework

3 Steps to a successful demo Step 1: Position the Video Framework Step 2: Show the reference design running AND the design flow Show the design block diagram Show the design running (Video on the screen) Show SOPC Builder (Rapid hardware system capture) Show Nios II IDE (Software configurable) Step 3: Focus on the customer’s needs Solution enabler (Video Framework) OR Solution provider (Virtual ASSP) SOPC Builder provides abstracted view and application specific switch fabric I/O, algorithmic, component, framework

Training Setup – Design Tools
Install ACDS 9.0 Quartus II, SOPC Builder, IP, Nios II, Modelsim AE Launch Quartus II 9.0 to set QUARTUS_ROOTDIR Install VIP Framework Lab Training Slides and Labs Copy vip_training_90.zip from USB memory stick to your local drive and uncompress Do not choose a location which contain spaces in the path name For the training, you will use ACDS 9.0 No patches required – Hoorah!! You will not be able to build V2 in full – Booo!! V2 requires a 9.0 patch Changes to Quartus II install, new Interlacer IP, Deinterlacer 4:2:2 mode An ACS Patch with full V2 design will be distributed after NAB

Training Setup – Equipment
You need access to the following hardware Equipment 1 x Stratix II GX A/V Development Board 2 x SDI cables (BNC connectors) 1 x SDI->DVI Converter Box 1 x DVI cable 1 x 24” Monitor (capable of displaying 1080p60) 1 USB Blaster Typically you will be sharing equipment

Training Setup – Equipment
SDI_IN0 SDI_OUT_P2 SD/HD/3G SDI Video Source 1 DVI cable SDI cable (BNC)

General Video System Design Flow
We’re mainly going to focus on implementation But first the Design Specification Design Specification No OK? Yes Hardware Implementation Software Implementation for System Bring Up No OK? Yes Yes Software only change? Full Software Implementation No OK? Yes Project complete

Design Specification Input specification Output specification
All legal inputs and their functional description (e.g. SD-SDI, HD-SDI, 3G-SDI, DVI, etc.) Output specification All possible outputs and their functional description (e.g. SD-SDI, HD-SDI, 3G-SDI, DVI, etc.) External memory specification External memory bandwidth, memory parameterization, efficiency requirement Runtime specification What is configurable at runtime (eg. scaler resolution change) Board components for user interaction and status information Eg. Display overflow status with LEDs, Buttons for mode switching Device/Board System functions The building blocks Existing and new IP cores/features Block diagram The block connectivity Output of Design Specification is a Functional Description of the design

Design Specification - System Functions
Step 1: Identify the building blocks NTSC input Clocked Video Input Clocked Video Output DVI output Chroma Resampler Frame Buffer Color Plane Sequencer Scaler CSC Deinterlacer Clipper Test Pattern Generator Mixer DDR2

Step 2: Consider algorithmic constraints Requires 4:4:4 input Clocked Video Input Clocked Video Output NTSC input DVI output R G CSC B Requires progressive 4:4:4 input 8bit Parallel 4:4:4 Color Plane Sequencer Chroma Resampler Deinterlacer (MA) Scaler (Polyphase) Cb Y Cr Y Y Y Y Cb Cr 10 bit Sequence 4:2:2 Cr 10 bit Parallel 4:2:2 Cb Requires 4:4:4 input in 9.0 (4:2:2 support in 9.1) 10 bit Parallel 4:4:4 DDR2

Step 3: Define the system connectivity 27 MHz 100 MHz NTSC input Clocked Video Input Color Plane Sequencer Chroma Resampler CSC 65 MHz 60fps 59.94fps Clocked Video Output DVI output Frame Buffer Scaler (Polyphase) Deinterlacer (MA) Clipper Mixer Test Pattern Generator The Frame Buffer is just doing Double buffering (no rate conversion). This smooths out the burstiness of the system and can make it easier to meet the output video data rate (systems can be designed without it). For systems with multiple video streams it is typically placed after the Scaler and before the mixer on each streams video path. 75 MHz DDR2

Implementation Build system incrementally in 6 Steps
Full Software Implementation Design Specification No Yes Project complete Software Implementation for System Bring Up OK? Hardware Implementation only change? Software Implementation Build system incrementally in 6 Steps Always follow them! 1. Implement top level HDL (done for you!) 2. Implement video output 3. Add soft processor for control and debug Can be removed later if not required 4. Implement video input 5. Implement frame sync and memory interface 6. Integrate video processing functions Build System Incrementally

Hardware Implementation - Video Output
Use a TPG instead of video source to test video output PLLs SDI-Tx SDI Tx SDI-Rx SDI Rx Clocked Video Output Test Pattern Generator VCXO DVI Tx DVI Tx PFD DVI Rx SOPC Builder Top level (HDL)

Hardware Implementation – Nios II
Add Nios II. Write some software for control and debug Add board peripherals PLLs SDI-Tx LEDs JTAG UART Buttons Nios II SDI Tx SDI-Rx SDI Rx Clocked Video Output Test Pattern Generator VCXO DVI Tx DVI Tx PFD DVI Rx SOPC Builder Top level (HDL)

Hardware Implementation - Video Input
Add Video Input (no datapath yet) PLLs SDI-Tx LEDs JTAG UART Buttons Nios II SDI Tx SDI-Rx SDI Rx Clocked Video Output Clocked Video Input Test Pattern Generator Terminator VCXO DVI Tx DVI Tx PFD DVI Rx SOPC Builder Top level (HDL)

Hardware Implementation - Video Input
Add Frame Buffer and Memory Controller PLLs SDI-Tx LEDs JTAG UART Buttons Nios II SDI Tx SDI-Rx SDI Rx Clocked Video Output Clocked Video Input Test Pattern Generator Terminator Frame Buffer VCXO DVI Tx DVI Tx PFD DVI Rx SOPC Builder DDR2 Memory Controller Top level (HDL)

Hardware Implementation – Video Processing
Add Video Processing Functions PLLs SDI-Tx LEDs JTAG UART Buttons Nios II SDI Tx SDI-Rx SDI Rx Clocked Video Output Clocked Video Input Frame Buffer Chroma Resample CSC Deinterlace Scale VCXO DVI Tx DVI Tx PFD DVI Rx DDR2 Memory Controller Top level (HDL)

Software Implementation (Full)
Full software implementation Only software changes now expected For example, scaler coefficient generation code Full Software Implementation Design Specification No Yes Project complete Software Implementation for System Bring Up OK? Hardware Implementation Software only change?

Building a broadcast level reference design using the VIP Framework
Training Agenda 11:00 AM 11:30 AM VIP framework History and Value Proposition 12:15 PM VIP Framework NAB Reference Designs : V2, SDI to HDMI Building a broadcast level reference design using the VIP Framework 12.15 PM 12.30 PM Training Setup – Design Tools, Equipment 12:30 PM 1:00 PM VIP Development Flow 1:30 PM Quartus II Top Level HDL Template Explained 2:00 PM Lab 1: SOPC Builder, Video Output 2.15 PM Break 2:15 PM 3.30 PM Lab 2: Video Input, Nios instantiation and SW APIs 3:30 PM 4:30 PM Lab 3: Frame Sync 4.30 PM 5.30 PM Debug 5.45 PM 5:45 PM 6:30 PM Lab 4: Video Processing 7:15 PM Lab 5: Custom IP Simulation 8:00 PM Q&A, Common Issues and FAQ

Quartus II Top Level Template
All Stratix II GX A/V Kit VIP Reference Designs use the same Top Level Template

Contents The template consists of
A set of HDL modules that are used to interface with the different features on the board (MAXII, VCX0s, SDI, DVI etc...) via their pins A top level HDL file called s2gxav.v that ties all the HDL modules together A configuration file config.v that enables or disables the various interfaces in the s2gxav.v file Pin mappings for the board in the shape of a TCL file – s2gxav.tcl A BAT file to make the Quartus project – make_project_V2.bat

Block Diagram Key: Orange Circles – clock domains Red lines – clocks
Black lines – data Green lines – memory interfaces Onewire – a single wire interface to the MAXII chip that is used to control the LEDs and Muxes and receive button presses VCXO (Voltage Controlled Crystal Oscillator) – the frequency of the output clock can be controlled by varying the voltage (V in the diagram) Charge Pump – collection of capacitors and resistors that convert pulses on the up and down wires into a constant voltage PFD (Phase Frequency Detector) – detects the difference in frequency between the two hrefs and speeds up or slows down the output href to match the input Rate Detection – detects whether the input is American or European standard and sets the output clock to match

Configuration Uncomment interfaces in the config.v to make them appear in the top level (s2gxav.v) Then add the correctly named block to your SOPC Builder system It will be connected to the correct pins by the top level file Only change the config.v and SOPC Builder system – nothing else

Naming config.v s2gxav.v SOPC Builder

Building a broadcast level reference design using the VIP Framework
Training Agenda 11:00 AM 11:30 AM VIP framework History and Value Proposition 12:15 PM VIP Framework NAB Reference Designs : V2, SDI to HDMI Building a broadcast level reference design using the VIP Framework 12.15 PM 12.30 PM Training Setup – Design Tools, Equipment (Break) 12:30 PM 1:00 PM VIP Development Flow 1:30 PM Quartus II Top Level HDL Template Explained 2:00 PM Lab 1: SOPC Builder, Video Output 2.15 PM Break 2:15 PM 3.30 PM Lab 2: Video Input, Nios instantiation and SW APIs 3:30 PM 4:30 PM Lab 3: Frame Sync 4.30 PM 5.30 PM Debug 5.45 PM 5:45 PM 6:30 PM Lab 4: Video Processing 7:15 PM Lab 5: Custom IP Simulation 8:00 PM Q&A, Common Issues and FAQ

Lab 1: Video Output Open the training slides vip_training_90.ppt and go to slide 57 (this slide!)

Lab 1: Video Output In this lab you will
Use the Video framework to output a progressive test pattern (1280x720) via triple rate SDI

Lab 1: Video Output Block Diagram

Lab 1: Video Output Browse to directory <Training install>\Labs\Lab_1_Video_Output Open (with a text editor or in Quartus II) and examine configuration file .\V2_stage1\config.v Only the SDI output is enabled. ‘define sdi_out_1_fixed_HD removes the unused top level run time control signals for the clocked video output (CVO) IP, because the CVO is not run-time controllable in this design.

Lab 1: Video Output Create a Quartus II Project
Open DOS Command Window cd <Training install>\Labs\Lab_1_Video_Output Type make_project_V2_stage1.bat Create a Quartus II project file (s2gxav.qpf) using the s2gxav.tcl script Copies V2_stage1\config.v and V2_stage1\V2.sopc into the Quartus II project directory

Lab 1: Video Output Confirm a Quartus II project has been created containing global and location assignments s2gxav.qsf

External clocks have been created for you
Lab 1: Video Output External clocks have been created for you Open Quartus II Project s2gxav.qpf Launch SOPC Builder Tools->SOPC Builder ( ) Opens V2.sopc cpu_clk – Nios II Processor Clock vip_clk - Video Processing Clock mem_clk - Reference Clock for Memory Controller

Lab 1: Video Output Add the Clocked Video Output (CVO) IP core to the SOPC Builder system Select “Clocked Video Output” Click “Add” button to launch parameter GUI

Digital SDI standards (BT656, BT1120) map to “Embedded in Video”
Lab 1: Video Output Parameterise CVO for 1280x720 progressive output, 60 fps as below Do not use the Control Port (for now) Click Finish Digital SDI standards (BT656, BT1120) map to “Embedded in Video” Resolution Wait until FIFO is almost full before starting output to avoid underflow caused by burstiness

Lab 1: Video Output Rename Clocked Video Output Module name to sdi_out_1 Select <module name> -> right click -> Rename The CVO clocked video signals are exported to the top level design Renaming the module will match the name of the CVO exported signals to the top level s2gxav.v template signal names

Lab 1: Video Output Add the Test Pattern Generator (TPG) to the SOPC Builder system Select System Contents->Video and Image Processing ->Test Pattern Generator Click Add… Parameterize as shown Click Finish

Lab 1: Video Output Connect the TPG Avalon-ST source to the CVO Avalon-ST Sink in SOPC builder patch panel Change the TPG clock domain to vip_clk Change the CVO clock domain to vip_clk CVO contains a Dual Clock FIFO which bridges between the video processing clock domain and the input pixel clock clock domain vip_clk is the clock used in the video processing clock domain The pixel clock (sdi_out_1_clk) is exported to the top level Confirm there are no errors or warnings Click here

Lab 1: Video Output Generate SOPC Builder System
Save V2.sopc when prompted Confirm generation is successful Click here

Lab 1: Video Output In Quartus II, compile design
Processing->Start Compilation Confirm design compilation completes successfully 5 minute compile (hard boiled)

Lab 1: Video Output Program FPGA with s2gxav.sof Tools->Programmer
Hardware Setup …->Select USB-Blaster Add File … s2gxav.sof Click Start

Lab 1: Video Output Confirm a 1280x720 test pattern is displayed on the Monitor TIP: The test pattern has a single black line of pixels around border so you can easily detect vertical scrolling TIP: If the test pattern fills the screen change the monitor’s display setting to 1:1. Congratulations! You have completed Lab 1!

Lab 1: Video Output Optional: Test 1080i output Add 1080i CVO
Change the resolution and interlaced parameters in the TPG and CVO Re-generate SOPC Builder System and re-compile in Quartus II Add 1080i CVO

Lab 2: Nios II and Video Input
In this lab you will Add a Nios II processor to the SOPC Builder system Re-parameterize the TPG and CVO to output interlaced video and enable runtime change of output video resolution, controlled by a button press Build a Nios II software project to read/write from/to IP core registers Add Board Peripherals to accelerate system bring up Add a Clocked Video Input IP core to unit test the input video TIP: Adding a Nios II processor to the design early in the hardware implementation will accelerate development Very good for both debug and control 1080i60 PAL

Block Diagram (Part A) In this section you will Add a Nios II Processor and memory to store program code Add board peripherals to SOPC Builder system Parameterise TPG and CVO for interlaced output Enable control interface on Test Pattern Generator and Clocked Video Output Build a Nios II IDE software project

Browse to directory <Training install>\Labs\Lab_2_NiosII_and_Video_Input Open (with a text editor or in Quartus II) and examine configuration file .\V2_stage2\config.v

Create a Quartus II Project (similar to Lab 1) In a command window, cd <Training install>\Labs\Lab_2_NiosII_and_Video_Input Type make_project_V2_stage2.bat Open Quartus II Project s2gxav.qpf Launch SOPC Builder Tools->SOPC Builder Opens V2.sopc as shown below

Edit the existing Clocked Video Output module (sdi_out_1) Select module sdi_out_1 and click Edit … First, select preset conversion SDI 1080i60, and click “Load values into controls” button Confirm that the “Image Data Format” and “Syncs Configuration” Groups are configured Then, parameterise CVO to enable runtime configuration of sync data via control port as shown on the next page

Click Finish Set Fifo level at which to start output to 1919 Check “Use Control Port” An Avalon MM Slave interface will appear on the CVO IP block in SOPC Builder “Set Runtime configurable video modes” to 3

Edit the existing Test Pattern Generator module (alt_vip_tpg_0) Select module alt_vip_tpg_0 and click Edit … Re-parameterise Test Pattern Generator to output interlaced video and enable runtime configuration of the resolution Set parameterisation as shown Check “Runtime control of image size” An Avalon MM Slave interface will appear on the TPG IP block in SOPC Builder Set “Maximum image width” to 1920 Set “Maximum image height” to 1080 Set “Interlacing” to Interlaced Output F0 synchronized”

We will now add a Nios II Processor to the SOPC Builder system Initially, we will use the Nios II Processor to write control data to the Clocked Video Output Avalon-MM Slave port to load sync data; later we’ll do a lot more! In V2, the Nios II program code is stored in separate SRAM Program too large for available on-chip memory Desirable to store in DDR2 memory with video buffer, but memory access efficiency prevents us A study to maximise memory access efficiency for such system configurations is in progress. Watch this space! TIP: Add the memory to the system before adding the Nios II processor. The memory will then be selectable in the Nios II parameter GUI to assign the reset and exception vector offsets

Select Memories and Memory Controllers->SRAM->Cypress CY7C1380C SSRAM in Component Library Click Add … Use default parameters Click Finish Use default Base and End addressing Rename ssram_0 to ssram Select Module, right click, rename

Add a tri-state bridge to interface to off chip SRAM We want to share data and address signals. We can’t share the address signal until the bridge is connected to the SRAM Select Bridges and Adapters>Memory Mapped->Avalon-MM Tristate Bridge Use default parameters Click Finish Rename tri_state_bridge_0 to tristate_bridge Select Module, right click, rename Connect ssram to tristate_bridge and lock SRAM address Click here to lock SRAM address (No change of address if you apply System->Auto-Assign Base Addresses) Click here to connect

In the Tristate Bridge, share the address signal Select the module tristate_bridge and Click Edit… Select Shared Signals tab and select address Click Finish

Add Nios II/e processor for system control and debug Click Finish Nios Data Master will automatically connect to all Avalon-MM Slave interfaces Includes Clocked Video Output (sdi_out_1) Avalon MM Slave interface Rename cpu_0 to cpu Select Nios II/e core Select ssram component for reset and exception vector memory

Auto assign base address System->Auto-Assign Base Addresses Confirm there are no errors or warnings in the system The CPU, SRAM and tri-state bridge are in the cpu_clk clock domain

Add board peripherals to Nios II address map Added as Avalon-MM Slaves connected the Nios II Data Master Slight increase in hardware compile times but significant acceleration in system bring up All peripherals get automatically connected to Nios II data master We will add System ID Peripheral Verifies that the hardware SOF and the software ELF file match JTAG UART Streams serial characters between SOPC Builder system and the host Timer Not used in this design Parallel I/O interfaces for DIP Switches Push Buttons Switch between display modes LEDs Status information on video stream and display mode

Add a System ID Peripheral Nios II processor systems use the system ID core to verify that an executable program was compiled targeting the actual hardware image configured in the target FPGA Select Peripherals->Debug and Performance->System ID Peripheral, click Add…, Finish Rename the component to sysid to remove warning

Add JTAG UART Peripheral Select Interface Protocols ->Serial -> JTAG UART Click Add… Rename the component to jtag_uart The JTAG UART implements a method to communicate serial character streams between a host PC and an SOPC Builder system For the Nios II processor, device drivers are provided in the HAL system library, allowing software to access the core using the ANSI C Standard Library stdio.h routines Use default settings

Add a Timer Select Peripherals->Microcontroller Peripherals -> Interval Timer click Add…, Finish Rename the component to timer Enable Full-featured. Your processor system may require a timer with variable period that can be started and stopped under processor control

Add DIP Switch I/O Peripheral Select Peripherals->Microcontroller Peripherals -> PIO (Parallel I/O) Click Add… Rename the component to dip 8-bit Input ports only

Add LEDs Peripheral Select Peripherals->Microcontroller Peripherals -> PIO (Parallel I/O) Click Add… Rename the component to leds 8-bit Output ports only

Add Buttons Peripheral Select Peripherals->Microcontroller Peripherals -> PIO (Parallel I/O) Click Add… Rename the component to buttons 2-bit Input ports only

Confirm board peripherals are connected to the Nios II processor data master in the cpu_clk domain

In SOPC Builder, generate SOPC system Click Generate button Confirm design generates successfully In Quartus II, compile design Processing->Start Compilation 25 minute compile TIP: After the SOPC builder system has generated, you can start building your software. The software system library project uses the SOPC Builder generated V2.ptf for system component and connectivity information You don’t need to wait for the Quartus hardware compilation to complete

Full Software Implementation Design Specification No Yes Project complete Software Implementation for System Bring Up OK? Hardware Implementation Software only change? Now it’s software time! The Nios II processor implements the control layer HDL free C++ API included (abstraction layer to Avalon-MM read/write transactions) Fast debug cycle The control layer is compiled in seconds Initially, we will create and build a Nios II software project that will configure the Clocked Video Output IP Core at run-time

Launch Nios II IDE Start -> Programs -> Altera -> Nios II EDS 9.0 -> Nios II 9.0 IDE Create a new project in NIOS II IDE 9.0 File>Switch Workspace.., browse to “software” directory in Labs\Lab_2_NiosII_and_Video_Input\software

Click here Click Workbench

In Nios II IDE, create a new Nios II Application project File>New->NIOS II C/C++ Application Click Finish Change name to s2gxav_controller Select Blank Project Add onchip_mem, dvi_output_i2c_master, quad_video_i2c_master and NIOS Browse to select V2.ptf file. Ensure there are no spaces in path

Open system library page In the Nios II C/C++ Projects tab, select s2gxav_controller project, right click, select System Library Properties Add onchip_mem, dvi_output_i2c_master, quad_video_i2c_master and NIOS

Parameterise system library Select s2gxav_controller project, right click, select System Library Properties Confirm that the following devices are selected: JTAG_UART for stdout, stderr, stdin TIMER for system clock timer (not used in this example) Add onchip_mem, dvi_output_i2c_master, quad_video_i2c_master and NIOS Enable: Program never exits Lightweight device drive API to reduce code size

For help on the system library page, click the help button and select Nios II help Lab 2: Nios II and Video Input

Add C++ source files to Application project Copy files in software\source directory into the s2gxav_controller application project directory In Nios II IDE, update Nios II C/C++ Projects view Select s2gxav_controller project right click, Refresh Confirm a list of .cpp and .hpp files appear in view

The application project contains Initialization code for each run-time configurable hardware component C++ Class APIs for each Video and Image Processing IP core which has an Avalon-MM Slave control interface Manager code Event manager to handle interrupts Class APIs for development board peripherals

VIP Suite C++ Class APIs All Video and Image Processing Suite MegaCore function classes are derived from the base class Vipcore.hpp The address offset of the GO register and the STATUS register are the same (0x0 and 0x1 respectively) for all VIP Suite IP The class contains methods common to all functions such as starting and stopping the video function processing at a frame boundary CLOCKED_VIDEO_OUTPUT my_cvo(SDI_OUT_1_BASE); my_cvo.start(); my_cvo.stop(); In the application project, double click on ipcore.hpp to review the class

Clocked Video Output API allows you to easily configure the IP core at run-time for different output standards (eg. 1920x1080p60) void configure_mode(..) Also provides status and diagnostic information FIFO fill level Underflow In the application project, double click on Clocked_Video_Output.hpp to review the class API In the application project, also take time to review a sample of the other Class APIs, eg. Mixer.hpp, Scaler.hpp, Button_manager.hpp Clocked_Video_Input.hpp

Review the configuration C++ source code Double click on main.cpp main.cpp has 2 key stages: 1. Instantiate and initialize the components via Avalon-MM Slave interfaces using the Class APIs 2. Infinite loop to manage input resolution changes and button presses while (1) {…}

Sync data for different output modes 1920 x 1080 interlaced, 60 frames per second (fps), 720 x 576 interlaced, 50 frames per sec PAL Lab 2: Nios II and Video Input Write configuration data to clocked video output control interface

If the input resolution changes read CVI registers and print resolution and format to nios2-terminal (Lab 2: PART B) Lab 2: Nios II and Video Input Loop forever Flashing LED3 to show software is running LED0 lit when underflow at output Check for button press Change resolution

In the Quartus II programmer, program the FPGA with s2gxav.sof Pre-compiled s2gxav.sof provided in project directory if required Build and download the NIOS II software Right-click on s2gxav_controller and select Run As->NIOS II Hardware Add onchip_mem, dvi_output_i2c_master, quad_video_i2c_master and NIOS

Confirm a 1920 x 1080 test pattern is displayed on the Monitor The Blackmagic/AJA converter box performs the deinterlacing Weave/bob algorithm to output progressive Confirm LED 3 is flashing (software running) Press push button 0 Confirm the resolution switches to 720x576 (PAL) GOTCHA: Check that the monitor is in 1:1 pixel mode and not scaling the image to the full screen resolution Congratulations! You haven’t quite completed Lab 2!!!

It’s now time to get the Video Input working! We got the video output working with minimal dependencies. i.e. no input stream, no memory interface Ideally we can get the input working with minimal dependencies on other system components

In this section we will: Enable SDI Rx (SDI_IN_1) interface (using config.v) Connect SDI Rx-> Clocked Video Input IP->Terminator IP core The terminator is always ready to receive (and discard) data Enable the control port on the CVI IP core to allow Nios II software to read the input resolution from the register map (Avalon-MMSlave interface) Part B

In Quartus, open and edit file Lab2_NiosII_and_Video_Input\config.v To enable SDI Rx in top level template Uncomment line `define sdi_in_1 true Save file

In SOPC Builder, Add Clocked Video Input Select Video and Image Processing -> IO -> Clocked Video Input Click Add… Configure as shown and click Finish Rename module to sdi_in_1 1. Set default to SDI 1080i60 and click “Load values in controls” button 3. Enable to allow input of both SD video (10bit) and HD/3G standards (20 bit) at run-time 2. Use control port to provide access to video status/format information

Add Terminator IP Core Select Video and Image Processing -> Terminator Click Add… Use default settings as shown Click Finish The terminator is a simple IP core (not included in ACDS 9.0) that is always ready to receive (and discard) data. It is used to test the video input in isolation, by terminating the Clocked Video Input Avalon-ST Source interface.

Connect CVI Source to Terminator Sink Remove overlapping addresses Select System->Auto-Assign Base Addresses Connect CVI Source to Terminator Sink Provide unique IRQ if requested

Confirm no Errors are reported Click Generate Confirm that the SOPC Builder system generates successfully after a couple of minutes

In Quartus II, click Compile 35 min compile Program FPGA with s2gxav.sof s2gxav.sof provided in project directory

Back to Software! We have enabled the control port (Avalon-MM Slave interface) on the Clocked Video Input so we can now read the video format/resolution of the input video stream from the Nios II processor You will use the Nios II class API (Clocked_Video_Input.hpp) The software will output the format/resolution via the JTAG UART and nios2-terminal on the host (printf) if (the_cti_1.is_input_interlaced()) { printf("Interlaced, F0 %d x %d, F1 %d x %d, Stable = %d\n", the_cti_1.get_f0_sample_count(), the_cti_1.get_f0_line_count(), the_cti_1.get_f1_sample_count(), the_cti_1.get_f1_line_count(), the_cti_1.is_input_stable()); }

In Nios II IDE, review main.cpp code #define will be set in system.h file after re-building the software project with the latest SOPC Builder system, so no need to change the code (see next slide for instructions) The resolution/format of the input stream will be written to the nios2-terminal

In Nios II IDE, re-build the software program (.elf) and run the NIOS II software Right-click on s2gxav_controller and select Run As->NIOS II Hardware This will cause the system library project and application projects to be re-built before downloading the s2gxav_controller.elf file to the FPGA After rebuilding, SDI_IN_1_BASE will be #define’d in the system library (system.h) The system.h file is automatically re-generated based on the contents of the .ptf file generated earlier by SOPC Builder The system.h file is generated in the system library project at s2gxav_controller_syslib/Debug/system_description/system.h The resolution/format of the input stream will be written to the nios2-terminal Try changing the resolution of the video source and confirm that the resolution detected changes in the nios2-terminal window Congratulations! You have now actually completed Lab 2!

Lab 3: Frame Sync In this lab you will:
Add an external DDR2 Memory Controller to the SOPC Builder system Add a Frame Buffer to the SOPC Builder system Connect the Test Pattern Generator -> Frame Buffer -> Clocked Video Output Connect the Frame Buffer to the Memory Controller

Lab 3: Frame Sync You will also understand
How to maximise external memory bandwidth efficiency in SOPC Builder Suitable addressing to enable bank interleaving On-chip buffering System topology using pipeline bridges Burst sizes Synchronising video streams with Genlock

Lab 3: Frame Sync Browse to directory <Training install>\Labs\Lab_3_Frame_Sync Open (with a text editor or in Quartus II) and examine configuration file .\V2_stage3\config.v

Lab 3: Frame Sync Create a Quartus II Project (similar to Lab 1)
In a command window, cd <Training Install>\Labs\Lab_3_Frame_Sync Type make_project_V2_stage3.bat Open Quartus II Project s2gxav.qpf Launch SOPC Builder Tools->SOPC Builder (Opens V2.sopc)

NOTE: The DDR2 Memory Controller and Avalon-MM Pipeline Bridge are already added to the LAB 3 SOPC system. Parameterization instructions are included in this presentation, but can be skipped Lab 3: Frame Sync Avalon-MM Pipeline Bridge Slave interface is not connected

Lab 3: Frame Sync Add DDR2 High Performance Memory Controller
Select Memories and Memory Controllers->SDRAM->DDR2 SDRAM High Performance Controller Click Add… Skip this step

Select Memory Micron MT9HTF6472AY-53E
Lab 3: Frame Sync Speed grade: 3 PLL reference clock: 100MHz Memory clock frequency: Half rate Select Memory Micron MT9HTF6472AY-53E Skip this step

Lab 3: Frame Sync Click Finish Rename altmemddr_0 to altmemddr
50 ps for SII GX 90 A/V Board Skip this step

Lab 3: Frame Sync Add Avalon-MM Pipeline Bridge
Reduce register to register delay and provides flexibility in system topology to increase memory access efficiency Lab 3: Frame Sync Change altmemddr clock to ‘mem_clk’ Add Avalon-MM Pipeline Bridge Select Bridges and Adapters->Avalon-MM Pipeline Bridge Click Add… Skip this step

Lab 3: Frame Sync Add a Frame Buffer to test memory interface (parameterised as below) GOTCHA: In 9.0, the frame buffer does not support dropping and repeating INTERLACED fields. Individual fields are dropped/repeated producing incorrect sequences of F0-F0, or F1-F1 fields. The desired behaviour is to drop/repeat F0-F1 pairs. GOTCHA: If there is no frame/field dropping/repeating in the datapath, you will need to genlock/framelock the input and output video clocks Do NOT enable Dropping and Repeating when Video in interlaced

Lab 3: Frame Sync Connect Frame Buffer to TPG and CVO
Connect TPG dout (Avalon-ST Source) ->Frame Buffer din (Avalon-ST Sink) Connect Frame Buffer dout (Avalon-ST Source)->CVO din (Avalon-ST Sink) Connect Frame Buffer to Pipeline Bridge Connect Frame Buffer write_master (Avalon-MM Write Master) to Pipeline Bridge s1 (Avalon-MM Slave) Connect Frame Buffer read_master (Avalon-MM Read Master) to Pipeline Bridge s1 (Avalon-MM Slave)

Clock crossing handled efficiently in Frame Buffer
Lab 3: Frame Sync Clock crossing handled efficiently in Frame Buffer Change Frame Buffer din clock to vip_clk Dout clock will also change Change Frame Buffer read_master clock to altmemddr_sysclk (memory clock) Change Frame Buffer write_master clock to altmemddr_sysclk (memory clock)

Lab 3: Frame Sync Auto-Assign Base Addresses
System->Auto-Assign Base Addresses Confirm that the SOPC builder system is error free In SOPC Builder, Click Generate In Quartus II, click Compile 1 hr compile Program FPGA with s2gxav.sof S2gxav.sof provided in project directory 8% of memory usage due to Signal Tap

Lab 3: Frame Sync Software for this project is pre-compiled
Output modes supported are HD-SDI 1080i (1080i59.94/60) and PAL as in Lab 2 Launch Nios II Command Shell Start ->All Programs->Altera->Nios II EDS 9.0->Nios II Command Shell Download program cd <Training install>/labs/Lab_3_Frame_Sync nios2-download –g –r s2gxav_controller.elf ; nios2-terminal Confirm that the test pattern is displayed on the monitor Press push button 0 to switch between 1080i and PAL Congratulations, you have successfully completed Lab 3.

Efficient Use Of Memory Bandwidth
Main aim is to reduce the impact of bank management commands .i.e. non data transfer commands We do this by using: long bursts large on-chip buffers bank Interleaving

Burst Size Long bursts (32, 64, 128) within a row
minimizes the penalty for switching rows (by making your burst size a multiple of the row size you ensure that a burst doesn’t cross a row boundary) requires large on-chip buffers (inside the deinterlacer and framebuffer) to create/receive the burst

On-Chip Buffering Large buffers
by making your on-chip buffers twice the burst size you can be processing a burst whilst transferring another to/from memory allows the video processing path to cope with the longer latency caused by waiting for access to the memory controller using a separate clock for the memory, with sufficient on-chip buffering, allows the memory to be run at a higher rate without being limited by the speed of the datapath (this allows more masters to be handled efficiently)

Bank Interleaving Consecutive bursts should be to different banks
further minimizes the penalty for switching rows by overlapping the bank management commands of one bank with the data transfer to/from another bank 9.1 controller will improve this further by adding predictive bank management, this will allow the bank management commands to be issued earlier Set the base address of each framebuffer and deinterlacer to a different bank

Multi-port front end using SOPC Builder (1)

Multi-port front end using SOPC Builder (2)
Configure your masters (Framebuffer and Deinterlacer) to perform large bursts with on-chip buffers set to twice the burst size Group masters that are on the same bank together using a pipeline bridge (set each Framebuffer and Deinterlacer base address to a different bank) Use the round robin arbitration of SOPC Builder to switch between the pipeline bridges (bank interleaving)

Limitations Works best when :
bandwidth requirements for each master are equal bandwidth requirements for each group of masters are equal there are enough banks for each framebuffer and deinterlacer to have one each no random access masters, keep NIOS on a separate memory We have achieved 78% read/write efficiency with this system - this was the requirement for V2, the maximum we can achieve may be higher

Genlock Genlock – aligning the hsync and vsync of the output video to a reference source (of the same format and frame rate) e.g. 1080p60 to 1080p60 Framelock (Crosslock) – aligning the start of frame of the output video to a reference source (of a different format with a lockable frame rate) e.g. 720p60 to 1080i60, 720p30 to 1080p60 Sources that are locked to the same reference source can be switched between cleanly on a frame boundary

Block Diagram Key: Orange Circles – clock domains Red lines – clocks
Black lines – data Green lines – memory interfaces Onewire – a single wire interface to the MAXII chip that is used to control the LEDs and Muxes and receive button presses VCXO (Voltage Controlled Crystal Oscillator) – the frequency of the output clock can be controlled by varying the voltage (V in the diagram) Charge Pump – collection of capacitors and resistors that convert pulses on the up and down wires into a constant voltage PFD (Phase Frequency Detector) – detects the difference in frequency between the two hrefs and speeds up or slows down the output href to match the input Rate Detection – detects whether the input is American or European standard and sets the output clock to match

Clock locking (1) Detect input format and frame rate – read from CVI registers Configure the CVI and CVO to output a divided down version of the refclk and vcoclk respectively (for genlock these can be the hsyncs) – the pfd compares these and speeds up/slows down the vcoclk to match the refclk Lock the clocks (using pfd and VCX0) – output clock must track any changes in the input clock to avoid buffer overflow/underflow

Clock locking (2)

Frame Locking (1) Detect input format and frame rate – read from CVI registers Configure the CVI and CVO to output a start of frame signal that can be compared CVO compares the start of frame signals and counts the difference between the two CVO then removes or repeats that many lines and samples to align the two start of frame signals

Frame Locking (2)

Debug In this session you will learn which signals to capture in Signal Tap to aid video system debug SignalTap II Logic Analyzer allows designers to observe the behavior of hardware Includes streaming interfaces, peripheral registers, memory buses, and other on-chip components Requires signals of interest to be specified before hardware compilation

Debug – Quick Checklist
Reminder: Building system incrementally will reduce time spent overall debugging Isolate and bring up system components Check if the CVI FIFO has overflowed Use CVI signal overflow or status register Check if the CVO FIFO has underflowed Use CVO signal underflow or status register Check the Avalon-ST Video data flow Ready, valid, data, eop, sop signals between each VIP core

Debug – Quick Checklist
Check the Avalon-ST Video control packet contents Decode the Avalon-ST Video control packet TIP: All VIP cores use the last Avalon-ST Video control packet received before the video data. Check the VIP cores that buffer data in external memory produce Avalon-ST Video data at the required rate Check the Av-ST source and sink and the Avalon-MM Masters

Debug Top level (HDL) Signal Tap Video signals
Signal Tap Av-ST Signals Signal Tap Av-ST Signals Signal Tap Av-ST Signals Signal Tap Av-ST Signals Signal Tap Av-ST Signals PLLs SDI-Tx LEDs JTAG UART Buttons Nios II SDI Tx SDI-Rx SDI Rx Clocked Video Output Clocked Video Input Chroma resample CSC Deinterlace Scale VCXO DVI Tx DVI Tx PFD DVI Rx Signal Tap Av-MM Masters DDR2 Memory Controller Top level (HDL)

Debug In Quartus II, open SignalTap Logic Analyzer Sample depth = 4K
Tools>SignalTap II Logic Analyzer Sample depth = 4K Add vip_clk clock Add NIOS, onchip_mem, dvi_output_i2c_master, quad_video_i2c_master

Debug In Node Finder, select Filter = Signal tap II -> pre-synthesis Add Video signals, Av-ST Video signals and Av-MM Master signals Video Signals Add video input and output signals (vid_*) for Clocked Video Input Clocked Video Output

Debug Avalon-ST Video signals Add din_* signals for each VIP core
din_data, din_valid, din_ready, din_sop, din_eop The CVO calls the signals is_* Except CVI, the input to this is the video

Debug Avalon-MM Read/Write Masters *av*
write, read, readdata, readdatavalid, writedata, address, waitrequest, burstcount, readdatavalid Add NIOS, onchip_mem, dvi_output_i2c_master, quad_video_i2c_master

Debug Cheatsheet in the Future
Problem Possible Cause (s) Debug No Output video …… Output video is frozen on a frame Red and Blue colors are swapped Output video is green and purple Output video is glitches every few seconds Output video scrolls vertically Output video scrolls horizontally The output video judders …..

Lab 4: Video Processing In this lab you will:
Remove the Terminator and the Test Pattern Generator Add video processing functions to the SOPC Builder system, including a scaler and deinterlacer Build a configurable video processing datapath Display the format converted video stream Something more interesting than a test pattern!!

NTSC/PAL/720p/1080i Video Source 1
Lab 4: Video Processing Output video resolution 1280x720p60 DVI cable SDI_IN0 SDI_OUT_P2 NTSC/PAL/720p/1080i Video Source 1 SDI cable (BNC)

Lab 4: Video Processing Browse to directory <Training install>\Labs\Lab_4_Video_Processing Open (with a text editor or in Quartus II) and examine configuration file .\V2_stage4\config.v

Lab 4: Video Processing Create a Quartus II Project (similar to Lab 1)
In a command window, cd <Training Install>\Labs\Lab_4_Video_Processing Type make_project_V2_stage4.bat Open Quartus II Project s2gxav.qpf Launch SOPC Builder Tools->SOPC Builder (Opens V2.sopc)

Include Avalon MM Slave interface to allow run-time clipping
Lab 4: Video Processing Remove the Terminator and the Test Pattern Generator IP functions from the SOPC Builder system Hold down Ctrl key, select alt_vip_terminator_0 and alt_vip_tpg_0, and click Remove button Add a Clipper IP function to the SOPC Builder system For NTSC input the first 3 lines of the Active Picture region contain Closed Caption data (eg. pop-up text) NTSC has 244 lines in Field F0 and 243 lines in Field F1 Some of the VIP Suite IP cores do not support processing video with varying field sizes Select Clipper in the Component Library and click Add… Parameterize as shown and click Finish Include Avalon MM Slave interface to allow run-time clipping

Lab 4: Video Processing Connect CVI to Clipper
Connect CVI dout (Avalon-ST Source) ->Clipper din (Avalon-ST Sink) Set Clipper clock to ‘vip_clk’ Connect the Nios II CPU data_master (Avalon-MM Master) to the Clipper Control port (Avalon-MM Slave) Connect cpu data_master to alt_vip_clip_0 control Add Chroma Resampler (4:2:2 to 4:4:4) to SOPC Builder system In 9.0, the MA Deinterlacer does not support 4:2:2 input, so we need to chroma resample (4:2:2 to 4:4:4) before deinterlacing MA Deinterlacer 4:2:2 support will be added in 9.1 (and V2 Beta Patch) Select Chroma Resampler in the Component Library and click Add…

Lab 4: Video Processing Parameterise Chroma Resampler as shown, click Finish

Lab 4: Video Processing Connect Clipper to Chroma Resampler
Connect Clipper dout (Avalon-ST Source) ->Chroma Resampler din (Avalon-ST Sink) Set Chroma Resampler clock to ‘vip_clk’ Add Deinterlacer to SOPC Builder system Select Deinterlacer in the Component Library and click Add…

Lab 4: Video Processing Parameterise Deinterlacer as shown, click Finish TIP: Set ‘Frame buffer mode’ first, then ‘Deinterlacing Method’. This will activate the greyed out parameters

Lab 4: Video Processing Add Avalon-MM Pipeline Bridge
Reduce register to register delay and provides flexibility in system topology to increase memory access efficiency Lab 4: Video Processing Add Avalon-MM Pipeline Bridge Select Bridges and Adapters->Avalon-MM Pipeline Bridge Click Add…

Lab 4: Video Processing Set pipeline_bridge_1 clock to altmemddr_sysclk Set pipeline_bridge_1 Base Address to 0x0 and Lock address by clicking on the lock symbol When a new component, with a Memory Mapped Slave, is added to an SOPC Builder system, the tool may automatically connect any Nios II data_master/instruction_master(s) to the Avalon MM Slave port Ensure the Nios II CPU data_master and instruction_master (Avalon-MM Masters) are NOT connected to the pipeline_bridge_1 Slave port s1 Disconnect the cpu data master and instruction_master from the pipeline_bridge_1 s1 port to remove overlapping address errors

Lab 4: Video Processing Connect the Pipeline Bridge (pipeline_bridge_1) to the DDR2 Memory Controller (altmemddr) Connect pipeline_bridge_1 m1 (Av-MM Slave) -> altmemddr s1 (Av-MM Slave)

Lab 4: Video Processing Connect Deinterlacer to new Pipeline Bridge (pipeline_bridge_1) Connect DIL write_master -> pipeline_bridge_1 s1 (Av-MM Slave) Connect DIL read_master_0 -> pipeline_bridge_1 s1 (Av-MM Slave) Connect DIL read_master_1 -> pipeline_bridge_1 s1 (Av-MM Slave) Connect DIL motion_write_master -> pipeline_bridge_1 s1 (Av-MM Slave) Connect DIL motion_write_master -> pipeline_bridge_1 s1 (Av-MM Slave) Set Deinterlacer clocks to altmemddr_sysclk for all Av-MM Masters and vip_clk for Av-ST as show above

Lab 4: Video Processing Connect Chroma Resampler to Deinterlacer
Connect Chroma Resampler dout (Avalon-ST Source) ->Deinterlacer din (Avalon-ST Sink) Add Scaler to SOPC Builder system Select Scaler in the Component Library and click Add…

Lab 4: Video Processing Parameterize as shown, each tab
Avalon-MM Slave interface to change the output scaler resolution at run-time upto a max resolution of 1920x1080.

Use default parameters
Lab 4: Video Processing Parameterize as shown and click Finish 4 Taps horizontally and 4 taps vertically. This is the recommended number of taps when performing upscaling. We aren’t dynamically loading coefficients in this Lab, so the image quality will not be high, particularly for downscaled images. Use default parameters

Lab 4: Video Processing Connect Deinterlacer to Scaler
Connect Deinterlacer dout (Avalon-ST Source) ->Scaler din (Avalon-ST Sink) Set Scaler clock to ‘vip_clk’ Connect the Nios II CPU data_master (Avalon-MM Master) to the Scaler Control port (Avalon-MM Slave) Connect cpu data_master to alt_vip_scl_0 control Add Chroma Resampler (4:4:4 to 4:2:2) to SOPC Builder system Select Chroma Resampler in the Component Library and click Add…

Lab 4: Video Processing Parameterize as shown and click Finish

Lab 4: Video Processing Connect Scaler to Chroma Resampler (alt_vip_crs_1) Connect Scaler dout (Avalon-ST Source) -> Chroma Resampler din (Avalon-ST Sink) Set Chroma Resampler clock to ‘vip_clk’ Re-parameterise frame buffer to handle change in video rate across pipeline Select alt_vip_vfb_0 and click Edit … Button. TIP: Only drop/repeat frames in one buffering IP function in a pipeline. Otherwise you will see increased judder in output (multiple consecutive dropped/repeated frames) The deinterlacer does not drop/repeat frames in this example

Enable Frame dropping (at input) and Frame repetition (at output)
Lab 4: Video Processing Parameterize as shown and click Finish Enable Frame dropping (at input) and Frame repetition (at output)

Lab 4: Video Processing Confirm that the Nios II data masters is connected correctly to the VIP Suite Avalon-MM Slave ports TIP: To view data master connections Select View->Collapse All Expand cpu by clicking on by module cpu

Lab 4: Video Processing Ensure that the Nios II instruction master is only connected to the tristate_bridge jtag_debug_module Disconnect instruction master from other Avalon-MM Slave ports. This will remove overlapping address errors for this master Auto assign base addresses to remove overlapping address errors for the Nios II data master System->Auto-Assign Base Addresses Confirm that the SOPC builder system is error free

Lab 4: Video Processing In SOPC Builder, Click Generate
In Quartus II, click Compile > 1 hr compile, so try the sof provided Program FPGA with s2gxav.sof S2gxav.sof provided in project directory 8% of memory usage due to Signal Tap

Lab 4: Video Processing Software for this project is pre-compiled (s2gxav_controller.elf) The video pipeline deinterlaces the source and outputs progressive video Output mode configuration is now HD-SDI 720p60 (1280x720) If you’re keen, create a new Nios II project and build your own executable using the software in software/source directory Launch Nios II Command Shell Start ->All Programs->Altera->Nios II EDS 9.0->Nios II Command Shell Download program cd <Training install>/labs/Lab_4_Video_Processing/software nios2-download –g –r s2gxav_controller.elf ; nios2-terminal Confirm that the video source is displayed on the monitor in format 1280x720p60 Congratulations, you have successfully completed Lab 4.

Lab 5: Custom IP Simulation
In this Lab you will Simulate an example VIP Verilog IP function The IP function is an RGB to greyscale converter with an Avalon-ST Video source and sink The IP function uses a Avalon-ST Video Verilog Template Use a SystemVerilog testbench that Reads a binary data file and drives the data into the Design Under Test (DUT), according to the Av-ST Video Protocol Receives data according to the Avalon-ST Video Protocol and writes a binary data file Convert between the binary data files and AVI files using a utility provided

Browse to directory <Training install>\Labs\Lab_5_Custom_IP_Simulation The Avalon-ST BFM files (avalon_st_sink_bfm.sv and avalon_st_source_bfm.sv) and Avalon-ST Video File Source/Sink files (avalon_st_video_file_source.sv and avalon_st_video_file_sink.sv) are located in directory ./common: The Avalon-ST Video File Source/Sink modules instantiate lower level functional models of the general SOPC Builder Avalon-ST Source and Sink The general Avalon-ST BFMs will be available in a future version of SOPC Builder File Source and Sink call functions from these models to drive data onto and read data from the Avalon-ST interface

Avalon-ST Video File Source component Written in System Verilog Reads binary data from files and drives it into the DUT according to the Avalon-ST Video protocol Referred to as ‘File Source’ from here on Avalon-ST Video File Sink component Also written in System Verilog Receives Avalon-ST Video compliant data from the DUT and writes it to output files Referred to as ‘File Sink’ from here on TIP: Instantiate one File Source for every Avalon-ST sink on the DUT TIP: Instantiate one File Sink for every Avalon-ST source on the DUT TIP: Calls to a few simple functions allow whole frames of data to be streamed through the DUT

Browse to directory <Training install>\Labs\Lab_5_Custom_IP_Simulation\template_testbench The directory contains: The template testbench (tb.v) Features one File Source and one File Sink connected together with no DUT in between Plus commands to run some basic tests A Makefile to build and run the testbench in ModelSim Can be called from a console window or from ModelSim No waveforms displayed A ModelSim DO file (wave.do) to run the testbench and display waveforms A directory dut_files, which contains the RGB to greyscale converter source code. The IP function has One Avalon-ST Video sink (input) One Avalon-ST Video source (output) Both use RGB data, 8 bits per symbol, 3 symbols in parallel

Browse to directory <Training install>\Labs\Lab_5_Custom_IP_Simulation\avi_to_binary_utility The directory contains Utility to turn AVI files into binary data files (and vice versa) (cygwin_avi_bin_convert.exe) AVI file vip_car.avi Browse to directory <Training install>\Labs\Lab_5_Custom_IP_Simulation\lib The directory contains useful simulation tasks and functions, such as test status reporting Browse to directory <Training install>\Labs\Lab_5_Custom_IP_Simulation\Docs The directory contains functional description of the Avalon-ST Video BFM avalon_st_video_BFM_v0.doc The following slides give step by step instructions to include the DUT in the template testbench, build and run the testbench

In Quartus II, open template_testbench\tb.v Insert and connect the RGB to greyscale component The assignments in template testbench connect the outputs of the File Source to the inputs of the File Sink (so it works when there is no DUT) - Delete or comment these lines out so data goes through the RGB to greyscale converter Code for instantiation is already there, but commented out - Uncomment instantiation

The Verilog source files for the algorithm (RGB to greyscale converter) are in the “dut_files” directory Open the file template_testbench/makefile and edit it to build the RGB to greyscale converter Under the “specify the compile of files for the dut here” comment enter compile commands for all the required files Example commands are included (commented out), just change the file names on these Make sure the order they are listed matches the compile order The example uses a VIP Verilog Template for Av-ST Video interfaces. IP developer only needs to change the algorithm code (alt_vip_rgb2grey)

Update file template_testbench/wave.do (ModelSim DO file) to show the RGB to greyscale converter waveforms Command to insert the signals from the RGB to greyscale core are already included but commented out. Uncomment the command “add wave ..” Waves for other blocks can be added using the names specified in the Verilog instantiations e.g. add wave -noupdate -format Logic -radix decimal /tb/dut/encoder/* add wave -noupdate -format Logic -radix decimal /tb/dut/decoder/*

Use the utility provided (avi_to_binary_utility\cygwin_avi_bin_convert.exe) to generate an input binary file to match the testbench configuration Use “vip_car.avi” file as the input and create “vip_car_0.bin” file Name matches a string in the testbench so it must match exactly New file should be formatted according to RGB24 fourcc code 24 bits per pixel: 8 bits red, 8 bits green, 8 bits blue Open a command window or cygwin window: cd <Labs>\Lab_5_Custom_IP_Simulation Follow the commands on the following slide

Once complete, copy “vip_car_0.bin” to the “template_testbench” directory Also copy “expected_0.bin” to the “template_testbench” directory This file contains the data that the RGB to greyscale converter should produce. The testbench will compare the results with this file

Lab 5: Custom IP Simulation - Summary of the Tests
Before we run the testbench, a quick look at the tests performed Some very basic tests to check that the DUT conforms to the Avalon-ST Video protocol Check that output matches expected results for 7 frames 7 frames represent: 4 general cases for data transmission on Avalon-ST interface Source always ready to send, sink always ready to receive Source always ready to send, sink not always ready to receive Source not always ready to send, sink always ready to receive Source not always ready to send, sink not always ready to receive plus 3 error cases that most cores should be able to cope with Early end of packet (fewer pixels than stated in control packet) Late end of packet (more pixels than stated in control packet) No control packet sent before video frame packet

Lab 5: Custom IP Simulation - Compile and Run the Testbench
Open ModelSim Start->Programs->Altera->Modelsim-Altera 6.4a (Quartus II 9.0) In Modelsim, change directory to <Labs>/Lab_5_Custom_IP_Simulation/template_testbench Compile testbench In the ModelSim command line, type “make compile” Run testbench and check results In the ModelSim command line, type do wave.do Some messages will be printed: This is expected for the given input data Several files will be produced – we are mostly interested in “ _received_frames_ bin” which contains the output frames We can convert these back to AVI using the utility provided Follow instructions on the next slide Contents of other generated files are explained in the functional description document Docs\Avalon_st_video_BFM_v0.doc

Lab 5: Custom IP Simulation - Converting Back to AVI
Copy the generated files “ _received_frames_ bin” and “ _received_frames_ spc” to the “avi_to_binary_utility” directory Rename files to “out.bin” and “out.spc” (less to type later) In the command window (or cygwin window): cd <Labs>\Lab_5_Custom_IP_Simulation\avi_to_binary_utility Follow the commands on the following slide The video out.avi should look the same as “expected_out.avi”

Known issue – if 0 fields generates an application error, re-run and choose number of fields = 3 for this example

The video out.avi should look the same as “expected_out.avi” Congratulations, you have successfully completed the testbench simulation exercise.

Lab 5: Custom IP Simulation – The Av-ST Video Template
Avalon-ST Input (Sink) Avalon-ST sink interface with READY_LATENCY = 1 Registers the Avalon-ST signals Provides flow control support Av-ST Video source and sink

VIP Control Packet Decoder Decodes the VIP control packets from the data stream and sends the decoded data (width, height, interlaced) as separate signals to the algorithm. The signal 'vip_ctrl_valid' goes high for one cycle as soon as the control data is valid for the next video frame. The signal 'is_video' is high when active video data are output to the algorithm. This signal is used in the flow control wrapper and not visible to the user algorithm. The signal 'end_of_video' is high when the last pixel of the active video data is output to the algorithm. Zero latency across Avalon-ST signals Av-ST Video source and sink

Wrapper Performs conversion Avalon-ST to/from convenient simple read/write interface. Removes VIP control packets data stream as they have already been decoded Only the active video data is sent to the algorithm Other control signals are passed through Lab 5: Custom IP Simulation – The Av-ST Video Template Av-ST Video source and sink

User Algorithm The user changes/adds their own algorithm here! To request input data, the 'read' signal is asserted, and incoming data is valid when 'read' is high and 'stall_in' is low. To send valid output data, the 'write' signal is asserted, and the data needs to be held while 'stall_out' is high Data is captured when 'stall_out' is low and 'write' is high, Lab 5: Custom IP Simulation – The Av-ST Video Template Av-ST Video source and sink

VIP Control Packet Encoder Encodes the VIP control packets based on the received width, height and interlaced information and inserts them into the data stream. When 'vip_ctrl_send' goes high, a VIP control packet is inserted. However, 'vip_ctrl_send' can only be asserted if 'vip_ctrl_busy' is low. This is to make sure that control packets are not inserted inside video data packets. Only active video data are input into the encoder via the Avalon- ST interface.The output stream always contains one VIP control packet, followed by one video data packet. Zero latency between respective Avalon-ST signals at input and output of the encoder. Av-ST Video source and sink

Avalon-ST Output (Source) Avalon-ST source interface with READY_LATENCY = 1 Registers the Avalon-ST signals Latency of 1 cycle between Avalon-ST input and output signals Provides flow control support Av-ST Video source and sink

Common Issues Open Discussion and Feedback

FAQs Qu. What Input Frame Rates do reference designs V1 and V2 support? Ans. Both V1 and V2 can support all input frame/field rates upto 60 frames per sec This includes Hz, 30Hz, Hz, 60Hz Qu. Where do I get Sync Timing information to parameterize the Clocked Video Output (Eg. Ans. There are some presets in the GUI (1080p60 DVI, SDI). Otherwise you will need to refer to the SMPTE standards Qu. Can there be multiple Avalon-ST Video control packets between video frames? Ans: Yes. Only the information from the last Avalon-ST control packet received before the video data is used by the core.

FAQs Qu. How many taps should I use in the scaler for upscale?
Ans. Only use 4 taps for the upscale. Using more than this degrades quality Qu. How many taps should I use in the scaler for downscale? Ans. Recommended number of taps = 4 x downscale ratio Qu. How do I pass-through data when the scaler is doing 1:1 Ans. Each tap corresponds to exactly 1 input pixel in a continuous line. Therefore, set the tap which is at the centre pixel to 1 and all others to 0 (where 1 may be larger depending on the coeffcient precision chosen) Qu. How many phases should I use? Ans. There is no visible benefit going beyond 16 Qu. Can the Color space converter convert from Bayer? Ans. No. Bayer isn’t actually a color space, it is a subsampled variation of RGB, which changes sampling pattern between lines. We do not currently have a Bayer conversion IP core. Qu. Why would I want to half the width value of a control packet? Ans. When splitting 4:2:2 subsampled video into Y and CbCr components, the CbCr image produced is half the width the orginal control packet specified, thus to correct for this the width value of the control packet can be reduced by 2x

FAQs Do I have to clock the VIP cores at the pixel clock rate?
Ans. No, you can decouple from the pixel clock domain uisng the Clocked Video Input and Clocked Video Output functions, and increase the video processing clock rate as desired Qu. How do I connect my scaler to the SDI core or an ADC? Ans. Use the Clocked Video Input and Clocked Video Output MegaCore functions Qu. What order does the Chroma resampler expect YCBCr data? Ans. CbYCrY or Y Y CbCr (LSB)

FAQs Qu. How do I decode the Av-ST Video Control Packets?
Ans. The user guide contains information on the control packets structure. However, in 9.1 the VIP User Guide will be improved to contain an expanded explanation of the Avalon-ST Video protocol..

The End Thank You

VIP Training NAB 2009.

Similar presentations

Presentation on theme: "VIP Training NAB 2009."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

VIP Training NAB 2009.

Similar presentations

Presentation on theme: "VIP Training NAB 2009."— Presentation transcript:

Similar presentations

About project

Feedback