1 Presented By: Eyal Enav and Tal Rath Eyal Enav and Tal Rath Supervisor: Mike Sumszyk Mike Sumszyk

◦ Implementing real-time streaming of webcam video over LAN using System on Programmable Chip. ◦ Learning SoPC environment. ◦ Designing a system for real-time demands. ◦ Establishing a USB and Ethernet connections. ◦ Investigating and implementing suitable internet protocols. 2

 DE2 Altera board  Ethernet 10/100M port.  DM9000 Ethernet controller.  USB at full speed ISP-1362 (Philips).  Cyclone II FPGA  Nios II soft-core.  Avalon switch fabric.  Environment: SOPC builder, Quartus II and NIOS IDE II. 3

4 Cyclone II Nios II Soft Processor Avalon Switch Fabric On Chip Memory Philips ISP1362 (USB) DM9000A (Ethernet Controller) Ethernet Port 10/100M LAN Remote PC Local PC USB Port USB Cable USB Cable RJ-45

5 Camera Local PC USB - device USB - host USB Service Program DE2 USB - device Frames Nios II Ethernet Port Wait for Client request Create UDP Packets Get data From USB Send over Ethernet Remote IP & MAC Remote PC LAN UDP Receive & Video Display Program Ethernet Ether. Packets Frames Screen Network Subsystem NIOS Network Module

 DE2 is connected to LAN. ◦ DM9000A PHY/MAC Controller supplies an Ethernet interface for NIOS II.  Remote computer is connected to LAN. 6 LAN

 Real time uncompressed video stream demands.  Layers 1,2:Ethernet - DM9000A Ethernet controller.  Layer 3: IP.  Optional Transport layer protocols: ◦ TCP (Connection oriented)  Reliable and provides an ordered delivery of a stream.  Slow. ◦ UDP (Connectionless)  Fast.  Not reliable. 7

 UDP protocol was chosen due to strict speed demands.  DE2 network module was implemented. 8

1. Creates an IP,UDP and Ethernet headers that wraps the stream data. ◦ Priority fields were set to maximum priority. ◦ Destination PC’s IP and MAC address are hardcoded. 9

2. Sends packet to Ethernet controller. 3. Initializes sending process.  Packet size: Max Ethernet packet size Bytes without overheads. ◦ Reduced header overheads. 10

 Timing optimizations were made in order to gain speedup. ◦ Using DM9000A UDP and IP fast checksum functionality. ◦ Minimum delays between packets.  System test configurations: 1.PC to DE2 over Diglab network. 2.Direct cable from PC to DE2. 11

 Direct cable from PC to DE2.  Maximum transfer rate: 23.3 Mbps. ◦ Packet Error rate:  Mean: 6.12%.  Stdev:1.29%. ◦ Allows us to attain a RGB video rate of 15 FPS with a pixel resolution of 230X230 (Uncompressed). (Or 400X400 BW ) 12

13 Camera Local PC USB - device USB - host USB Service Program DE2 USB - device Frames Nios II Ethernet Port Wait for Client request Create UDP Packets Get data From USB Send over Ethernet Remote IP & MAC Remote PC LAN UDP Receive & Video Display Program Ethernet Ether. Packets Frames Screen Usb Subsystem NIOS USB Device Module

 DE2 board ◦ Connected to PC through USB port. ◦ Using DE2 Philips controller. ◦ DE2 functions as a USB device. ◦ PC functions as a USB host.  USB webcam ◦ Simpler than the DE2 connection. ◦ Generic USB webcam. ◦ Connected to PC through a USB port. 14

 Optional endpoint transfer modes: ◦ Bulk. ◦ Isochronous. ◦ Interrupt.  Isochronous mode supports real-time video streams: ◦ Guaranteed access to USB bandwidth. ◦ Bounded latency. ◦ Error detection via CRC, but no retry or guarantee of delivery. ◦ No data toggling.  Therefore, Isochronous mode was chosen and will operate in full speed mode (Maximum of 12Mbps). 15

 Complex.  Sends a series of requests to the device.  Device replies by returning filled enumerations.  Host then chooses the preferred transfer mode (In our case: Isochronous).  A continuous data transfer begins. 16

 USB Host module was implemented as a C# program. ◦ Implements data transactions between webcam and DE2 board through PC USB ports. ◦ Works under real-time demands. ◦ Uses a USB driver generated by Jungo Windriver software. ◦ Based on an open source. 17

 On power up device is being reset.  Device acquires an appropriate host-side driver by communicating with the host.  Returns answers to host requests according to USB specification.  Being set to a specific configuration by host and starts send/receive data. 18

 USB device specification document provides detailed information about device implementation.  A Bulk mode demonstration was supplied with DE2 board.  Isochronous mode was required for video streaming. 19

 The USB protocol requires correspondence between several layers: ◦ NIOS – USB Philips controller (DE2 Board). ◦ USB DE2 controller – Windows driver (DE2 Board – PC). ◦ Windows driver – Windows USB host application (PC).  The demo code that was supplied with DE2 is for Bulk transfer, which is completely different from Isochronous.  Supplied code was very messy, and could hardly be used.  USB code was written for Isochronous transfer from scratch. 20

 USB controller methodology : ◦ Consists of several Hardware Abstraction and protocol Layers (HAL). ◦ HAL code is CPU specific and may not be fully compatible with the NIOS CPU. (no documentation) ◦ An Interrupt Service Routine (ISR) should be handled by the NIOS for USB controller events. 21

22 Camera Local PC USB - device USB - host USB Service Program DE2 USB - device Frames Nios II Ethernet Port Wait for Client request Create UDP Packets Get data From USB Send over Ethernet Remote IP & MAC Remote PC LAN UDP Receive & Video Display Program Ethernet Ether. Packets Screen System Endpoints Camera USB program Movie Player Program

 Two endpoints programs written by us (C#):  “Webcam Streamer” ◦ Accepts raw frames from webcam. ◦ Disassemble frames to USB packets. ◦ Sends USB packets to device (DE2).  “Movie Streamer” ◦ Accepts UDP packets from the SoPC. ◦ Uses buffering and reassembles to video frames. ◦ Error packets are considered as black frame regions. Possible Enhancement: Using compression to improve bandwidth. Both programs work under real time demands 23

 UDP and USB Isochronous features were proved as useful when transferring uncompressed video over Ethernet.  USB implementation involves a fundamental understanding of protocol components.  A complete system was developed, and can be used for further USB and network SoPC based systems. 24

 USB Module should work in isochronous mode.  USB implementation involves a fundamental understanding of protocol components.  UDP features proved as useful when transferring uncompressed video over Ethernet. 25