1. 3/9/2007EECS150 Lab Lecture #81 Wireless Video Conferencing EECS150 Spring 2007 - Lab Lecture #10 Neil Warren Allen Lee

2. 3/9/2007EECS150 Lab Lecture #82 Checkpoint #3 Information(1) Check off Requirement: #1) Be able to set 2 sets different source and destination addresses on 2 boards and communicate. (CP4 video streaming) #2) Be able to set both sides’ destination address to 0xFF and a unique source address on each and have communications occur. (CP4 Handshake)

3. 3/9/2007EECS150 Lab Lecture #83 Checkpoint #3 Information If you’re behind, you can forfeit checkpoint #3 and receive a black box. You will receive a 0 for CP3 but will be able to move on to CP4 which is worth many more points. Talk to Neil if you want this option. You lose the points before you get the BB.

4. 3/9/2007EECS150 Lab Lecture #84 Welcome to Checkpoint 4 Combine CP2 and CP3 to make two-way wireless video conferencing system Note: Specification has changed to make your life much easier – based on questions from students so far. Make sure you work with the non-draft version of the Checkpoint4.doc

5. 3/9/2007EECS150 Lab Lecture #85 Overview Revisit CP2 and CP3 Communications protocol Video compression Picture-in-picture display

6. 3/9/2007EECS150 Lab Lecture #86 Recall from Checkpoint 2

7. 3/9/2007EECS150 Lab Lecture #87 Checkpoint 4 Datapath

8. 3/9/2007EECS150 Lab Lecture #88 Our Advice (1) Separate the work between you and your partner to make the best use of your time. Communications System Graphics System Agree on an interface and make sure you conform to it.

9. 3/9/2007EECS150 Lab Lecture #89 Our Advice (2) Debug Independently! Communications System: Assume that the graphics system works perfectly (conforms to your interface). Hardcode in data and requests using timers and debug the system by using the packet sniffer. Graphics System: Assume that the communications system works perfectly. Hardcode transmission inputs, and whatever interface method you use for send/receive acknowledgements. Much easier to debug each piece, then just put the 2 working pieces together. If you conform to your interface it should work!

10. 3/9/2007EECS150 Lab Lecture #810 Communications Module (1) Responsibilities Maintain communications link. Lost packet robustness Provide cross-compatibility Conform to communications protocol Encode and Decode packets and pass data to graphics engine Handle the initial handshake between master and slave.

11. 3/9/2007EECS150 Lab Lecture #811 Communications Module (2) Recommended Pieces: State Machine Keeps track of the current state of the communications module. Handles handshaking and properly re-synchronizing the communications link after lost packet. Sends commands to Packet Encoder, receives updates from Packet Decoder Packet Encoder Generate payload data for each type of packet and handle handshake with transceiver. Take commands from state machine. Packet Decoder Passes data to the graphics engine and gives status updates to the state machine.

12. 3/9/2007EECS150 Lab Lecture #812 Communications Module (3) Initial Handshaking Purpose: To allow any 2 boxes without knowledge of the other’s address to communicate. To set up the timing of master and slave send and receives. Note: At the beginning both the master and slave begin with their destination addresses being 0xFF and their source addresses being unique

13. 3/9/2007EECS150 Lab Lecture #813 Communications Module (4) Handshake Method: Master broadcasts “Master-Seek” packets once every 2.4ms. When a slave receives a “Master-Seek” packet it sends a “Slave- ReqAck” When the master receives the “Slave-ReqAck” packet it changes its destination address to the source address of the slave and then transmits a “Master-Ready” packet with the new destination address. When the slave receives a “Master-Ready” packet the slave sends a “Slave-Ready” packet and moves into a state where it can receive active video packets from the master. The “Slave- Ready” packet should be sent with its destination address as the source of the address of the master. When the master receives the “Slave-Ready” ready packet, it sends its first packet of active video data. Now the communication should alternate between the master and slave transmitting active video data.

14. 3/9/2007EECS150 Lab Lecture #814 Communications Module (5) Communications Protocol Purpose: To allow cross-compatibility between different implementations. Abstracts away the communications aspect of the project for the graphics module. The 8-bit header designates the type of packet. The 8-bit line number indicates which line the respective active video data corresponds to. The 80-bit data contains the AV data for 32 pixels. (16 pixel pairs) 8 bit headers and line numbers make it easy to view on the packet sniffer!

15. 3/9/2007EECS150 Lab Lecture #815 Communications Module (6) Type8-Bit Header 8-Bit Line Number80-Bit Data Payload Active Video 18’d18’b{Xmit Line}SAMPLE # 0-31 Active Video 28’d28’b{Xmit Line}SAMPLE # 32-63 Active Video 38’d38’b{Xmit Line}SAMPLE # 64-95 Active Video 48’d48’b{Xmit Line}SAMPLE # 96-127 Active Video 58’d58’b{Xmit Line}SAMPLE # 128-159 Active Video 68’d68’b{Xmit Line}SAMPLE # 160-176* Master-Seek8’d23XX80’h2 Slave-ReqAck8’d24XX80’h1 Master-Ready8’d25XX80’h2 Slave-Ready8’d26XX80’h1

16. 3/9/2007EECS150 Lab Lecture #816 Communications Module (7) Time Outs – let you know if a communication loss has occurred. After the expiration of 10ms from the last AV packet transmission, if the slave not does receive a packet it should return to a state where it can receive AV packets from the master. If this occurs 25 consecutive times without ever receiving an AV packet, then a communication loss is declared and the system must be reset. A message should appear on screen indicating a communication loss has occurred. After the expiration of 10ms from the last AV packet received, if the master does not receive a packet it should send its next AV packet. If this occurs 25 consecutive times without ever receiving an AV packet, then a communication loss is declared and the system must be reset. A message should appear on screen indicating a communication loss has occurred.

17. 3/9/2007EECS150 Lab Lecture #817 Communications Module (8) Tips: Make use of the packet sniffer to determine what is *actually* flying through the air. The communications protocol was re-designed specifically for this purpose.

18. 3/9/2007EECS150 Lab Lecture #818 Video Compression (1) Wireless transfer is the bottleneck Uncompressed video: 507x720x16 = 5,840,640 bits per frame That’s ~175 Mbits per second! Theoretical max bandwidth for wireless is 256kbps, so ~22.8 seconds per frame.

19. 3/9/2007EECS150 Lab Lecture #819 Video Compression (2) Simple lossy compression algorithm Resolution: 176x127 5 bits grayscale per pixel pair

20. 3/9/2007EECS150 Lab Lecture #820 Video Compression (3) Compressed Video: 176x127x5/2 = 55,880 bits per frame Approximately 105:1 compression At theoretical max wireless bandwidth, that’s 4.5 frames per second

21. 3/9/2007EECS150 Lab Lecture #821 Video Compression (4) Vertical compression Sample every other odd active line (254 active odd lines)/2 = 127 lines Horizontal compression Sample every fourth pixel pair (except for the last eight) (360-8)/4 = 88 pixel pairs = 176 pixels For each sampled pixel pair, keep only the upper five bits of the average luminance

22. 3/9/2007EECS150 Lab Lecture #822 Video Compression (5) Why 176x127 instead of 180x127? Good question. There’s a good answer, figure it out!

23. 3/9/2007EECS150 Lab Lecture #823 Video Decompression Reverse of compression, but lossy due to lack of chrominance 5-bit grayscale represents a pixel pair Decompresses to 32 bits, with appropriate padding for chrominance values

24. 3/9/2007EECS150 Lab Lecture #824 Writing Compressed Frames Set aside an area in SDRAM for sending and receiving compressed frames Update the stored frame to send only after the entire frame has been sent It is okay to lose a local frame to store a compressed one

25. 3/9/2007EECS150 Lab Lecture #825 Writing Compressed Frames Sequence for handling compressed frames: Wait for SDRAM and ChipCon initialization Store a compressed frame Tell transmitter FSM that frame in SDRAM is ready Wait as compressed frame is sent Receive signal from transmitter that last packet for frame has been sent Once entire compressed frame is sent, store a new compressed frame

26. 3/9/2007EECS150 Lab Lecture #826 Writing Compressed Frames Two options for implementing compressed video controller: Merge with VD processor, using the same FIFO Advantage: Arbiter sees one write port for local video Drawback: Need to cross clock boundaries Separate module with own FIFO Advantage: May not need to cross clock boundaries Drawback: Arbiter sees two write ports for local video Pick the one that makes more sense to you

27. 3/9/2007EECS150 Lab Lecture #827 Dual-Clock Communication Modules running on the camera clock and the system clock cannot safely communicate with direct control signals. If your modules need to communicate but are on different clocks, you must either implement a safe handshaking protocol or use the memory element we provide. For your convenience, we’ve included a dual- clocked one-bit-wide FIFO to safely exchange control signals, but you don’t have to use it.

28. 3/9/2007EECS150 Lab Lecture #828 Sender and Receiver FSMs Sender/Transmitter Waits for frame to be available in SDRAM Reads from SDRAM and packages data into a packet Requests a new frame once last packet for frame is sent Receiver Decodes the compressed data Writes received data into SDRAM

29. 3/9/2007EECS150 Lab Lecture #829 Displaying Compressed Video

30. 3/9/2007EECS150 Lab Lecture #830 Displaying Compressed Video Constantly display compressed frames to avoid SDRAM decay This way, you don’t have to auto- refresh!

31. 3/9/2007EECS150 Lab Lecture #831 Multi-Port Arbiter At least two more ports are needed for the sender and receiver FSMs. A fifth port may be needed depending on how you handle writing compressed frames.

32. 3/9/2007EECS150 Lab Lecture #832 Tips, Hints, & Common Pitfalls Start early! If you’re already done with CP3, start now! We are not providing any Verilog besides the optional one-bit-wide memory element. Run the TA solution to see what the final result should look like.

33. 3/9/2007EECS150 Lab Lecture #833 Tips, Hints, & Common Pitfalls Split up the project into graphics and communications. These parts are fairly independent so you and your partner can work in parallel. Do not combine until you’ve verified each part works independently. It can take up to 25 minutes to push to board after combining!

34. 3/9/2007EECS150 Lab Lecture #834 Tips, Hints, & Common Pitfalls Work Incrementally Display a sectioned screen Display only compressed video at full frame rate Simultaneously display local compressed and uncompressed video at full frame rate Display uncompressed video at full frame rate and local compressed at 1 FPS Test sender and receiver FSMs by sending local “packets” to yourself

35. 3/9/2007EECS150 Lab Lecture #835 Tips, Hints, & Common Pitfalls Be careful how you handle the half-full packet (packet 6): Make sure the receiver knows which half of the payload to look at for the video data

36. 3/9/2007EECS150 Lab Lecture #836 Tips, Hints, & Common Pitfalls 2 Clocks with phase offset: Camera Clock != Normal Clock Use the 1-bit wide dual-clocked fifo we provide to cross clock boundaries.

37. 3/9/2007EECS150 Lab Lecture #837 Requirements Must be TA Solution Compatible. EC can be non-compatible, but you must still develop a compatible solution first. 1 bit file. Both master and slave systems should contained on 1 bit file. Onscreen prompts. Should display to the user the communications status (see TA solution for more info) Some method of outputting the (max) time between new frames.

38. 3/9/2007EECS150 Lab Lecture #838 Extra Credit Maximum 20% Early check off 10% Some Ideas (EC% TBD): Faster frame rate Larger packet sizes Faster transmission Non-full duplex communications UI options Menus Video output modifications (change contrast with buttons, etc.) Use N64 controller for extra buttons to add functionality We will provide a black box to interface Open-ended! Come up with something special, cool, clever, unique, and/or challenging.

39. 3/9/2007EECS150 Lab Lecture #839 Questions?