LZRW3 Decompressor dual semester project Characterization Presentation Students: Peleg Rosen Tal Czeizler Advisors: Moshe Porian Netanel Yamin

Presentation Content Project Motivation and Goals Project Requirements Background – LZ77 algorithm LZRW3 algorithm Compressed file format Proposed Solution Decompression Example Project Top Block Diagram Planned GUI Project Schedule and Gantt

Project Motivation Why do we need an hardware data compression?

-Reduce storage capacity Project Motivation

Why do we need an hardware data compression? -Reduce storage capacity -Reduce amount of data to be handled Project Motivation

Why do we need an hardware data compression? -Reduce storage capacity -Reduce amount of data to be handled -Reduce communication costs Project Motivation

Why do we need an hardware data compression? -Reduce storage capacity -Reduce amount of data to be handled -Reduce communication costs -Speed Project Motivation

Project Goals

Implementation of LZRW3 data decompression core.

Implementation of a verification environment. Project Goals

Project Requirements

Part A: Core Requirements: – Process data at the speed of 1 Gbps. – Support data blocks with output of 2KB – 32KB. – Relay only on the FPGA’s internal memory. – VHDL Implementation.

Part A: Core Requirements: – Process data at the speed of 1 Gbps. – Support data blocks with output of 2KB – 32KB. – Relay only on the FPGA’s internal memory. – VHDL Implementation. Full simulation environment (golden model and checkers). Project Requirements

Part A: Core Requirements: – Process data at the speed of 1 Gbps. – Support data blocks with output of 2KB – 32KB. – Relay only on the FPGA’s internal memory. – VHDL Implementation. Full simulation environment (golden model and checkers). Part B: Synthesis & implementation of FPGA device (Xilinx Virtex-5). Project Requirements

Part A: Core Requirements: – Process data at the speed of 1 Gbps. – Support data blocks with output of 2KB – 32KB. – Relay only on the FPGA’s internal memory. – VHDL Implementation. Full simulation environment (golden model and checkers). Part B: Synthesis & implementation of FPGA device (Xilinx Virtex-5). GUI implementation in VisualStudio. Project Requirements

Background – LZ77 algorithm

Lossless data compression algorithm that was published by Abraham Lempel and Jacob Ziv in 1977.

Output item : (offset, length ) + next literal In this case Output item = (, ) Meaning : BABDAC ABDBCAA History bufferLook-ahead buffer B Background – LZ77 algorithm Lossless data compression algorithm that was published by Abraham Lempel and Jacob Ziv in > Take 3 literals -> Next literal will be BGo 5 steps back

Output item (Copy item): [slot address, length ] In this case Output item = [, ] BABDACABDBCAA LZRW3 compression algorithm Hash Function Hash Table ABD Slot address BAB 0 ABD Slot address Send every 3 literals to the hash function Put offset in the hash table If the slot is occupied and the literals match - make copy item 6

Compressed file format Structure

File header (8 byte) Compressed file format

Structure File header (8 byte) Groups: Compressed file format

Structure File header (8 byte) Groups: - control bytes (2 bytes) Compressed file format

Structure File header (8 byte) Groups: - control bytes (2 bytes) - data bytes ( bytes) * The last group might be smaller

Compressed file format File header contains information about the file size and whether it is compressed or not.

Compressed file format Control bytes Hold the information about the items type (copy or literal).

Compressed file format Literal items Single bytes of data in their original form.

Compressed file format Copy items Two bytes representing hash table slot holding the offset and length of a literal sequence.

Proposed Solution Header decoding Determine file size and if it is compressed or not.

Proposed Solution Header decoding Determine file size and if it is compressed or not. File decompressed! File not compressed

Proposed Solution Header decoding Determine file size and if it is compressed or not. Control bytes decoding Determine number and location of copy items. File decompressed! File compressed File not compressed

Proposed Solution Header decoding Determine file size and if it is compressed or not. Control bytes decoding Determine number and location of copy items. File decompressed! File compressed File not compressed Group decoding Literal items: 1. Store in hash table 2. Copy Copy items: 1. Decode 2. Replace 3. Copy

Proposed Solution Header decoding Determine file size and if it is compressed or not. Control bytes decoding Determine number and location of copy items. File decompressed! File compressed File not compressed Next group Group decoding Literal items: 1. Store in hash table 2. Copy Copy items: 1. Decode 2. Replace 3. Copy

Proposed Solution Header decoding Determine file size and if it is compressed or not. Control bytes decoding Determine number and location of copy items. Group decoding Literal items: 1. Store in hash table 2. Copy Copy items: 1. Decode 2. Replace 3. Copy EOF File decompressed! File compressed File not compressed Next group

Decompression example

Header decoding The header is composed of 8 bytes: Meaningless zeroes

Decompression example Header decoding The header is composed of 8 bytes: File size information: SIZE = (0 x 256) = 39 Byte 3 Byte 4

Decompression example Header decoding The header is composed of 8 bytes: Compression indicator: 0 = compressed 1 = not compressed

Decompression example Meaningless zeroes Header decoding The header is composed of 8 bytes:

Decompression example Control Bytes Analysis The control bytes are consisted of 2 bytes, each bit represent an item in the group:

Decompression example Control Bytes Analysis The control bytes are consisted of 2 bytes, each bit represent an item in the group: 0 – A literal item in the appropriate group position 1 - A copy item in the appropriate group position

Decompression example Control Bytes Analysis The control bytes are consisted of 2 bytes, each bit represent an item in the group: 0 – A literal item in the appropriate group position 1 - A copy item in the appropriate group position

Decompression example Control Bytes Analysis The control bytes are consisted of 2 bytes, each bit represent an item in the group: 0 – A literal item in the appropriate group position 1 - A copy item in the appropriate group position

Decompression example Data Bytes Analysis Bytes 11-22: literal item. Sent to hash function and copied as is to output file.

Decompression example Data Bytes Analysis Bytes 11-22: literal item. Sent to hash function and copied as is to output file.

Decompression example Data Bytes Analysis Bytes 11-22: literal item. Sent to hash function and copied as is to output file.

Compressed stream Hash func. Offset Hash table Decompression example Hashing The decompressor maintains a hash table identical to the one created by the compressor.

Decompression example Data Bytes Analysis Bytes 23-24: copy item

Decompression example Data Bytes Analysis Bytes 23-24: copy item

Decompression example Data Bytes Analysis Bytes 23-24: copy item

Decompression example Data Bytes Analysis Bytes 23-24: copy item Length = 0 + 3

Decompression example Data Bytes Analysis Bytes 23-24: copy item The number of copied literals is 3. The offset is found in slot 3645 in the hash table. Length = 0 + 3

Decompression example Data Bytes Analysis Bytes 23-24: copy item The number of copied literals is 3. The offset is found in slot 3645 in the hash table. Length = 0 + 3

Project Top Block Diagram LZRW3 DECOMPRESSION CORE

Project Top Block Diagram LZRW3 DECOMPRESSION CORE Serial compressed data bit/sec

Project Top Block Diagram LZRW3 DECOMPRESSION CORE Parallel data (bytes) wishbone protocol 125MHz REUSE

Project Top Block Diagram LZRW3 DECOMPRESSION CORE Compressed data bytes 125 MHz REUSE

Project Top Block Diagram LZRW3 DECOMPRESSION CORE Decompressed data bytes 125 MHz REUSE

Project Top Block Diagram LZRW3 DECOMPRESSION CORE Decompressed bytes wishbone protocol 125 MHz REUSE

Project Top Block Diagram LZRW3 DECOMPRESSION CORE Serial decompressed data bit/sec REUSE

Load uncompressed file PROCESS & COMPARE Input size x x Output (golden model): Messages (progress): (sending packets to FPGA/ decompressing/ getting files back/ comparing results) (browse) RAW INPUT COMPRESSED INPUT STATUS Planned GUI Load expected output file (browse) DECOMPRESS Load compressed file (browse) COMPARE Manually insert data PROCESS & COMPARE

DateGoals 21/3/2014 – 5/4/2014Project Characterization & Algorithm interpreting 6/4/2014Characterization Presentation 7/4/2014 – 20/4/2014Full Characterization of all blocks 21/4/2014 – 10/6/2014 System blocks VHDL Design 11/6/2014Mid presentation 11/6/2014 – 25/6/2014Building a simulation environment 26/6/2014 – 25/7/2014Work on project paused for exams Project Schedule 1/2

DateGoals 30/7/2014 – 3/9/2014Simulation run & debug 4/9/2014Part A - Final presentation 21/9/2014 – 5/10/2014FPGA synthesis & implementation 6/10/2014 – 23/10/2014GUI implementation 24/10/2014 – 8/11/2014Tests & debug 9/11/2014 – 23/11/2014Writing project portfolio 24/11/2014Final project presentation Project Schedule 2/2

Weeks: – – Characterization & interpretation Characterization presentation Blocks characterization VHDL blocks implementation Mid presentation Simulation environment Exams Simulation – Cont. Part A - Final pres. FPGA synthesis GUI implementation Tests & debug Writing portfolio Final presentation …… ………...…………… ….…….……………………………………………………….… Project Gantt 2 2 ….…….……………………………………….