1. Data Reduction Schemes for MicroBoone Wu, Jinyuan Fermilab

2. Mar 1-2 2010CD-1 Review2 Introduction  Waveform digitization is a necessary readout approach for TPC detectors but it creates large volume of data.  It is necessary to reduce data volume without losing useful information.  Accelerator neutrino data is compressed using lossless Huffman Coding scheme, with a typical (1/10) reduction ratio to save DAQ and data storage cost.  Dynamic Decimation and Huffman coding are applied to supernova data with a (1/60) to (1/100) total reduction ratio so that Supernova data can be taken within a reasonable equipment budget.

3. Mar 1-2 2010CD-1 Review3 The Data Paths Serial to Parallel Conversion 16MHz to 2MHz Decimation Data Merging RAM Dynamic Decimation External Memory Output Interface Huffman Coding Huffman Coding Serial to Parallel Conversion 16MHz to 2MHz Decimation Serial to Parallel Conversion 16MHz to 2MHz Decimation Serial to Parallel Conversion 16MHz to 2MHz Decimation ADC Accelerator Neutrino Events Supernova Data 1/10 Lossless 1/10 With Loss

4. Mar 1-2 2010CD-1 Review4  Three planes with Y, U and V views, 120 degrees between two views  32 wires/view  Digitized at 5 MHz, 2048 samples/event Drift Time Wire Number Data from BO detector of FNAL Induction #1 Induction #2 Collection A Typical BO Detector Event This is not a simulation

5. Mar 1-2 2010CD-1 Review5  The U(n+1)-U(n) value with highest probability is assigned to shortest code, i.e., single bit 1.  Values with lower probabilities are assigned with longer codes, e.g., 01, 001, 0001 etc.  Huffman coded words and regular words are distinguished by bit- 15. U(n+1)- U(n) Code -4 and others Full 16 bits word -3000001 -20001 01 01 +1001 +200001 +30000001 1 00 ADC value (13-bit) Regular ADC data for first point or when U(n+1)-U(n) is outside +-3 Huffman Coded 000+1+2 Padding or Continue to Next Word In this example, 6 differences of the data samples are packed in the 16-bit data word. 111111000000000 The Huffman Coding DFF Q A B A-B U(n+1) D U(n+1)-U(n)

6. Mar 1-2 2010CD-1 Review6  On typical TPC events a compression ratio of about 10 can be achieved.  Compression ratio is sensitive to high frequency noise. N N/(10.7) The Compress Ratio of Huffman Coding

7. Mar 1-2 2010CD-1 Review7 Dynamic Decimation (DD)  Only small time intervals, i.e., region of interest (ROI) must be sampled at high rate.  Most time intervals can be sampled with lower rate, without losing useful information.

8. Mar 1-2 2010CD-1 Review8 Dynamic Decimation Block  The two blocks are able to operate at up to 250MHz clock.  The Dynamic Decimation in our case reduces data by a factor of 10.  The supernova data will go through two compression stages. All data Supernova Data N N/(10)

9. Mar 1-2 2010CD-1 Review9 Any Differences ? Raw With Dynamic Decimation

10. Mar 1-2 2010CD-1 Review10 Cost and Schedule  M&S: No dedicated significant purchases/services.  To do list:  Study Huffman Coding with new coding table to further increase compression ratio. (See back up slides)  Stabilize data compression schemes.  Test the compression FPGA code in evaluation card (in hand).  Integrate data compression functions into digitization board designed by Nevis (Q3-Q4, 2011).  FNAL EE Manpower:  J. Wu: 0.4 FTE  Summer Students: 0.25 FTE 10

11. Mar 1-2 2010CD-1 Review11 The End Thanks

12. Mar 1-2 2010CD-1 Review12  The U(n) may keep unchanged for many samples and U(n+1)-U(n) value can be 0 for multiple points.  The new coding table takes this advantage.  Compression ratio can be up to 1/64, (old table is 1/15). U(n+1)- U(n) Code -4 and others Full 16 bits word -3000001 -20001 01 01 +1001 +200001 +30000001 The Huffman Coding with New Table U(n+1)- U(n) Code -4 and others Full 16 bits word -300000001 -2000001 0001 001 00001 +1001 +200001 +30000001

13. Mar 1-2 2010CD-1 Review13  More than 99% points differ from previous points by -1, 0 or +1.  Huffman Coding can be applied to the differences of the data points. DFF Q A B A-B U(n+1) D U(n+1)-U(n) Slow Variation of Raw Data

14. Mar 1-2 2010CD-1 Review14  The U(n+1)-U(n) value with highest probability is assigned to shortest code, i.e., single bit 1.  Values with lower probabilities are assigned with longer codes, e.g., 01, 001, 0001 etc.  Huffman coded words and regular words are distinguished by bit-15. U(n+1)- U(n) Code -4 and others Full 16 bits word -3000001 -20001 01 01 +1001 +200001 +30000001 1 00 ADC value (13-bit) Regular ADC data for first point or when U(n+1)-U(n) is outside +-3 Huffman Coded 000+1+2 Padding or Continue to Next Word In this example, 6 differences of the data samples are packed in the 16-bit data word. 111111000000000 The Huffman Coding

15. Mar 1-2 2010CD-1 Review15 The Huffman Coding Block  The block is able to operate at up to 250MHz clock in Altera Cyclone III FPGA devices.  The block uses 245 logic cells, taking 0.6% in an EP3C40F484C6 device ($129) containing 39600 logic cells. Raw Data Huffman Coded Data 245 Logic Cells (245/39600)*$129 = $0.80 1 00 ADC value (13-bit) 000+1+2 111111000000000

16. Mar 1-2 2010CD-1 Review16 The Knobs of Data Volume Control  The filtering schemes and parameters in the Dynamic Decimation block are knobs for data volume control.  Most of analog noises can be filtered out. Serial to Parallel Conversion 16MHz to 2MHz Decimation Data Merging RAM Dynamic Decimation External Memory Output Interface Huffman Coding Huffman Coding Serial to Parallel Conversion 16MHz to 2MHz Decimation Serial to Parallel Conversion 16MHz to 2MHz Decimation Serial to Parallel Conversion 16MHz to 2MHz Decimation ADC Total Compress Ratio: 60 - 100 from BO events. Accelerator Neutrino Events Supernova Data

17. Mar 1-2 2010CD-1 Review17 A Mystery of Dynamic Decimation & Huffman Coding  Dynamic Decimation reduces number of samples by factor of 10.  Huffman Coding reduces number of bits from raw data by factor of 10.  When cascaded, the combination reduces number of bits by factor of 60. Dynamic Decimation Huffman Coding NN/10.6 Dynamic Decimation Huffman Coding N N/60 NN/10.7

18. Mar 1-2 2010CD-1 Review18 Huffman Coding Ratios for Dynamic Decimation  The Huffman Coding compress ratio improves as the filter in Dynamic Decimation improves.

19. Mar 1-2 2010CD-1 Review19 A “ Mystery ” of Huffman Coding Ratios on Down Sampled Data  The 5MHz data is down sampled to 1MHz.  The Huffman Coding compress ratio drops from 10.7 to 7.5 when the data is down sampled. N N/(10.7) (N/5) (N/5)/(7.5)

20. Mar 1-2 2010CD-1 Review20 Averaging in Decimation: A Re-discovery  Simple “down-sampling” is not good.  When the decimation factor is D, an averaging over D samples is good either.  An averaging over 2*D samples is necessary.  There is still aliasing with averaging over 2*D samples but it is less severe than averaging over D samples. Nyquist Frequency < (1/2) Sampling Frequency

21. Mar 1-2 2010CD-1 Review21 Weighted Average, The CIC-2 Filter  Filter performance can be further improved with weighted average over 4*D samples.  The filter is called Cascade-Integrate-Comb filter of order 2 (CIC-2).  The CIC-1 filter is the moving average.

22. Mar 1-2 2010CD-1 Review22 Huffman Coding Ratios for 5MHz to 1MHz  The Huffman Coding compress ratio improves as the filter in Dynamic Decimation improves.