Data Reduction Schemes for MicroBoone Wu, Jinyuan Fermilab

Mar CD-1 Review2 Introduction  Waveform digitization is a necessary readout approach for TPC detectors but it creates large volume of data.  It is necessary to reduce data volume without losing useful information.  Accelerator neutrino data is compressed using lossless Huffman Coding scheme, with a typical (1/10) reduction ratio to save DAQ and data storage cost.  Dynamic Decimation and Huffman coding are applied to supernova data with a (1/60) to (1/100) total reduction ratio so that Supernova data can be taken within a reasonable equipment budget.

Mar CD-1 Review3 The Data Paths Serial to Parallel Conversion 16MHz to 2MHz Decimation Data Merging RAM Dynamic Decimation External Memory Output Interface Huffman Coding Huffman Coding Serial to Parallel Conversion 16MHz to 2MHz Decimation Serial to Parallel Conversion 16MHz to 2MHz Decimation Serial to Parallel Conversion 16MHz to 2MHz Decimation ADC Accelerator Neutrino Events Supernova Data 1/10 Lossless 1/10 With Loss

Mar CD-1 Review4  Three planes with Y, U and V views, 120 degrees between two views  32 wires/view  Digitized at 5 MHz, 2048 samples/event Drift Time Wire Number Data from BO detector of FNAL Induction #1 Induction #2 Collection A Typical BO Detector Event This is not a simulation

Mar CD-1 Review5  The U(n+1)-U(n) value with highest probability is assigned to shortest code, i.e., single bit 1.  Values with lower probabilities are assigned with longer codes, e.g., 01, 001, 0001 etc.  Huffman coded words and regular words are distinguished by bit- 15. U(n+1)- U(n) Code -4 and others Full 16 bits word ADC value (13-bit) Regular ADC data for first point or when U(n+1)-U(n) is outside +-3 Huffman Coded Padding or Continue to Next Word In this example, 6 differences of the data samples are packed in the 16-bit data word The Huffman Coding DFF Q A B A-B U(n+1) D U(n+1)-U(n)

Mar CD-1 Review6  On typical TPC events a compression ratio of about 10 can be achieved.  Compression ratio is sensitive to high frequency noise. N N/(10.7) The Compress Ratio of Huffman Coding

Mar CD-1 Review7 Dynamic Decimation (DD)  Only small time intervals, i.e., region of interest (ROI) must be sampled at high rate.  Most time intervals can be sampled with lower rate, without losing useful information.

Mar CD-1 Review8 Dynamic Decimation Block  The two blocks are able to operate at up to 250MHz clock.  The Dynamic Decimation in our case reduces data by a factor of 10.  The supernova data will go through two compression stages. All data Supernova Data N N/(10)

Mar CD-1 Review9 Any Differences ? Raw With Dynamic Decimation

Mar CD-1 Review10 Cost and Schedule  M&S: No dedicated significant purchases/services.  To do list:  Study Huffman Coding with new coding table to further increase compression ratio. (See back up slides)  Stabilize data compression schemes.  Test the compression FPGA code in evaluation card (in hand).  Integrate data compression functions into digitization board designed by Nevis (Q3-Q4, 2011).  FNAL EE Manpower:  J. Wu: 0.4 FTE  Summer Students: 0.25 FTE 10

Mar CD-1 Review11 The End Thanks

Mar CD-1 Review12  The U(n) may keep unchanged for many samples and U(n+1)-U(n) value can be 0 for multiple points.  The new coding table takes this advantage.  Compression ratio can be up to 1/64, (old table is 1/15). U(n+1)- U(n) Code -4 and others Full 16 bits word The Huffman Coding with New Table U(n+1)- U(n) Code -4 and others Full 16 bits word

Mar CD-1 Review13  More than 99% points differ from previous points by -1, 0 or +1.  Huffman Coding can be applied to the differences of the data points. DFF Q A B A-B U(n+1) D U(n+1)-U(n) Slow Variation of Raw Data

Mar CD-1 Review14  The U(n+1)-U(n) value with highest probability is assigned to shortest code, i.e., single bit 1.  Values with lower probabilities are assigned with longer codes, e.g., 01, 001, 0001 etc.  Huffman coded words and regular words are distinguished by bit-15. U(n+1)- U(n) Code -4 and others Full 16 bits word ADC value (13-bit) Regular ADC data for first point or when U(n+1)-U(n) is outside +-3 Huffman Coded Padding or Continue to Next Word In this example, 6 differences of the data samples are packed in the 16-bit data word The Huffman Coding

Mar CD-1 Review15 The Huffman Coding Block  The block is able to operate at up to 250MHz clock in Altera Cyclone III FPGA devices.  The block uses 245 logic cells, taking 0.6% in an EP3C40F484C6 device ($129) containing logic cells. Raw Data Huffman Coded Data 245 Logic Cells (245/39600)*$129 = $ ADC value (13-bit)

Mar CD-1 Review16 The Knobs of Data Volume Control  The filtering schemes and parameters in the Dynamic Decimation block are knobs for data volume control.  Most of analog noises can be filtered out. Serial to Parallel Conversion 16MHz to 2MHz Decimation Data Merging RAM Dynamic Decimation External Memory Output Interface Huffman Coding Huffman Coding Serial to Parallel Conversion 16MHz to 2MHz Decimation Serial to Parallel Conversion 16MHz to 2MHz Decimation Serial to Parallel Conversion 16MHz to 2MHz Decimation ADC Total Compress Ratio: from BO events. Accelerator Neutrino Events Supernova Data

Mar CD-1 Review17 A Mystery of Dynamic Decimation & Huffman Coding  Dynamic Decimation reduces number of samples by factor of 10.  Huffman Coding reduces number of bits from raw data by factor of 10.  When cascaded, the combination reduces number of bits by factor of 60. Dynamic Decimation Huffman Coding NN/10.6 Dynamic Decimation Huffman Coding N N/60 NN/10.7

Mar CD-1 Review18 Huffman Coding Ratios for Dynamic Decimation  The Huffman Coding compress ratio improves as the filter in Dynamic Decimation improves.

Mar CD-1 Review19 A “ Mystery ” of Huffman Coding Ratios on Down Sampled Data  The 5MHz data is down sampled to 1MHz.  The Huffman Coding compress ratio drops from 10.7 to 7.5 when the data is down sampled. N N/(10.7) (N/5) (N/5)/(7.5)

Mar CD-1 Review20 Averaging in Decimation: A Re-discovery  Simple “down-sampling” is not good.  When the decimation factor is D, an averaging over D samples is good either.  An averaging over 2*D samples is necessary.  There is still aliasing with averaging over 2*D samples but it is less severe than averaging over D samples. Nyquist Frequency < (1/2) Sampling Frequency

Mar CD-1 Review21 Weighted Average, The CIC-2 Filter  Filter performance can be further improved with weighted average over 4*D samples.  The filter is called Cascade-Integrate-Comb filter of order 2 (CIC-2).  The CIC-1 filter is the moving average.

Mar CD-1 Review22 Huffman Coding Ratios for 5MHz to 1MHz  The Huffman Coding compress ratio improves as the filter in Dynamic Decimation improves.