© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 1 Vertica to HDFS Capstone Project University of Pittsburgh August30th, 2013 Tharanga Gamaethige, Engineer, Data Management, Vertica

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 2 Agenda What is Vertica Bridge from Vertica to HDFS Success criteria Benefits to you

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 3 What Is Vertica Founded in 2005 by database researcher Michael Stonebraker and a small group of engineers Acquired by Hewlett Packard on March 2011.

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4 What Is Vertica SQL Database for Real-time Analytics Runs on x86 hardware MPP Columnar Architecture – scales to PBs! Reduced footprint via Advanced Compression Extensible analytics capabilities Easy to setup and use Elastic - grow/shrink as needed Extensive Ecosystem of analytic tools Speed Scale Simplicity

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 5 Bridge from Vertica to HDFS Vertica database cluster HDFS cluster Use as a database to database export tool. Export data from Vertica tables into external targets e.g. to HDFS Extensible to facilitate different data formats, storage formats and data targets.

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6 Bridge from Vertica to HDFS Vertica database cluster HDFS cluster Formatter > Tuples to Blocks Prism > Blocks to Blocks Target > Blocks to Storage Pipe delimited ORC file Etc. Zip TAR Etc. HDFS File system Etc.

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7 Success criteria a)Plugin that can read data from Vertica tables and export into an external target. E.g. HDFS cluster. b)Design the plugin to be scalable to export terabytes of data. c)Design the plugin to be extensible to support different data formats (pipe delimited, ORC files, etc.), storage formats (zip, tar, plain data, etc.) and data targets (HDFS, QFS, etc.)

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8 Benefits to you Get hands-on experience in using Vertica and HDFS. Learn to provide real-life design and implementation for extensibility, in the face of big data and distributed processing. Recognition of being part of the open source community. Potential recognition from Vertica’s 1000s of customers. Most importantly free espressos, t-shirts and a coffee mug.

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 9 Thanks! Tharanga Gamaethige : Sennott Square 5404