Overview SCALE14x 2016

Agenda/Schedule -Apache Bigtop Overview -Apache Spark Overview/Getting Started -Lunch Break -Apache Ignite -Workshop, tutorial, open time (click on Agenda button)

What is Bigtop? Setting the standard for testing, packaging and integration of leading big/fast data components

and many other… Components as Building Blocks

Dependency Hell!! hdfs zookeeper hbase kafka spark. mapred oozie hive etc Build all the Things!!!

The BOM Build of Materials (BOM) * List of >=1 components * Gradle for build/actions * Produce sets of debs/rpms

Bigtop Origins Yahoo!, 2010 Created, fostered early Hadoop community Working on Hadoop 0.20 stack 2011 Yahoo!’s to Cloudera, solving early problems of packaging and maintaining first commercial supported Hadoop distro

Early value add Provide a common foundation for proper integration of growing number of Hadoop family components Foundation provides solid base for validating applications running on top of the stack(s) Provide neutral packaging and deployment/config

Early Mission Accomplished Foundation for commercial Hadoop distros/services Leveraged by app providers…

What now? We are done right?1?!?

Industry/Ecosystem Evolution & New Community Needs/Ideas

Where should we spend our time?, which users should benefit?

Moving beyond oob mapreduce…

Lambda/Stream Architectures HDFS + Zookeeper +

Get out from the Apache dome

New focus and target end users Data engineers vs distro builders Enhance Operations/Deployment Reference implementations & tutorials

Laying new foundation with 1.0+ Self-starter, non-kitchen sink building -Making gradle tooling smarter -Jenkins job autogen -leveraging containers for parallelization

Data data data… Smarter/Realistic test data -bigpetstore -bigtop-bazaar -weather data gen Tutorial/Learning Data sets -githubarchive.org -more tbd…

Deployment/Mgmt Updated puppet modules -newest best practices -next level enhanced security options Wider range of starter deployment topologies Include some handling of test/tutorial data

More components…

Sounds interesting, how can I help? *Join mailing list, ask questions, suggest features, etc *Contribute (components, tutorials, docs) *Report bugs

Thank You, Q&A Nate