Streaming Applications for Robots with Real Time QoS Oct-28-2015 Supun Kamburugamuve Indiana University.

Slides:



Advertisements
Similar presentations
Big Data Open Source Software and Projects ABDS in Summary XIV: Level 14B I590 Data Science Curriculum August Geoffrey Fox
Advertisements

SDN + Storage.
Programming Models for IoT and Streaming Data IC2E Internet of Things Panel Judy Qiu Indiana University.
SALSA HPC Group School of Informatics and Computing Indiana University.
REAL-TIME COMMUNICATION ANALYSIS FOR NOCS WITH WORMHOLE SWITCHING Presented by Sina Gholamian, 1 09/11/2011.
Introduction CSCI 444/544 Operating Systems Fall 2008.
Spark: Cluster Computing with Working Sets
1 Accessing nearby copies of replicated objects Greg Plaxton, Rajmohan Rajaraman, Andrea Richa SPAA 1997.
Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn.
Routing.
COM S 614 Advanced Systems Novel Communications U-Net and Active Messages.
1 Real time signal processing SYSC5603 (ELG6163) Digital Signal Processing Microprocessors, Software and Applications Miodrag Bolic.
FLANN Fast Library for Approximate Nearest Neighbors
Pregel: A System for Large-Scale Graph Processing
Tree-Based Density Clustering using Graphics Processors
1 System Models. 2 Outline Introduction Architectural models Fundamental models Guideline.
M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.
IoTCloud Platform – Connecting Sensors to Cloud Services Supun Kamburugamuve, Geoffrey C. Fox {skamburu, School of Informatics and Computing.
Distributed Computation in MANets Robot swarm developed by James Rice University.
Chapter 3 Parallel Algorithm Design. Outline Task/channel model Task/channel model Algorithm design methodology Algorithm design methodology Case studies.
BLU-ICE and the Distributed Control System Constraints for Software Development Strategies Timothy M. McPhillips Stanford Synchrotron Radiation Laboratory.
Young Ki Baik, Computer Vision Lab.
1 Blue Gene Simulator Gengbin Zheng Gunavardhan Kakulapati Parallel Programming Laboratory Department of Computer Science.
SALSA HPC Group School of Informatics and Computing Indiana University.
RAM, PRAM, and LogP models
U.S. Department of the Interior U.S. Geological Survey Dealing with Streaming Data CDI Connected Devices Working Group Jeff Allen and Dell Long USGS Fort.
ﺑﺴﻢﺍﷲﺍﻠﺭﺣﻣﻥﺍﻠﺭﺣﻳﻡ. Group Members Nadia Malik01 Malik Fawad03.
1 Distributed and Optimal Motion Planning for Multiple Mobile Robots Yi Guo and Lynne Parker Center for Engineering Science Advanced Research Computer.
 Apache Airavata Architecture Overview Shameera Rathnayaka Graduate Assistant Science Gateways Group Indiana University 07/27/2015.
A Utility-based Approach to Scheduling Multimedia Streams in P2P Systems Fang Chen Computer Science Dept. University of California, Riverside
Lecture 4 TTH 03:30AM-04:45PM Dr. Jianjun Hu CSCE569 Parallel Computing University of South Carolina Department of.
Design an MPI collective communication scheme A collective communication involves a group of processes. –Assumption: Collective operation is realized based.
Recipes for Success with Big Data using FutureGrid Cloudmesh SDSC Exhibit Booth New Orleans Convention Center November Geoffrey Fox, Gregor von.
FIT5174 Parallel & Distributed Systems Dr. Ronald Pose Lecture FIT5174 Distributed & Parallel Systems Lecture 5 Message Passing and MPI.
MPI implementation – collective communication MPI_Bcast implementation.
Pipelined and Parallel Computing Partition for 1 Hongtao Du AICIP Research Dec 1, 2005 Part 2.
Scheduling MPI Workflow Applications on Computing Grids Juemin Zhang, Waleed Meleis, and David Kaeli Electrical and Computer Engineering Department, Northeastern.
High Performance Processing of Streaming Data Workshops on Dynamic Data Driven Applications Systems(DDDAS) In conjunction with: 22nd International Conference.
1 Plaxton Routing. 2 History Greg Plaxton, Rajmohan Rajaraman, Andrea Richa. Accessing nearby copies of replicated objects, SPAA 1997 Used in several.
Cluster computing. 1.What is cluster computing? 2.Need of cluster computing. 3.Architecture 4.Applications of cluster computing 5.Advantages of cluster.
Indiana University Faculty Geoffrey Fox, David Crandall, Judy Qiu, Gregor von Laszewski Data Science at Digital Science Center.
Chapter Five Distributed file systems. 2 Contents Distributed file system design Distributed file system implementation Trends in distributed file systems.
Application-Aware Traffic Scheduling for Workload Offloading in Mobile Clouds Liang Tong, Wei Gao University of Tennessee – Knoxville IEEE INFOCOM
SPIDAL Java High Performance Data Analytics with Java on Large Multicore HPC Clusters
Towards High Performance Processing of Streaming Data May Supun Kamburugamuve, Saliya Ekanayake, Milinda Pathirage and Geoffrey C. Fox Indiana.
INTRODUCTION TO HIGH PERFORMANCE COMPUTING AND TERMINOLOGY.
High Performance Processing of Streaming Data in the Cloud AFOSR FA : Cloud-Based Perception and Control of Sensor Nets and Robot Swarms 01/27/2016.
| presented by Vasileios Zois CS at USC 09/20/2013 Introducing Scalability into Smart Grid 1.
Cloud-based Parallel Implementation of SLAM for Mobile Robots Supun Kamburugamuve, Hengjing He, Geoffrey Fox, David Crandall School of Informatics & Computing.
Organizations Are Embracing New Opportunities
Big Data is a Big Deal!.
Distributed and Parallel Processing
Distributed Programming in “Big Data” Systems Pramod Bhatotia wp
Chilimbi, et al. (2014) Microsoft Research
Introduction | Model | Solution | Evaluation
Spark Presentation.
Apache Spark Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing Aditya Waghaye October 3, 2016 CS848 – University.
Distinguishing Parallel and Distributed Computing Performance
Accessing nearby copies of replicated objects
IEEE BigData 2016 December 5-8, Washington D.C.
Applying Twister to Scientific Applications
Distinguishing Parallel and Distributed Computing Performance
CS110: Discussion about Spark
Distinguishing Parallel and Distributed Computing Performance
Threads Chapter 4.
Indiana University, Bloomington
Real time signal processing
Parallel Programming in C with MPI and OpenMP
EdgeWise: A Better Stream Processing Engine for the Edge
Convergence of Big Data and Extreme Computing
Presentation transcript:

Streaming Applications for Robots with Real Time QoS Oct Supun Kamburugamuve Indiana University

Research at IU Digital Science Center Offloading robotic algorithms to servers Save power Increase mobility We consider expensive computations Latency and Jitter are the most important performance characteristics Need parallel computations to reduce latency Both at robot level and across robots We work with mostly constant rate data streams

Data pipeline Message Brokers RabbitMQ, Kafka Gateway Sending to pub-sub Sending to Persisting to storage Streaming workflow A stream application with some tasks running in parallel Multiple streaming workflows Streaming Workflows Apache Storm Hosted in FutureSystems OpenStack VMs which are accessible through IU network End to end delays without any processing is less than 10ms

Simultaneous Localization & Mapping (SLAM) Particles are distributed in parallel tasks Build a map given the distance measurements from robot to objects around it and its pose Streaming Workflow Parallel Speedup Rao-Blackwellized particle filtering based algorithm for SLAM. Distribute the particles across parallel tasks and compute in parallel. Map building happens periodically

Multi-Robot Collision Avoidance Streaming Workflow Information from robots Runs in parallel Velocity Obstacles (VOs) along with other constrains such as acceleration and max velocity limits, Non- Holonomic constraints, for differential robots, and localization uncertainty. Control LatencyCollisions with number of robots 5 Compute Velocity Tasks

No Cut Fluctuations decrease after Cut on #iterations per swarm member Jitter Nature of application System level influences Network Garbage collection Scheduling of tasks Virtualization SLAM Latency variations

Bringing Optimal Communications to Storm 7 Optimized 20 or 50 tasks Original 50 Tasks Original 20 Tasks Memory mapped files for intra-node communications Both process based and thread based parallelism is used Worker and Task distribution of Storm Broadcast takes advantage of task locality Binary tree broadcast

Summary of Challenges Maintaining the latency within reasonable bounds Scheduling for predictable performance Duplicate computations to remove random increases Higher level abstractions for complex streaming applications Storm API is low level (MPI like, fast) and Spark Streaming is high level (Slow). Scaling to multiple data sources

Thank You References Projects Apache Storm SLAM Collision Avoidance