M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur

Proposed Work Develop a distributed internet storage platform that will allow internet enabled devices to manage storage over the internet.

What is an Internet File System User’s view  Storage provided on the Internet.  Can access the data stored in the File System on the go.  No need to buy expensive memory.  Seamless integration into the users file system eg: Gmail Drive’ Technically,  Not a Real File System  Simulates a file system by having a central repository of folder and file links.  Files are stored on multiple storage servers.  API’s are provided for users to log in, create and manage files and folders.

Motivation - Why another Internet File System?  # of mobile internet users increasing exponentially.  Mobile clients have limited memory but growing memory requirements. – Potential User Base  Current internet file systems cannot cater to these clients due to client resource constraints (Memory, Bandwidth).  Why? What are the issues involved? How can I build a file system platform that can provide storage services to all types of clients?

Other sources of motivation  Show Gmail drive demo here

Solution The Block Level Meta File System - Files are stored and accessed at a block level. - Allows users with limited memory and bandwidth requirements to access large files. - File is downloaded in chunks as needed.

Challenges  Where Will Data Be Stored? Requiring lot’s of space is the easy part of the problem!  Availability  Scalability  Privacy and security  Efficient data updates  Storage Access and Management Problems  Customizing system to users’/applications’ needs  Designing and managing block level meta data  Adaptivity to existing or new applications How do you share storage on top of blocks?  Communication between components

Important Design Decisions  System Interface  Scalable and High Availability Architecture  Data Structures  What Meta-Data to capture? Fast Indexing User Based Views Ease of file sharing  Efficient Algorithms for creation of new files and allocation of new data blocks  File caching strategy  Technology

System Design

System Interface and Functionality  Simple API for user login (password based authentication).  APIs to create, and delete files and directories.  APIs to read and write to files at a block level granularity.  API to share files with friends

Access Server - Functionality  Users connect to the file system and access their files via the Access Server.  Client requests for File Block.  The Access Server gets the file information from the MFS Server.  Checks if file available in Cache.  If not, it connects to the storage server fetches the first block from the SS.  Delegates the task of fetching rest of the blocks to cache manager.  Send the requested data block to the client.

Key Features  Asynchronous Communication  Multithreaded – New thread for every connection (uses a thread pool).  Block Level File Caching and Pre-Fetching.  Cache Manager runs in separate thread, pre- fetches next few blocks.  Cache Replacement Policy – Least Recently Used.

Storage Server  Essentially a flat file store.  Each file can be distributed across several actual files on one or more storage servers.  How then can we allocate new files and blocks on storage servers?  Handled by the Meta File System Server.

Meta File System  Stores Meta-data related to users and files  Uses hash maps for quick indexing of user name spaces and files.  Efficient allocation of new blocks and files to storage servers. Tries to first allocate contiguous blocks so that they can be pre-fetched together. If not possible, tries to allocate all blocks on same storage server.

Design of the Meta File System

Main Contributions  Propose a Scalable, Highly-Available Architecture for a Distributed Internet File System.  Design and implement an efficient Meta-File structure for a block level file system implementation.  Implemented a simple file caching and pre-fetching mechanism, to improve efficiency of block level file access.  Design and implement APIs for accessing files block by block for client with resource constraints.  Implement a proof of concept system, and built a basic web based file storage application on this platform to demonstrate its capabilities.

Open Research Areas  Persistence of Meta File System structure  Improved File Caching Mechanism in the Access Server  Test the performance of the Block Allocation Algorithm and improve it.  Towards a Google Docs type of implementation (Multiple users can write different blocks of data simultaneously)  Handling Multimedia Files

Technologies Used  + Front End Web Application PHP + Server Apache, C++, Adaptive Communication Environment, STL/BOOST + Communication JSON over HTTP

Questions?