Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas (Computer Architecture, Communications and Systems Group) UNIVERSIDAD CARLOS III DE MADRID A parallel File System for Networks of Windows Workstations José María Pérez Jesús Carretero José Daniel García Félix García Alejandro Calderón

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –2– Outline  Introduction  Goals  Design  Evaluation  Conclusion

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –3– High performance and data storage  Growing need for high performance data storage. Growing capacity of disks. Growing data storage from applications.  I/O becomes in bottleneck.  Typical solution: Parallel I/O Join several storage resources  Large storage. Increased scalability and performance. Load balancing.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –4– Parallel File Systems  Several nodes with storage devices. Accesses performed in parallel. Data striped among nodes.  Striping allows: Parallel access to different files. Parallel access to the same file. Striping originally used in RAID.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –5– Current state  Current solutions are neither general nor flexible.  Do not use standard servers. Difficult to integrate in existing networks of workstations. Need to install new difficult servers. Available for specific platforms.  Implementation outside the operating system. A new I/O API is needed. Applications need to be modified or recompiled.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –6– Outline  Introduction  Goals  Design  Evaluation  Conclusion

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –7– WinPFS: Goal  Build a parallel file system for networks of Windows workstations using standard data sharing services (as Windows Shared Folders). A first prototype has been built using CIFS/SMB servers.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –8– Detailed goals  Integrate existing storage resources using shared folders rather than installing new servers. Accomplished by using Windows Redirectors.  Simple setup. Implemented as a new Windows File System in the kernel (a new stackable driver in the I/O hierarchy).  Easy to use. No special API’s  Applications work without recompilation.  Enhance performance, scalability and capacity. Request splitting, balanced data allocation, load balancing,...

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –9–

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –10– Outline  Introduction  Goals  Design  Evaluation  Conclusion

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –11– WinPFS design  Design based in a new Windows kernel component: A file system redirector. Implements the basis of the file system. Isolates users from the parallel file system Uses protocols to connect to different network file systems.  Redirector  redirects requests to remote servers with specific protocol (e.g.: CIFS/SMB).  WinPFS is registered as a virtual remote file system, implement the parallel I/O mechanisms and use other remote data services (redirectors).

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –12–

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –13– Remote data access User point of view  Access to remote data through shared folders.  WinPFS creates a new shared folder: \\PFS.  Users can access parallel files through this shared folder. Kernel point of view  Access through CIFS/SMB, …  Capture requests through the usage of Universal naming Convention (UNC).  Special kind of file system: a redirection of redirector drivers.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –14– File striping and requests

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –15– Layered I/O  Windows NT family has a layered I/O model. Several layers to process a request in the I/O subsystem. Each layer is a driver which can receive a request and pass it to lower layers in the I/O stack. The model allows the insertion of new layers, using new drivers. File systems are implemented as drivers in the I/O model, so new file systems can be added at the kernel level.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –16– I/O Request Management  IRP (I/O Request Packet) Describes an I/O request. Sent to kernel-mode drivers by I/O Manager (in behalf of the client).  I/O Manager Receives system calls. Creates IRP describing the request. Deliver the IRP to the appropriate driver.  MUP (Multi UNC Provider) Identifies the kernel-mode driver in charge for a network name.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –17–

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –18– Request management  Create: IRP’s are replicated and sent to each server.  Read/Write: Request split in smaller subrequests.  Create Directory: IRP’s are replicated and set to each server.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –19– Using WinPFS  Administration / Installation: Install a new driver in client nodes. Share folders in server nodes. Indicate shared folders using registry in client nodes.  User Prefix paths with \\PFS. We plan to map remote names to common driver letters.  WinPFS may be used with any API that is on top of Windows Services. Win32, POSIX, DOS, cygwin, …

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –20– Other Features  Caching. Caching mechanisms performed by redirectors. Limited to Windows caching model. More advanced caching for future work.  Security and Authentication Current model works on a Windows Domain, forests and trusted domains. Standard Windows mechanisms used to managed policies and security in enterprises, labs and departments. Uses standard Windows security model. Changes to be done for workgroup or not trusted domains.  Data consistency between clients. Currently only solved for all servers using CIFS, using the default mechanism used by CIFS redirector  oplocks (oportunistic locks).

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –21– Outline  Introduction  Goals  Design  Evaluation  Conclusion

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –22– Evaluation  Creating a file of 100 MB.  Write sequentially.  Read sequentially.  Static buffer size.  Client cache disabled.  Two clusters with four nodes.  Node BiProcessor Pentium III. 1 GHz. 1 GB main memory. 200 GB disk.  GigaEthernet network  1 Windows 2003 Server  7 Windows XP Professional

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –23– Evaluation infrastructure

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –24– Configurations  CIFS: One server.  PFS88: 8 servers in parallel.  PFS44: 4 servers in parallel.  PFS84: 4 servers in parallel and selected randomly from a set of 8. In all cases 8 clients running (1 client per node)

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –25– Write results

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –26– Read results

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –27– Results  All WinPFS solutions provide better results than CIFS.  PFS88 provides the best performance as its parallelism degree is maximum.  Performance reaches to 250 Mbit/s for write operations. Writes limited by the disks.  Performance reaches to 1200 Mbit/s for read operations. Reads limited by the network.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –28– Write Speedup PFS88/CIFS

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –29– Read Speedup PFS88/CIFS

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –30– Speedup results  Speedup is higher with more concurrent clients.  Write speedup from 500% to 700% may be achieved.  Read speedup is less 100% because data are obtained from server caches without disk accesses.  WinPFS performance is limited by the striping size buffer.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –31– Outline  Introduction  Goals  Design  Evaluation  Conclusion

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –32– Conclusions  WinPFS is a parallel file system implemented as a kernel- mode driver.  Integration into the kernel provides higher performance. Uses existing mechanisms at kernel level.  No change or recompilation needed in client applications.  We can run an application that uses parallel I/O taking advantages of the shared folders in our organizations, without affecting users. For example, launch an I/O intensive application in a classroom, and accessing to shared folders.

Grupo de Arquitectura de Computadores, Comunicaciones y Sistemas UNIVERSIDAD CARLOS III DE MADRID Expanding Windows Kernel to Integrate Heterogeneous Resources on Data Grids Page –33– Future Work  Use of Active Directory Service to create metadata repository and give a consistent image of parallel file systems. Objective: No need of manual edition of client registry to provide information about shared folders.  Evaluation with other operating systems. Linux, FreeBsd and Solaris sharing folders with Samba.  Evaluation with other protocols (redirectors). NFS (redirector provided by Services for UNIX 3.5) and WebDAV. So, a WinPFS can connect to more servers, including NAS.  Parallel usage of heterogeneous resources and protocols in networks of workstations.  Dynamically addition and removal of storage nodes.  Data allocation and load balancing for heterogeneous distributed systems.