1. Chapter 16-1 Distributed System Structures

2. 17.2 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Chapter 16 Distributed System Structures Chapter 16.1 Background Motivation Types of Distributed Operating Systems Network Structure Chapter 16.2 Network Topology Communication Structure Communication Protocols Robustness Design Issues

3. 17.3 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Chapter Objectives To provide a high-level overview of distributed systems and the networks that interconnect them To discuss the general structure of distributed operating systems.

4. 17.4 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Background Distributed systems - collections of processors sharing neither memory nor a clock. Each processor has its own memory, their own facilities and other resources, and they communicate with each other via local area networks or wide area networks. Systems can vary greatly in size, performance, and a host of other characteristics. A distributed file system, in contrast, is a file-service system whose users, servers, and storage devices are distributed over different nodes in a distributed system. Associated file services must be carried out across a network. In distributed file systems, we do not have a single, centralized data repository. Rather, we have many independent storage devices, each with their characteristics. Benefits of distributed systems are many and varied. These include providing user access to resources maintained ‘elsewhere.’ Of course, such power breeds complexity and a host of other issues. Ability to speedup computations either my distributing computations, providing much greater availability to data, and the potential to distribute processes themselves are among the many benefits of distributed systems. All these ‘nice’ things require dealing with many issues such as deadlock and failures that are different than those in centralized systems.

5. 17.5 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Distributed System Structures Here in this chapter, we will discuss many of the general issues and communication issues that support distributed systems. Clearly communication is ‘a’ key and we will talk a lot about this in this chapter. It is part and parcel to understanding the immense issues kand complexity of distributed systems. We discuss a very general structure of a distributed system (in this chapter) and the networks needed for interconnection. We will also talk about the main differences in OS design between distributed systems and centralized systems.

6. 17.6 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Motivation In distributed systems we have a collection of loosely coupled processors interconnected by a communications network. From the view of a local user, all other processors appear to be remote. Nodews on a network may be referred to as nodes, sites, computers, machines, hosts, etc. We will say site to indicate the location of a machine and host to refer to a specific system at a site. Generally one host at some site, the server, has a resource that a host at some other site, a client, wants to use. It is as simple as this – at least conceptually.

7. 17.7 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Four Important Motivations for Distributed Systems Motivation for distributed systems are plentiful. Here are four. Resource Sharing Computation Speedup Reliability, and Communications. Each of these are very important.

8. 17.8 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Motivation Reasons -1 Resource Sharing Resource sharing refers to the ability to access resource(s) at a remote site, whether it is a processor, processes, printers, files, etc. Of course, many mechanisms, safeguards, and communications must be in place. We will discuss these ahead. Computation Speedup Some operations may be broken down into different run units that might be able to be run independently. In this case, they can be run concurrently – indeed simultaneously on a number of sites at the same time. Another example of computational speedup is simply moving a job to another site because of overloading at the resident site and running that process desired at a remote site which might have system availability. Sometimes this is called load sharing.

9. 17.9 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Motivation Reasons – 2 of 3 Reliability If computers in a distributed system are somewhat general-purpose, such that the failure of one will not necessarily affect others, then similar sites may continue to run when one node goes down. If machines are special purpose or if the system consists of a number of small machines where each has some special roles in the overall system, then there may be a problem unless there is sufficient redundancy to continue. Of course, if a failure occurs, the system must be able to recover in some way.  A specific faulty system must be taken out and its functionality must be transferrable to another node in the system, if we wish to maintain the same degree of ‘robustness’ that the distributed network offers..  When repairs are made, then this node must be able to be brought back into the system with minimal disruption.

10. 17.10 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Motivation Reasons – 3 of 3 Communications Sites are connected by a communications network- local or wide. Users must, then, be able to communicate via messages to other nodes. Availability. All message passing conventions, such as file transfer, mail, login, RPCs, etc. must be available to the nodes in a distributed system. Long distance communications. One nice thing is that all these communications may take place at large distances and are not constrained by location. Downsize central sites. One very significant consequence of a distributed system is that one potentially large site may be able to be downsized such that resources it needs may be found in other nodes in a distributed system to which it belongs.  Many adherents to the mainframe technology are replacing these boxes with smaller workstations and personal computers. The notion of sharing is becoming more and more pervasive in the industry-wide.

11. 17.11 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Types of Distributed Operating Systems Two Kinds: Network Operating Systems Distributed Operating Systems Distributed Operating Systems, by their very nature, are very complicated and are hugely reliant on much communications support and facilities for sharing Reliability distribution. All of the details of file distribution, distribution of processes, and so much more must be accommodated and must be reliable. Network Operating Systems are conceptually much simpler but more difficult for users to access and use. This is because, in principle, distributed systems should be transparent! While there are similarities between network and distributed operating, there are major differences. Let us consider these…

12. 17.12 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Network Operating Systems In a NOS, users can take advantage of many facilities that are made available to them using resources of remote machines. One big differentiation factor: users of networked systems are aware of the network and what facilities are available at which nodes, etc. Users of NOS’s either 1. remotely log in to these remote resources to secure services made available to them, or 2 download/upload data to/from remote machines by simply undertaking data transfer requests. So, let’s look at the two ways to access data and other services: Remote Login, and Remote File Transfer

13. 17.13 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Remote Login Need Valid Account. As long as a user has a valid account on a target machine, users can use a telnet program, available on the Internet, to access resources on a remote computer. Telnet allows a user to communicate with a remote machine by creating a socket connection on both computers. telnet (telecommunication network) is a network protocol used on the Internet or local area network. It was developed in 1969. Bidirectional Connection. This software results in a bidirectional connection such that everything sent from one computer is sent to the other and outputs produced by the second computer and sent back to the first.. Once this connection is established, a remote user can do most things on the receiving machine as a local user on that machine can. Telnet also refers to software which implements the client part of the protocol. But there are security issues with Telnet, and its use has waned and is mostly replaced by the use of SSH for remote access.

14. 17.14 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Remote File Transfer The facility to remotely transfer files is a bigee! NOSs provide a facility to transfer files, and the assumption is that both communicating machines do have their own local file systems. file transfer protocol (ftp) (provided by the Internet) provides the user the ability to transfer a file from a remote machine to his/her own, and vice versa. Still requires a login – usually. Procedurally, a user must invoke the ftp program (which may require a login and password) followed by a request to obtain a copy of the desired file. Once connected to appropriate subdirectory, file can be transferred. Sample commands might be: ftp cs.yale.edu followed by get Server.java Anonymous ftp is commonly available. Here, such files are usually located in a special subdirectory available for public access. A request for userid - user responds with anonymous, and Password - user may be able to enter anything or cancel, Anonymous users are only allowed access to these public directories.

15. 17.15 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Daemons for telnet and ftp Daemons are computer programs that run in the background, rather than under the direct control of a user; they are usually initiated as background processes. They are often running all the time looking for some work, or looking for some kind of network request. Systems often start (or "launch") daemons at boot time. Both telnet and ftp have daemons on the remote site that simply look for connection requests at the system’s ftp or telnet port. Once logged on, the user is now able to execute commands desired. Daemons: ftp daemon only allows a very few commands, like get, put, ls or dir, cd, etc. telnet daemon allows any command to be executed by the user. Platform Differences. It is important to note that once a user has telnetted or ftpd, they are now on a Unix machine and must use Unix commands. So, if a user is ‘Windows’ constrained, s/he may have to learn a few Unix commands! Of course, SSH and more modern techniques obviate this need.

16. 17.16 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Distributed Operating Systems Here, we will discuss the three general activities that are candidates for distribution across nodes in a distributed system: Data migration Computational migration, and Process migration It is the need for these capabilities that has brought about a huge increased in desire for distributed systems. Unfortunately, there is much complexity associated with providing these services.

17. 17.17 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Data Migration Get file transmitted from remote machine to local machine. One way to gain access to a data file on a remote machine is to get the file transmitted to a local machine. If Modified? Transfer file back. Inefficient especially for a large file. Better to only transfer the ‘part’ of the file needed vice whole file.  If additional parts of the file are needed, then they too can be downloaded – as needed (and returned if modified).  This is the approach commonly used. Downsides: Transfer entire file. If one needs several parts of the file or most of it, it is likely more efficient to pass the entire file. Transliteration. Note also that oftentimes data transliteration may be needed, different code sets with different ordering of bits, etc. All these things need to be considered and are important parts of the transfer.

18. 17.18 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Computation Migration Invoke Remote Routine Using RPCs: Using Remote Procedure Calls (RPCs) (which use a datagram protocol - UDP on the Internet), it may often be much more practical to invoke a remote routine at a remote location where, say, a desired data file resides and have that procedure access / process the file and return the result to the client site. Often much more efficient than transferring perhaps a large data file to a client site for processing. Using Messaging: Here, we can have a process on a local machine send a message to the remote site where OS will create a process for the task at hand Upon completion, process sends results back to requesting process via a message. Requesting Process can continue. The real plus about this is simply that the requesting process can continue todo other work while its message is being handled. Extrapolating, this process may have, in fact, a number of other processes executing at different sites via this messaging approach. Please note any process started at the remote site may itself request via messaging additional support from other sites in the same way as these services were invoked.

19. 17.19 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Process Migration In Computation Migration, we use requests via RPC or messaging to invoke existing processes on remote machines to do work for a local site. Process migration: Here, the process itself (or parts of it) may be moved and executed at different sites. This is done for any of the following reasons: To Balance Loads – distributes processes or sub-processes to other nodes who may have more availability than the origination local machine. To gain computational speedup. If sub-processes can be run concurrently on different sites, the overall turnaround time can be much quicker. To gain access to specialized software. Sometimes a process may require specialized only found in a remote machine. If the required software cannot be moved, or if it can be moved but it is less expensive to move the initiating process, then move (migrate) this process to the remote site. To gain access to data. If the data is considerable, it may be more efficient to move the process itself than to try to migrate the data to the local machine.

20. 17.20 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Process Migration – a bit more Two approaches here. System can hide fact that a process has been migrated to it!  Often done for load balancing or computational speedup with homogeneous systems (with few other problems…) Here we can explicitly require a user to define how the process is to migrate.  Usually done when a process must be moved to satisfy a hardware or software preference.  This is clearly a more involved approach, but it is an understandable approach when specialized hardware or special software is to be run and some ‘massaging’ might need to be done to effect the remote processing. The Web does this all the time – provides data migration between a web server and a web client – seems like all the time.. How about downloading applets, where the computation is run on the client machine. Clearly an example of process migration.

21. 17.21 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts ===Extra Stuff=== User Datagram Protocol (UDP) is one of the core protocols of the Internet Protocol Suite. Using UDP, programs on networked computers can send short messages sometimes known as datagrams (using Datagram Sockets) to one another. UDP does not guarantee reliability or ordering in the way that TCP does. Datagrams may arrive out of order, appear duplicated, or go missing without notice. Avoiding the overhead of checking whether every packet actually arrived makes UDP faster and more efficient, for applications that do not need guaranteed delivery. Time-sensitive applications often use UDP because dropped packets are preferable to delayed packets. UDP's stateless nature is also useful for servers that answer small queries from huge numbers of clients. Unlike TCP, UDP is compatible with packet broadcast (sending to all on local network) and multicasting (send to all subscribers). Common network applications that use UDP include: the Domain Name System (DNS), Voice over IP (VoIP), online games and others.

22. 17.22 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Network Structure Two main considerations: Local-Area Networks (LANs) and Wide-Area Networks (WANs)

23. 17.23 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Network Structure - Overview Distance is essentially the difference. LANs: we are talking distribution over a short distance, like within an office building or a compact campus. WANs: the venue is quite different: considering generally a number of autonomous processors distributed over a very wide area, such as the United States. Of course, there are going to be many more issues in performance, security, reliability, and communication in the WAN that must be addressed in the design of the distributed operating system.

24. 17.24 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Local Area Networks in context of Distributed Systems LANs arose to reduce expenditures associated with large mainframes in early 70s. A number of small machines, each with considerable, but often not totally enough, compute capabilities are now in wide use. Because of a concomitant need for sharing in the enterprise, usually networked. Decreased error rates; higher speeds; but higher cost cables are needed. Costs for cabling? Short distances? Costs are acceptable. Long distances? Costs may be too much. Communication Lines and Speeds: Most common links: twisted pair and fiber-optic cabling. Most common configurations: multi-access bus, ring, and star networks Communication speeds: 1 megabits/sec to 1 gigabits/sec (for gigabit Ethernet)  10BaseT Ethernet – most common and runs at 10 megabits/sec.  100BaseT Ethernet requires higher-quality cable but runs at 100 megabits/sec.  This is becoming quite common. Connect almost anything (mainframes, laptops, PDAs) printers, tapes,...gateways.) Ethernet network is the most common scheme for LANs. Ethernet network (multi-access bus) has no central controller. New hosts can be easily added to the network. But, because no central controller, there are other issues, such as contention…LANs

25. 17.25 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Wide Area Networks in Context of Distributed Systems WANs: came about in the late 60s as part of research in efficient communications among sites primarily for the DOD. Arpanet was the first in 1968. WANs cover wide areas and potentially with much diversity: Great diversity in machines, platforms, machine speeds, machines with different characteristics such as word lengths, OS, etc. WANs are often somewhat slow (slower than LANs) May have unreliable communication links. Communications Processors. WANs often use communication processors (mid sized machines) to assist in controlling and transferring data and providing interfaces for machines to communicate over the network. Use commujnication processors employing frequency and time division multiplexing to increase effective bandwidth and service; Concentrators, and other types of machines assist too. Communication Lines and Speeds: May be telephone lines, leased and sometimes dedicated, microwave links, and satellite channels.

26. 17.26 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Wide Area Networks in Context of Distributed Systems Remember, the Internet is really a network of networks. So, we often have LANs connected to regional networks which are then connected to the Internet. These regional networks are interlinked with routers (see upcoming slides) and form the worldwide network. Routers essentially control the path each message takes through the net.  For WANs, this diversity of computers, long distances, often vastly different platforms, communication lines, code sets, and more all contribute to sometimes decreased speeds, performance and cry for other attempts to optimize network performance

27. 17.27 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts WANs - more T1 Connectivity between networks often uses a telephone service called T1 that provides a transfer rate of 1.544 megabits/sec over a leased line. Interested students are encouraged to look up ‘conditioned lines’ and related topics dealing with line and line quality. T! and T3: T1s can be connected into multiple-T1 units that can run in parallel and provide better throughput. e.g. T3 consists of 28 T1 connections and has a data transfer rate of 45 megabits/sec. Some WANs simply use standard telephone lines as their primary means of communication. Modems. Many of us are familiar with modems that accept digital signals from the computer, convert them to analog signals for transmission over telephone lines, and are then converted back to digital signals by the receiving modem at the target computer. Speeds: WAN s are generally slower than LANs; transmission rates range from 1200 bits/sec to over 1 megabits / second. UUCP has been superseded by PPP, the point to point protocol. (next slides) PPP functions over modem connections, allowing home computers to be fully connected to the Internet.

28. 17.28 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts ===== UUUC =====.uucp was a pseudo-domain-style ‘suffix’ used in the late 1980s when identifying a hostname not connected directly to the Internet but possibly reachable through inter-network gateways. In this case, it indicated that the hostname preceding it was reachable by UUCP networking. This was one of several apparent "top-level domains" that were not actually in the Internet Domain Name System (DNS) root, but were sometimes used in addressing during the time when non- Internet networks remained in wide use. As UUCP hosts were not always uniquely named, and there was no official global table listing them

29. 17.29 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts ===== PPP ===== In networking, the Point-to-Point Protocol, or PPP, is a data link protocol (just above the physical layer) commonly used to establish a direct connection between two networking nodes. PPP can provide connection authentication and can also provide transmission encryption privacy and compression. PPP is used over many types of physical networks including serial cable, phone line, trunk line, cellular telephone, specialized radio links, or fiber optic links Most Internet service providers (ISPs) use PPP for customer dial-up access to the Internet. PPP is commonly used to act as a data link layer protocol for connection over synchronous and asynchronous circuits PPP was designed to work with numerous network layer protocols, including Internet Protocol (IP) and others including AppleTalk.

30. 17.30 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts ======= Routers ====== A router b whose software and hardware are usually tailored to the tasks of routing and forwarding information. They generally contain a specialized operating system (e.g. Cisco's IOS or Juniper Networks JUNOS and JUNOSe, etc. and one or more processors, as well as two or more network interfaces. High-end routers contain many processors and specialized Application-specific integrated circuits (ASIC) and do a great deal of parallel processing. Much simpler routers are used where cost is important and the demand is low, for example in providing a home internet service. With appropriate software (such as Untangle, SmoothWall, XORP or Quagga), a standard PC can act as a router. Routers connect two or more logical subnets, which do not necessarily map one- to-one to the physical interfaces of the router. The term layer 3 switch often is used interchangeably with router, but switch is really a general term without a rigorous technical definition. In marketing usage, it is generally optimized for Ethernet LAN interfaces and may not have other physical interface types.

31. 17.31 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts ======= Pictures of Routers ====== CISCO 1800 Router Cisco 7600 Routers

32. 17.32 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts End Chapter 16.1