Introduction Distributed Algorithms for Multi-Agent Networks Instructor: K. Sinan YILDIRIM

What is distributed computing? Centralized system – composed of a single controlling unit – at most one autonomous activity is carried on at any single moment Distributed system – many autonomous processors active at any moment – communicate with each other in order to coordinate their actions and achieve a reasonable level of cooperation – non-uniform characteristic processors are physically in different geographic locations they have different in size power, architecture,… so on.

Different Types of Distributed Systems Wide spectrum of types of distributed systems – parallel computing systems – to wide-area communication networks –…–… Coupling level of different processors – Tightly-coupled distributed systems (e.g., a parallel computing systems) Processors work in tight synchrony, share memory, have very fast and reliable communication – Loosely-coupled distributed systems (e.g., a wide-area communication network) processors are more independent, communication is less frequent and less synchronous, cooperation is more limited

Purpose of Cooperation Goals – in order to handle large complex problems, e.g. parallel machines to achieve a difficult goal together – supply the user with a centralized view of the system, i.e., to make the distributed nature of the system transparent to the user, ensure that different individual processes do not disturb each other. Actual systems usually provide a mixture (to a varying degree) of both extreme types of cooperation.

A basic Framework Message passing models – treat communication explicitly – a processor sends a message to communicate with another using some available communication media. Shared memory system – processors do not speak to each other directly – there is some common memory, storing variables that are shared by all processors – variables can be written to or read from by all processors can also be used for communication between processors – it is more suitable for discussing very tightly coupled concurrent or parallel systems

Message Passing Model We shall restrict ourselves to systems based on the message passing model – locality arises mainly in physically distributed systems To discuss locality issues, it is necessary to keep the communication aspect explicit in the model, rather than hide it. Point-to-point communication systems – allow direct information exchange between specific pairs of sites. Broadcast networks – Ethernet and radio-based networks – a single message to be delivered simultaneously to a number of recipients, rather than just one at a time.

Communication Network Described by a connected undirected graph – vertices represent the sites of the network each site consists of an autonomous processor – edges represent bidirectional communication channels connecting the different sites. – a vertex communicates directly with its neighbors messages to non-neighboring vertices must be sent along some path connecting them in the graph

Communication The need for communicating information between different entities participating in the system's activities. – Non-existent in centralized computing except at the very low level of hardware considerations – Communication does not come "for free" has certain costs associated with it. – This cost may dominate other costs e.g., processing time and storage capacity – There may be limits to the speed at which we can transmit information the amounts of information that can be transmitted.

Incomplete knowledge In centralized setting – the processor carrying out the computation has full knowledge – every data item required during the computation is known and available In distributed setting – A processor has only a partial picture of the system – The input may be physically distributed among the sites – In which stage of the computation each of the participants is currently at is unknown The extent of topological knowledge – anonymous networks: vertices have no identifying labels and they initially know nothing on the topology – a more realistic model: vertices are equipped with unique identifiers and that each vertex knows the identity of its neighbors – more permissive models: allow a vertex to know the topology of its entire region – the most powerful model: assumes that complete topological knowledge of the network is maintained at every vertex.

Coping with Failures Centralized environment – a bug in the software or hardware may cause the program to terminate abnormally or even crash the system. – a hardware failure may even crash the entire computer. Distributed setting – the failure of one component may not necessarily cause other processors to go down. – the system continues to function properly and does not collapse even in the presence of some (processor or link) failures. fault-tolerant algorithms

Timing and Synchrony Synchronous model – entire system as driven by one global clock. – the machine cycle of each processor 1. Send messages to (some of) the neighbors. 2. Receive messages from (some of) the neighbors. 3. Perform some local computation. Asynchronous model – algorithms are event driven processors cannot access a global clock – messages sent from a processor to its neighbor arrive within some finite but unpredictable time. – it is also impossible to rely on the ordering of message arrivals the order of arrivals may reverse due to different message transmission speeds. Both of these models somewhat unrealistic – they help to define and limit their potential behavior, show a lower bound or an impossibility result

Asynchrony and Non-determinism Asynchronous model is inherently non-deterministic – the ordering of message deliveries The executions of any particular protocol are strongly affected by – the timing and speed of actions which in turn are affected by other events concurrently occurring in the system – the order in which messages arrive may differ from one execution to another may subsequently affect the way the computation continues

Example Running precisely the same algorithm twice, on precisely the same inputs – would yield completely different executions terminating with different outputs.

Algorithm & Protocol & Programming Protocol – Local program executed by a vertex in the network. – Governs the way each processor modifies its local data and exchanges messages with neighboring processors Distributed algorithm – consists of n protocols, each dedicated to one of the vertices. – In asynchronous setting, the algorithm must be described as event driven

Complications Distributed algorithm design & programming involves several complications – non-deterministic nature of a distributed program. – all possible executions must be correct a hidden adversary has control over the network's actions at any given execution Analyzing the properties of distributed algorithms & proving their correctness – today's one of the most active research areas!

Locality-sensitive Distributed Algorithms Traditional protocols such as routing, – maintain full information (of various forms) about what happens in the network. – as distributed networks grow larger, it becomes increasingly harder to use many of those traditional protocols, not only in the increased need for memory, but maintaining and updating this global information becomes complex and resource consuming. Global knowledge is not always essential – "locality of reference” processors to know more about their immediate neighborhood and less about the rest of the world – even in a global operation, there are many cases that sub- operations are local in nature Costs can be considerably reduced by utilizing this locality of reference!