Design Patterns from Biology for Distributed Computing Andres J. Ramirez

Paper Information F Authors F Babaoglu, Canright, Di Caro, et al. (about 11 different dudes.) F Published F ACM Transactions on Autonomous and Adaptive Systems, Vol 1, No 1, September F Authors F Babaoglu, Canright, Di Caro, et al. (about 11 different dudes.) F Published F ACM Transactions on Autonomous and Adaptive Systems, Vol 1, No 1, September 2006.

Presentation Outline F What is a design pattern? F Current challenges in designing software systems. F Parallelism to biological systems. F Presentation of design patterns extracted from biological systems. F Experimentation and validation. F Conclusion. F What is a design pattern? F Current challenges in designing software systems. F Parallelism to biological systems. F Presentation of design patterns extracted from biological systems. F Experimentation and validation. F Conclusion.

What is a design pattern? F Various definitions proposed: F “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.” [Chris Alexander] F “Each design pattern systematically names, explains and evaluates an important and recurrent design in object-oriented system.” [Gamma et al.] F “A recurring solution to a standard problem.” [Schmidt] F Overall, most are rather similar. F Various definitions proposed: F “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice.” [Chris Alexander] F “Each design pattern systematically names, explains and evaluates an important and recurrent design in object-oriented system.” [Gamma et al.] F “A recurring solution to a standard problem.” [Schmidt] F Overall, most are rather similar.

Design Pattern Presentation F Bare minimum format: F Pattern name. F Problem description. F Solution to the problem. F Consequences of applying the design pattern. F Bare minimum format: F Pattern name. F Problem description. F Solution to the problem. F Consequences of applying the design pattern.

Current Challenges F Distributed environments are common place now F Extremely dynamic. F Unreliable. F Large scale. F Traditional approaches for designing distributed systems are not applicable. F Distributed environments are common place now F Extremely dynamic. F Unreliable. F Large scale. F Traditional approaches for designing distributed systems are not applicable.

Biological Systems F Effectively organize large numbers of unreliable and dynamically changing components (cells, molecules, individuals) into structures that implement a wide range of functions. F These structures exhibit: F Robustness to failure. F Adaptability to changing conditions. F Lack of reliance on an explicit central coordinator. F Effectively organize large numbers of unreliable and dynamically changing components (cells, molecules, individuals) into structures that implement a wide range of functions. F These structures exhibit: F Robustness to failure. F Adaptability to changing conditions. F Lack of reliance on an explicit central coordinator.

Why patterns from biology? F Biological entities evolve to solve a particular problem, usually related to survival issues. F This solution, by the notion of evolution, must be well tested and reliable to be in existence today. F Similarities exist between distributed computing systems and biological systems. F Solutions from one domain can transfer onto the other. F Biological entities evolve to solve a particular problem, usually related to survival issues. F This solution, by the notion of evolution, must be well tested and reliable to be in existence today. F Similarities exist between distributed computing systems and biological systems. F Solutions from one domain can transfer onto the other.

Key Idea F Abstract design patterns from biological systems and apply them in distributed systems. F Serve as a bridge between biological systems and computer systems. F How do they accomplish this? F Formulate the patterns as local communication strategies over arbitrary communication topologies. F Abstract design patterns from biological systems and apply them in distributed systems. F Serve as a bridge between biological systems and computer systems. F How do they accomplish this? F Formulate the patterns as local communication strategies over arbitrary communication topologies.

Design Pattern Presentation in Paper F Name F Handle for the pattern. Key. F Context F Defined by the system model (more in a bit.) F Problem F Possible functionality we are trying to achieve. F Name F Handle for the pattern. Key. F Context F Defined by the system model (more in a bit.) F Problem F Possible functionality we are trying to achieve.

Design Pattern Presentation in Paper F Solution F An algorithm which produces the desired output based on the problem. F Example F Sort of a case study. F Design Rationale F The inspiration from biology. F Solution F An algorithm which produces the desired output based on the problem. F Example F Sort of a case study. F Design Rationale F The inspiration from biology.

System Model F Basic Abstraction F Network where nodes communicate via message passing. F Additional Assumptions F Basic components are nodes. F Computing devices which maintain states and perform computations. F Neighbors F Only “visible” neighbors can send messages to each other. F Asynchronous message passing F No message delivery time bound. F Basic Abstraction F Network where nodes communicate via message passing. F Additional Assumptions F Basic components are nodes. F Computing devices which maintain states and perform computations. F Neighbors F Only “visible” neighbors can send messages to each other. F Asynchronous message passing F No message delivery time bound.

System Model F Nodes are unreliable F Nodes may fail. F Can leave and join at any time. F Communication mediums are unreliable F Messages can be lost. F Side note: No mention of corrupted message passing? F Three Generals Problem does not seem to be addressed. F Maybe animals are more trustworthy than humans? F Nodes are unreliable F Nodes may fail. F Can leave and join at any time. F Communication mediums are unreliable F Messages can be lost. F Side note: No mention of corrupted message passing? F Three Generals Problem does not seem to be addressed. F Maybe animals are more trustworthy than humans?

Topology Issues F The topology here is given by the graph defined by the neighbor relation. … typical topology definition from graph theory. F Two particular networks seen in this work: F Overlay Networks F Mobile Ad Hoc Networks (MANETs) F The topology here is given by the graph defined by the neighbor relation. … typical topology definition from graph theory. F Two particular networks seen in this work: F Overlay Networks F Mobile Ad Hoc Networks (MANETs)

Overlay Networks F Promising paradigm for building applications over large-scale wide-area networks. F Service Clouds is an example. F Logical structures built on top of a physical network with a routing service. F Any node can send to any other node granted it knows the target nodes network address. F Promising paradigm for building applications over large-scale wide-area networks. F Service Clouds is an example. F Logical structures built on top of a physical network with a routing service. F Any node can send to any other node granted it knows the target nodes network address.

Mobile Ad Hoc Networks F Set of wireless mobile devices which self- organize into a network without relying on a fixed infrastructure. F All nodes are treated equal. F Neighbor relations are dependent on the wireless connections between nodes. F Defined by transmission power and physical proximity.s F Set of wireless mobile devices which self- organize into a network without relying on a fixed infrastructure. F All nodes are treated equal. F Neighbor relations are dependent on the wireless connections between nodes. F Defined by transmission power and physical proximity.s

The actual Design Patterns F Plain Diffusion F Replication F Stigmergy F Chemotaxis (composite) F Reaction Diffusion (composite) F Plain Diffusion F Replication F Stigmergy F Chemotaxis (composite) F Reaction Diffusion (composite)

Plain Diffusion F Problem: F Bring the system to a state where each node contains the average value of all the values in the system. F Assign a gradient to each link that is proportional to the change in values when following the link. F Solution: F Rely on message passing. F For each link, each node periodically subtracts a fixed proportion from its current value and sends it along the given link. On the receiving side, add the message to current value. F Problem: F Bring the system to a state where each node contains the average value of all the values in the system. F Assign a gradient to each link that is proportional to the change in values when following the link. F Solution: F Rely on message passing. F For each link, each node periodically subtracts a fixed proportion from its current value and sends it along the given link. On the receiving side, add the message to current value.

Plain Diffusion F Solution presented maintains the sum of all the values in the system constant. F All the node values will quickly approach the average value. F Gradients are generated in this process. F Solution presented maintains the sum of all the values in the system constant. F All the node values will quickly approach the average value. F Gradients are generated in this process.

Plain Diffusion F Design Rationale F A form of diffusion. F Equalizing the concentration of some substance or some abstract quantity like heat. F Present in many biological and physical systems. F Known to be efficient at convergence. This will be important when testing in a distributed environment. F Design Rationale F A form of diffusion. F Equalizing the concentration of some substance or some abstract quantity like heat. F Present in many biological and physical systems. F Known to be efficient at convergence. This will be important when testing in a distributed environment.

Replication F Problem: F Propagate novel information to all other nodes. F Assign the maximal value present in the network to all the nodes. F Find a node which contains a document matching a given query. F Problem: F Propagate novel information to all other nodes. F Assign the maximal value present in the network to all the nodes. F Find a node which contains a document matching a given query.

Replication F Solution: F Nodes receive messages from neighbors and forward them according to application specific rules. F Flooding is an easy but expensive example. F Messages can stand for the maximum value (thus solving problem 2) F Messages can stand for the query until a match is found (thus solving problem 3) F Solution: F Nodes receive messages from neighbors and forward them according to application specific rules. F Flooding is an easy but expensive example. F Messages can stand for the maximum value (thus solving problem 2) F Messages can stand for the query until a match is found (thus solving problem 3)

Replication F Design Rationale: F Replication is common place in nature F Growth processes, signal propagation in certain neural networks, epidemic spreading. F Messages can be seen as infective agents which propagate through the system invading hosts (nodes.) F Design Rationale: F Replication is common place in nature F Growth processes, signal propagation in certain neural networks, epidemic spreading. F Messages can be seen as infective agents which propagate through the system invading hosts (nodes.)

Stigmergy F Problem: F Assuming that the links between nodes have weights attached, find the shortest path between two given nodes. F Nodes need not be directly connected. F Redistribute items found in one node over a small number of nodes where similar items are held at the same node. F Does not really address when all the items are the same? Does it even matter? F Problem: F Assuming that the links between nodes have weights attached, find the shortest path between two given nodes. F Nodes need not be directly connected. F Redistribute items found in one node over a small number of nodes where similar items are held at the same node. F Does not really address when all the items are the same? Does it even matter?

Stigmergy F Solution: F Let every node contain a set of variables called stigmergic variables. F Nodes generate messages and send and received these based on application dependent policies. F Reception of a message will trigger an action. F Defined by the message itself and the stigmergic variables of the node. F Stigmergic variables are updated and then the message (also updated) is forwarded. F Essentially, distributed reinforcement learning. F Solution: F Let every node contain a set of variables called stigmergic variables. F Nodes generate messages and send and received these based on application dependent policies. F Reception of a message will trigger an action. F Defined by the message itself and the stigmergic variables of the node. F Stigmergic variables are updated and then the message (also updated) is forwarded. F Essentially, distributed reinforcement learning.

Stigmergy F In the first problem, the estimated cost for a particular path is represented by the stigmergic variables. F As it progresses, the variables are updated with more exact costs. F In the second problem, clusters form by assigning items to the messages and determining whether the message is forwarded or not based on the stigmergic variables. F In the first problem, the estimated cost for a particular path is represented by the stigmergic variables. F As it progresses, the variables are updated with more exact costs. F In the second problem, clusters form by assigning items to the messages and determining whether the message is forwarded or not based on the stigmergic variables.

Stigmergy F Design Rationale: F Typically seen in distributed self-organizing behaviors in diverse social systems. F Nest building, labor division, path finding. F Classic example, ants. F Design Rationale: F Typically seen in distributed self-organizing behaviors in diverse social systems. F Nest building, labor division, path finding. F Classic example, ants.

Chemotaxis F Note: Composite pattern based on plain diffusion. F Problem: F Finding a short path from a given node to regions of the network where the concentration of a diffusive substance Is maximal F Does not seem to incorporate finding the shortest path? F Note: Composite pattern based on plain diffusion. F Problem: F Finding a short path from a given node to regions of the network where the concentration of a diffusive substance Is maximal F Does not seem to incorporate finding the shortest path?

Chemotaxis F Solution: F Just follow the maximal gradient. F Start at any given node F Select link with highest gradient F Repeat until local maximum concentration is found. F Greedy Algorithm! Not necessarily the shortest path, and not necessarily where the highest diffusive substance is found. F Solution: F Just follow the maximal gradient. F Start at any given node F Select link with highest gradient F Repeat until local maximum concentration is found. F Greedy Algorithm! Not necessarily the shortest path, and not necessarily where the highest diffusive substance is found.

Chemotaxis F Design Rationale: F Cells or organisms might direct their movements according to the concentration gradients of one or more chemicals in the environment. F Responsible for the development of certain multicellular organisms and pattern formations. F Design Rationale: F Cells or organisms might direct their movements according to the concentration gradients of one or more chemicals in the environment. F Responsible for the development of certain multicellular organisms and pattern formations.

Reaction-Diffusion F Not a pattern, a framework covering a large set of patterns. F A strong generalization of the plain diffusion pattern F Simultaneous diffusion of one or more materials. Also removal. F Nothing else on this framework, pattern, etc. F Not a pattern, a framework covering a large set of patterns. F A strong generalization of the plain diffusion pattern F Simultaneous diffusion of one or more materials. Also removal. F Nothing else on this framework, pattern, etc.

Evaluating Design Patterns F Insensitivity: F Self-repairing F Self-organizing F Adaptive F Intelligent F Quantifying the notion of good and bad in a sense of merit. F Dependent on too many things, domain specific, not perfectly defined. F Insensitive systems show little variation in the figure of merits as the environment varies. F Insensitivity: F Self-repairing F Self-organizing F Adaptive F Intelligent F Quantifying the notion of good and bad in a sense of merit. F Dependent on too many things, domain specific, not perfectly defined. F Insensitive systems show little variation in the figure of merits as the environment varies.

Evaluating Plain Diffusion F Distributed Aggregation Problem F Calculating global functions over the set of locally known quantities. F We saw these problems earlier. F Simplify the task of controlling, monitoring and optimizing distributed applications, among other things. F Building block for other patterns. F In the paper, the average is found. F Distributed Aggregation Problem F Calculating global functions over the set of locally known quantities. F We saw these problems earlier. F Simplify the task of controlling, monitoring and optimizing distributed applications, among other things. F Building block for other patterns. F In the paper, the average is found.

Evaluating Plain Diffusion F Algorithm: F Each node p has two threads, active and passive. F Active thread: periodically initiates an information exchange with peer node q selected at random. Message contains state of p. F Passive thread: waits for a message and replies with the local state. F Symmetric information exchange, constant update of values sent and received. F The update is defined by what the problem is trying to solve. In this example, take the average of the two messages. F Could also do a maximum, etc. F Algorithm: F Each node p has two threads, active and passive. F Active thread: periodically initiates an information exchange with peer node q selected at random. Message contains state of p. F Passive thread: waits for a message and replies with the local state. F Symmetric information exchange, constant update of values sent and received. F The update is defined by what the problem is trying to solve. In this example, take the average of the two messages. F Could also do a maximum, etc.

Evaluating Plain Diffusion F How good is this solution? F Value at each node will converge to the true global average. F IF the underlying overlay network remains connected. F Just how fast does it converge? F Exponential. F Very high precision estimates are achieved in a few cycles regardless of network size. F It is scalable! F How good is this solution? F Value at each node will converge to the true global average. F IF the underlying overlay network remains connected. F Just how fast does it converge? F Exponential. F Very high precision estimates are achieved in a few cycles regardless of network size. F It is scalable!

Evaluating Plain Diffusion F Simulation done on PeerSim. F Count protocol -> number of nodes in the network. F Average calculation over a starting set of numbers. F One node has value 1, rest 0. Obtain? F 1/N. F Why do this? F Very sensitive to failures. F Tests scalability and robustness. F Simulation done on PeerSim. F Count protocol -> number of nodes in the network. F Average calculation over a starting set of numbers. F One node has value 1, rest 0. Obtain? F 1/N. F Why do this? F Very sensitive to failures. F Tests scalability and robustness.

Evaluating Plain Diffusion F Converged to a specific value exponentially, as predicted. F What about failures? F If crashed node has a smaller value than the actual global average, estimated average will increase. F N will decrease. F Opposite case? Opposite results. F Crashes have the most impact in the first few iterations. F Churn? Adding and removing nodes (N remains constant though.) F Estimates still reliable. F Converged to a specific value exponentially, as predicted. F What about failures? F If crashed node has a smaller value than the actual global average, estimated average will increase. F N will decrease. F Opposite case? Opposite results. F Crashes have the most impact in the first few iterations. F Churn? Adding and removing nodes (N remains constant though.) F Estimates still reliable.

Evaluating Replication F Distributed Search. F Idea is to spread queries throughout nodes. F Typical, simple, stupid solution? F Flood the network. F Clone the queries received at a node and propagate to all neighbors. F Huge overhead. F Opposing objectives. Higher efficiency vs lower overhead. F Can we do better? F Distributed Search. F Idea is to spread queries throughout nodes. F Typical, simple, stupid solution? F Flood the network. F Clone the queries received at a node and propagate to all neighbors. F Huge overhead. F Opposing objectives. Higher efficiency vs lower overhead. F Can we do better?

Evaluating Replication F Design the algorithm for an unstructured overlay network. F No relation between the information stored at a node and its position in the overlay network. F Learn from proliferation F Replication strategy inspired by the immune system. F Basically acts as a rate limit on propagated messages. F B cells, after being stimulated by an antigen, proliferate generating antibodies. F After this, basically a gang of antibodies do several drive- bys on the antigens and you are no longer sick! F Design the algorithm for an unstructured overlay network. F No relation between the information stored at a node and its position in the overlay network. F Learn from proliferation F Replication strategy inspired by the immune system. F Basically acts as a rate limit on propagated messages. F B cells, after being stimulated by an antigen, proliferate generating antibodies. F After this, basically a gang of antibodies do several drive- bys on the antigens and you are no longer sick!

Evaluating Replication F Treat the query as the antibody and the searched items as the antigens. F Search can be started at any node. F Send query messages to k neighbors. F Receive a message? F Calculate the similarity between query and local contents. F Higher the similarity, more messages sent out. F Only new neighbors. F Treat the query as the antibody and the searched items as the antigens. F Search can be started at any node. F Send query messages to k neighbors. F Receive a message? F Calculate the similarity between query and local contents. F Higher the similarity, more messages sent out. F Only new neighbors.

Evaluating Replication F Restricted proliferation shown to be more effective than random walks. F Even though some fluctuations were present in the results, restricted proliferation performed roughly 50% better than restricted random walk. F Key notion? F Guiding message replication to areas of more promise yields better results. F Restricted proliferation shown to be more effective than random walks. F Even though some fluctuations were present in the results, restricted proliferation performed roughly 50% better than restricted random walk. F Key notion? F Guiding message replication to areas of more promise yields better results.

No more! F I am sure I have bored you by now. F General experiment results of the remaining patterns exhibit better performance and insensitivity to traditional approaches seen in distributed computing. F Want some more specifics, look at the paper. F You did do that already, right? F Good. F I am sure I have bored you by now. F General experiment results of the remaining patterns exhibit better performance and insensitivity to traditional approaches seen in distributed computing. F Want some more specifics, look at the paper. F You did do that already, right? F Good.

Conclusions F Biological systems have evolved through millions of years to reach their current point. F Evolution happens for a reason, it is a search for a solution to survival. F We can extract some of this behavior and apply it with success to distributed computing systems. F Great amounts of parallelism between the two. F Biological systems have evolved through millions of years to reach their current point. F Evolution happens for a reason, it is a search for a solution to survival. F We can extract some of this behavior and apply it with success to distributed computing systems. F Great amounts of parallelism between the two.

Conclusions F Solutions are not perfect, but they are good. F Few patterns extracted, certainly more are possible. F Translate ideas from large, varied and seemingly unrelated systems into one language F Applicable to our domain. F Solutions are not perfect, but they are good. F Few patterns extracted, certainly more are possible. F Translate ideas from large, varied and seemingly unrelated systems into one language F Applicable to our domain.