Peer vs hierarchical groups Peer has all members equal. Common hierarchy is depth 2 tree Called coordinator-workers Fits problem solving in a master/slave manner Can have more general hierarchies Peer has better fault tollerence but harder to make decision since need to run a consensus algorithm. What to do if coord fails Give up Elect a new leader (Will discuss in section 11.3) (This is last "soft" section. Algorithms will soon appear) Group addressing Have kernel know which processes on its own machine is in which groups. Hence must just get the msgs to the correct machines. Again can use multicast, broadcast or multiple point to point Group communication uses send-receive RPC doesn't fit group comm. Msg to group is NOT like a procedure call Would have MANY replies Atomicity Either every member receives the msg or none does. Clearly useful How to implement? Can't always guarentee that all kernels will receive msg Use acks What about crashes? Expensive ALGORITHM: Original sender sends to all with acks, timers, retries Each time anyone receives a msg for the first time it does what the sender did Eventually either all the running processes get msg or none do. Go through some scenarios. HOMEWORK 10.16 Ordering revisitied Best is if have GLOBAL TIME ORDERING, i.e. msgs delivered in order sent Requires exact clock sync Too hard Next best is CONSISTENT TIME ORDERNG, i.e. delivered in same order everywhere Even this is hard and we will discuss causality and weaker orders later this chapter If processes are in multiple groups must order be preserved between groups? Often NOT required. Different groups have different functions so no need for the msgs to arrive in the same order in the two groups. Only relevant when the groups overlap. Why? If have an ethernet, get natural order since have just one msg on ethernet and all receive it. But if one is lost and must retransmit ??? Perhaps kill first msg This doesn't work with more complicated networks (gateways) HOMEWORK 10.18 (ask me in class next time) ---------------- Case study: Group communication in ISIS ---------------- Ken Birman Cornell Synchrony Synchronous System: Communication takes zero time so events happen strictly sequentially Can't build this needs absolute clock sync needs zero trans time Loosely synchronous Transmission takes nonzero time Same msg may be received at different times at different receipients All events (send and receive) occur in the SAME ORDER at all participants Two events are CAUSALLY RELATED if the first might influence the second. The first "causes" (i.e. MIGHT cause) the second. Events in the same process are causally related in program order. A send of msg M is causally related to the receive. Events not causally related are CONCURRENT Virtually synchronous Causally related events occur in the correct order in all processes. In particular in the same order No guarantees on concurrent events. HOMEWORK 10.17 At least describe what is the main problem. Let's discuss this one next time. Click for diagram in postscript or html. 

        Not permitted to deliver C before B because they are causally
        related

Message ordering (again)

    Best (Synchronous)  Msgs delivered in order sent.

    Good (Loosely Synchronous): Messages delivered in the same order to all
    members.

    OK (Virtually synchronous) Delivery order respects causality.

ISIS algorithms for ordered multicast

    ABCAST: Loosely Synchronous

    Book is incomplete

        I filled in some details that seem to fix it.

        Learning to see these counterexamples is crucial for designing
        CORRECT distributed systems

    To send a msg

        Pick a timestamp (presumably bigger than any you have seen)

        Send timestamped msg to all group members

        Wait for all acks, which are also timestamped

        Send commit with ts = max { ts of acks }

    Wnen (system, not application) receives a msg

        Send ack with ts > any it has seen or sent

    When receive a commit

        Deliver committed msgs in timestamp order

    Counterexample

        4 processes  A, B, C, D

        Messages denoted (M,t) M is msg, t is ts

        time = 1

            A sends (Ma, 1) to B, C, D

            B sends (Mb, 1) to A, C, D

        time = 2

            B receives Ma acks with ts=2

            C receives Ma acks with ts=2

            A receives Mb acks with ts=2

            D receives Mb acks with ts=2

        time = 3

            A received ack with ts=2 from B & C

            B received ack with ts=2 from A & D

            D receives Ma acks with ts=3

            C receives Mb acks with ts=3

        time = 4

            A received ack with ts=3 from D

            B received ack with ts=3 from C

        time = 5

            A and B send commit with ts=3 to B,C,D and A,C,D
            respectively

        Now can have two violations

            C and D receive both commits and since they have the same
            ts, might break the tie differently.

                Solution: Agree on tie breaking rule (say Process
                number)

            C might get A's commit and D might get B's commit.  Since
            each has only one commit, it is delivered to the program

                Solution: Keep track of all msgs you have acked but
                not received commit for.  Can't deliver msg if a lower
                numbered ack is outstanding.

    CBCAST: Loosely Synchronous

        Each process maintains a list L with n components n=groupsize

            ith component is the number of the last msg received from i

            all components are initialized to 0

        Each msg contains a vector V with n components

        For Pi to send msg

            Bump ith component of L

            Set V in msg equal to (new value of) L

        For Pj to receive msg from Pi

            Compare V to Lj.  Must have

                V(i) = Lj(i) + 1 (next msg received from i)

                V(k) <= Lj(k)  (j has seen every msg the current msg
                depends on)

            Bump ith component of L