1. All hat, no answers Some issues related to the evaluation of architecture John Wroclawski USC/ISI NSF FIA Meeting March 19, 2013

2. NSF 13-538: An evaluation plan… Guide and inform your work (feedback) Explain to and convince others Advance the field – a {hard, transformational} research area in itself. Why think about this?

3. Architecture high level design principles that guide the technical development of a system, especially the engineering of its protocols and algorithms Two distinct levels: A specification of system modularity, functional decomposition, interface locations and definitions, etc. – the what. A set of fundamental structuring principles that drive the choices of the first bullet – the why.

4. Architecture versus System versus Prototype

5. System A realized instantiation of a design, that meets specific requirements Well studied discipline - multiple design approaches: Waterfall… Requirements analysis Detailed system engineering Detailed component engineering Construction.. Agile… Some evaluation criteria: Meets Requirements Performance Lifecycle Cost … DesignDeployDesignDeployDesignDeploy

6. Prototype A partial system development focusing on key components or issues Proof of concept Learn whats missing Expose ideas and convince others Test key system functions quickly … Some (meta?) evaluation criteria Focused on the key open questions Learn what you need to Not misleading Leverage existing resources Quick turnaround (See Craig Partridges comments in summary from last meeting)

7. Observation Design objectives, and hence evaluation processes, are significantly different for each of these things. All are useful. But: need to tease them apart. Need to discuss each separately.

8. Network * Architecture Basic technical requirements and properties Coordination and evolution in time – multiple, evolving technologies and uses Coordination in space – multiple independent implementers Evaluation *or maybe Internet… Distinction between requirements / properties of the architecture and requirements /properties of a system. Many systems can share one architecture

9. Diversion - Elegance Architects talk about it a lot. Means what? One view – economy of mechanism, minimalist, clean Another view (or maybe not..): structural principles intuitively apparent – architecture conveys understanding, not just rules. Feels like it makes sense. Why does this matter? Guidance for evolution in space and time. An evaluation point: are the architectures structuring principles clear, intuitive, and apparent?

10. Diversion – Architectural Decay The textbook Internet: E2E arguments – fate sharing, scalability, evolvability, fault tolerance,… Layering – protocol reuse, narrow waist, ubiquitous spanning layer,… Forwarding/routing split – universal high-performance mechanism vs flexible and evolvable policy The reality: Address translation, firewalls, deep packet inspection, header rewriting, protocol converting gateways, STUN, TURN, ICE, CATS, DOGS, and who-all knows what. A goal: self-reinforcing architectures Orthogonality, tussle management, clear & minimal fixed points, active countermeasures? An evaluation point: stabilizing strategy & mechanisms..

11. Evaluation Step 1: Clarify the proposed architecture – separate from system and prototype Step 2: Identify and execute evaluation strategies

12. Evaluation Strategies Today, we have basically two. Socratic discourse Build it, achieve success, and wait 30 years In fact, this – particularly the first – has worked rather better than my snide tone might suggest With the right structure and context, people have demonstrated good ability to reason about architectural principles Telephone vs IP – strengths of each ATM QoS vs IP QoS – a clear lesson in simplicity, coordination in space, and more Discussion point: Can we foster this structure and context? How?

13. Evaluation Strategies II Still, we would like more. Lets talk about two: 1. … 2. … 3. Moving towards Theoretical Frameworks – increased rigor and more structured understanding 4. Experimental Research – contrasted with prototype deployment

14. Theoretical Frameworks We do not today have a theory of architecture – (very) far from it But – there are glimmers. From Game theory Optimization theory Various bits of economics … A possible synthesis principle for increased architectural evaluability (among other benefits…): Choose modularities with intent to bring emerging theory into scope Allows evaluation of sub-architectures using these tools

15. An example: Theoretically Derived Architectural Modularity Network resource allocation formulated as global optimization problem Primal-dual decomposition generates a set of dual problems/algorithms/modules: Local (except scheduling) Tied together through congestion prices (sub)-system architecture traceable to theoretically provable optimality.. Utility function U_s{x_s} (strictly concave function of the sending rates) Applications Congestion control Routing Scheduling Channel Cross-layer interaction in form of congestion prices (cost per unit flow of sending data along a link to a destination ) Optimal Cross-Layer Congestion Control, Routing, and Scheduling Design in Ad Hoc Wireless Networks. Lijun Chen, Steven H. Low, Mung Chiang, John C. Doyle (Caltech and Princeton)

16. Experimental Research …as distinct from prototype construction and deployment Focus on Worst case experiments Intentional perturbation Accelerated evolution Comparison (implies repeatability?) Given current state of the art, likely to involve simulation of key system components, modeling of users, etc.

17. Takeaway Points 1. Clarify the domain – Architecture vs System vs Prototype Each domain both requires and deserves separate evaluation 2. Clarify the architecture – both underlying principles and resulting structure 3. Mixed mode evaluation for a still-empirical discipline Evaluate through discourse Evaluate through analysis Evaluate through experiment Driven by reasoning Driven by increasing rigor