1. Metrics for Evaluating ICEBERG ICEBERG Retreat Breakout Session Jan 11, 2000 Coordinators: Chen-Nee Chuah & Jimmy Shih

2. Goal of This Breakout Session We claim in ICEBERG that an IP-based core network can be better than the traditional telephony-based core network in terms of: Usability. Performance. Scalability. Robust Execution at a Reduced Cost. Our challenge for this breakout session is to come up with a set of metrics for evaluating ICEBERG against these claims.

3. Usability Preference Registry. How easy to setup and update preferences? Personal Activity Coordinator. How easy to interact with it? Personal Mobility. How easy to achieve this goal? Potentially Any Network Service (PANS). How easy to provide PANS given the limitations of different devices and networks? Service Creation. How easy to create new services? How much infrastructure needs to be there to allow new service creation?

4. Performance Operation in the Wide-area. End to end performance. Call set up time. Service Discovery Service/ Personal Activity Coordinator/ Preference Registry lookup time. Resource reservation set up time. Performance when services span multiple administrative domains. Performance when services span different types of networks and devices. Service Mobility. Performance during service handoffs.

5. Scalability Does ICEBERG scale to a large user base? How does the signaling complexity/overhead grow with the number of users? The number of service providers? Amount of state information needed to be stored at the ICEBERG point of presence(IPOP)? At the subsequent routers? Diversity and size of access networks that ICEBERG can provide services to? How many simultaneous calls per second can be handled?

6. Robust Execution at a Reduced Cost Fault Tolerance. Component failures. If a Call Agent breaks down, how long does it take to regenerate another Call Agent? Time to discover faults, time to recover from faults, loss of information due to faults. Availability. Network partitions. Percentage of time IPOP is available/accessible. Dependencies between components. Operation and Maintenance. Add, update and delete software agents or cached data (e.g. user preference). Security, Authentication and Privacy. Trusted computing base, time to discover attacks, damage due to attacks.

7. Deployment Issues How many people would use the ICEBERG communication services? How easy to develop new communication services? How easy to setup the system? How much does the system cost? How difficult to administer the system? How easy to extend an existing system?

8. Testing Platforms What is the reasonable analytical and simulation framework to study scalability performance? Experience learnt from web server studies: emulating multiple clients What about RMI based server? IPOP?