1. Customer Experience & Network Evolution Plans
Robert Calderbank
VP Research, AT&T Labs

2. AT&T Labs The Innovation Engine Behind AT&T’s World-Class Technology
6,500 of the world’s best scientists and engineers
AT&T’s patent portfolio includes 1,580 granted patents
120 years of technology breakthroughs and product/service innovation
Over 80% of our scientists & technologists hold a PhD or other advanced degree
Currently involved with approximately 90 U.S. & international universities
Middletown, NJ
Menlo Park, CA
Chart Owners: Beth Mihalkovitz – HR Porition
Jack Klingert – Patents Info
Rob Calderbank – Univ Info
Last Updated
Florham Park, NJ

3. Directed Research Infrastructure is Accelerating Development of End To End Solutions
Business Model*:
Information and Operations Support to ABS that enables Customer Focused Operations across all Networks and Services, and across the customer lifecycle. Rapid Response to transform customer experience.
Business Problems: Unique capability to monitor current market and operational process leading to dialog with Product and Operations that anticipates/frames the right questions and collaboratively provides competitive advantage.
Understand and (Re)Define the Problem(s)
Data Integration: Unique capability to capture, integrate and use diverse information across silos, processes and organizations at full AT&T scale
Business Solutions: Unique capability to build scalable, flexible prototypes that can be used immediately and then improved based on experience and evolving needs
Monitor & Control
Anticipate User’s Needs
Create a Solution and Iterate
“Test and Learn”
Data Publishing
Enhance the Infrastructure
Better/Quicker Solutions each time.
*Shared across ABS and ACS

4. The Problem: So many places to look; so little time
Go to any work center and reps will be using lots and lots of systems
Provisioning, maintenance, care
Users want integration (one stop shopping)
But large systems integration projects are expensive and risky
Virtual integration: benefits of integration without the costs
Rapid cycle times: hours rather than years
Why are reps using so many systems?
Typical investigation
Log into many systems, and hope you have enough of a key to find something
Tedious, expensive, often unsuccessful
Typical scenario:
Customer calls care and expects us to find their records quickly
They don’t know how our databases are organized
May not know product(s), primary key(s), spelling of their name in our DB(s) – – – – – – – –

5. Process System Support
VIP Architecture
DBOR: Current “Factory”
MetaSearch
Local Interfaces
External Interfaces
ETE Process Models
Simulation/Optimization
Process Lifecycles
VIP GUI
Custom Views
VIP Cache
Data Staging
Detailed ETE Process
Monitor & Control
Virtual Integration Tool
Web Crawlers
DB snapshots
Direct Access
Data Access
ConnectVu
LIFE
Process Workflow
PWOT
CSR
Martin
CARE
COLR
SCOT
MACD
BMP
PIC/CIC
Data Sources
Process System Support
Current Legacy

6. Number of Queries Daily
VIT/VIP Usage
Number of Queries Daily
Over 10,000 queries/day – LIFE and CSR pulls

7. Process System Support
Integrated, Automated “Factory” Process
Current “Factory”
ETE Process Models
Simulation/Optimization
Process Lifecycles
ETE Process Models,
Monitor & Control
Built off DBoR Access
Detailed ETE Process
Monitor & Control
Virtual Integration Tool
Process Workflow
Process Workflow
Process System Support
Current Legacy
DBoR POR

8. AT&T’s Focus in 2003 and Beyond
Strategy
Reduce cycle time, consolidate
similar functions and systems, deploy workflow,
auto inventory, E-enablement, self-srv, Retire systems
Scrub DBORs, Deploy MPLS, VoIP
Basic + Managed
System monitors, correlates
and recommends action
Predictive
Customer Requirements
Consistant & Predictable quality of service
Driving
Reliability & Security
Flexibility & Simplicity
Better SLAs
System monitors correlates and takes action
Adaptive
Cybernated Network - Integrated
Components, dynamically managed
by business rules/policies
Cybernated

9. Business Grade Networking
Leveraging Scale
Traffic Crossing the Network and Active BGP Entries
Defects per Million
AT&T IP Traffic Growth: Blue
Internet Core Routes: Red
BGP Routes
Now, I’d like to discuss some other pressure points – volume growth and revenue vs. CapEx/OpEx factors.
As you can see our IP traffic is still growing exponentially. In order for us to scale, it is important that BGP routing tables growing linearly over time, and that DPMs also come down very quickly. These improvements show that IP as a basic backbone transport mechanism will scale reliably!
However, if everything is to go over IP/MPLS, and IP pricing is declining at 45% per year with the growth at perhaps 80% per year, and flat Capex, there is tremendous pressure on the carrier to be highly efficient to say the least. Note the volume vs. revenue cross over points. Data traffic exceeded voice in volume several years ago, with only a recent cross-over in revenue.
The problem is that we have Voice, FR, ATM, PL, IP, Optical, Signaling, …. networks, each with their own scaling and technology limits. So, it would seem that if everything could go over IP/MPLS then we would have a hope of scaling all networking for customers. The problem is one of network, feature, OSS and customer migration – no small task!

10. Reliability and Performance of AT&T Networks
Customer Acquisition and Growth
MIS Acquire the Traffic Program
Analysis of daily usage and content mix by potential customers, specifically large content providers such as Microsoft, Real Networks, and Speedera
Web
reports
Discords
Low level
standard
form (tables)
Abstract
network
database
polled
queries
eNetdb
Router config files
Automation
fixing errors
The “discord checks”
embody the “rules” for
configuring the service
Customer Focused Operations – Signature Client Program
Significant contraction of the time to onboard or migrate a network or customer to an AT&T network or service
Optimization of IP infrastructure in AT&T MIS being upgraded to #1 ISP from a preliminary ranking of 9th in a survey of ISP conducted by Boardwatch

11. Major Applications – Backbone – Netflow Data
http
nntp
smtp
%flows/pkts/bytes by port number
ftp-data
napster
443
dns
4041
Bytes
9995
4040
6970
kshell
1755
pop3
HTTP
web-proxy
27005
napster
Flows
2048
5000
host2-ns
ftp-ctrl
1074
1044
6901
1050
1057
1027
1036
1049
6112
6701
2002
1042
2001
1025
28800
snmp
netbios-ns
31501
By Customer
1672
4000
telnet
4020
49608
1075
vid
771
NNTP
2816
203
rest

12. Gigascope – Application Layer Monitoring & Analysis
Gigascope - next-generation packet monitor
Non-invasive
Analyzes packet data at up to OC48 link speeds
AT&T’s GSQL language allows rapid development of new queries
Example
Monitored a particular customer application with Gigascope to determine: total number of active users, packet loss rate, etc.
Results being used to understand network impact on application performance, e.g., impact of packet loss on user experience
Loss rate on AT&T backbone is well within limits
Optical
Splitter

13. Getting to an Autonomic Network
Provide predictive applications to intelligently integrate correlate, and act on network information
Detection (noticing problems as they occur)
Diagnosis (identifying where and why the problem occurred)
Repair (reliable analysis of possible changes to the network)
A global view of the data is required to make this work
Topology (routers, links, capacity)
Traffic (offered load between points in the network)
Routing (configuration of routing protocols)
Use a data distribution bus and data warehouse to
provide real time access to current and historical performance data
obtain data off the data layer, rather than have each applications poll the network
provide views of data for query or extract for non real time application needs such as customer traffic studies
Link to other DBORs for non-performance data (e.g., INSTAR for IP customer data)
Use components with open interfaces and open data models, permitting use of plug and play components at each layer

14. Systems Architecture: Instrumentation, Data, Application Layers
Product/
Sales/
Tier III
Capacity
Management
NFO/CFO/GNOC
Network Care
Reporting
Anomaly
Detection
Network
Management
(GCFP)
Network/
Customer
Traffic Studies
Capacity
Planning
Reports
Including
lightweight
publish/
subscribe
capability
Data Distribution Bus
Real Time
Performance Data
Historical Performance Data
Data Distribution Bus
Data Collectors and Active Probes

15. A Global View of the Data is Critical
What happened on these peering links? A problem or an improvement?
Without a network-wide view, see only "effects" of problems (e.g., change in link load, degradation in performance), not root causes, and have no basis for knowing how the network will behave after making a change.
IP networks use “hot potato” routing --
packets take the “best” exit among several choices, where “best” is partly under our control, and partly under peers’ and users’ independent, dynamic control

16. SQL Slammer Worm: Why so potent
At a glance:
Installed itself on vulnerable systems
Exploited buffer overflow in SQL/MSDE server software
Generated pseudorandom IP addresses
Sent worm code to those addresses
Huge installed base of vulnerable code
MSDE software embedded in large number of other applications—130+ apps (e.g., Office XP, Visio)
Many systems did not apply available patch
Patches very difficult to apply in production systems
Many admins unaware of embedded MSDE in there apps
The Worm was built to probe the entire Internet
Addresses were generated more uniformly from entire address space than previous worms like Code Red
The Worm was built for speed
cpu did little else but generate addresses and send worm payload
Saturated high-speed LANs which amplified its effects
MSDE (Microsoft Data Engine) is a stripped down version of SQL used for database functionality in other applications.

17. Sat 1/18/03 Sat 1/24/03 What WAS the Worm? Normal Traffic
Curve for UDP
Sat 1/18/03
“SQL Slammer”
worm strikes
Sat 1/24/03

18. Worm Signature
Used TAP traffic monitors to determine flow signature: UDP flows of size 404 bytes to port 1434
TAP infrastructure with “smart sampling” allowed us to see this traffic accurately all across the network - in real time!
Also running in large customer network: was able to quickly detect hundreds of infected hosts using above signature and forward to customer for action

19. Traffic Effects of the SQL.slammer worm
Worm used UDP-based traffic
Majority of internet applications are TCP based (web, chat, news, peer-to-peer file sharing)
UDP traffic does not “back off” under congestion like TCP traffic does. Thus, UDP traffic can “squeeze” TCP traffic under heavy load:
• 1st hour after worm: TCP traffic was 22% lower than usual
• 4th hour after worm: TCP traffic was 14% lower than usual
• 24 hours after worm: TCP traffic was back to normal
Worm diffused across public and private networks
Infection was anywhere the affected Microsoft software was running; did not discriminate by network
The worm only needed to breach one badly configured firewall to go on to infect an entire Intranet

20. Effects on Other Traffic
Normal WEB Traffic
Curve
Sat 1/18/03
“SQL Slammer”
worm strikes
Sat 1/24/03

21. Traffic Effects of the SQL.slammer worm
Difference between “usual” Saturday TCP/web traffic and traffic on 1/25/03

22. Systems and Networks: Evolving to the Cybernated Network
Basic
Multiple networks and systems
Managed
Integration of data and actions through management tools, and intensive manual analysis
Predictive
System monitors, correlates and recommends actions
Adaptive
System monitors, queries as needed for additional data, correlates and takes action
Autonomic/
Cybernated
Integrated components, dynamically managed by network and business rules
We are
here?
work plan
The target
is here