1. Virtual SANs (VSANs) Fabric Optimization Through Fabric Consolidation
Tom Nosella, P.Eng, B.Eng Sr. Manager – Technical Marketing Storage Technology Group CCIE #1395

2. Agenda
Virtual SANs and their application
How do Virtual SANs work?

3. Common Deployment Practice – SAN Islands
Today many SAN environments consist of numerous islands of connectivity
Islands are physically isolated environments consisting of one or more interconnected switches
Each island is typically dedicated to a single or multiple related applications
Each island may be independently managed by a separate admin team
Strict isolation from faults achieved through physical isolation
‘Fan-out’ SAN Islands
‘Departmental’ SAN Islands
‘Larger Core-Edge’ SAN Island

4. SAN Islands Have Purpose – At a Cost
SAN islands are built to address several technical and non-technical issues:
Built to maintain isolation from fabric events or configuration errors
Provide isolated and controlled management of island infrastructure
Driven by bad experiences of large multi-switch fabrics
However . . .
Often over-provisioned port count for future growth – wasteful and costly
Very widespread issue today – some analysts still recommending islands
Island ‘A’
Island ‘B’
Island ‘C’

5. Introducing Virtual SANs (VSANs)
A Virtual SAN (VSAN) provides a method to allocate ports within a physical fabric to create virtual fabrics
Analogous to VLANs in Ethernet
Virtual fabrics created from larger cost-effective redundant physical fabric
Reduces wasted ports of island approach
Fabric events are isolated per VSAN – maintains isolation for HA (ie. RSCNs)
Hardware-based isolation - traffic is explicitly tagged across inter-switch links with VSAN membership info
Statistics can be gathered per VSAN
Cisco MDS 9000 Family with VSAN Service
Physical SAN islands are virtualized onto common SAN infrastructure

6. Virtualizing the Fabric – The Full Solution
To build a cost saving fabric virtualization solution, 7 key services are required:
Virtual Fabric Attachment – the ability to assign virtual fabric membership at the port level
Multiprotocol Extensions – the ability to extend virtual fabric service to iSCSI, FCIP, FICON, etc.
Virtual Fabric Services – the ability to create fabric services per virtual fabric (routing, zones, RSCNs, QoS, etc.)
Virtual Fabric Diagnostics – the ability to troubleshoot per virtual fabric problems
Virtual Fabric Security – the ability to define separate security policies per virtual fabric
Virtual Fabric Management – the ability to map and manage virtual fabrics independently
Inter-Fabric Routing – the ability to provide connectivity across virtual fabrics – without merging the fabrics
Virtual Fabric Service Model
Virtualized Fabric Attachment
Multiprotocol Transport Extensions
Virtualized Fabric Services
Virtualized Fabric Diagnostics
Virtualized Fabric Security Policies
Virtualized Fabric Management
Inter-Virtual Fabric Routing
ISL
MDS 9000 Family
Full Service End-to-End Virtual Fabric Implementation

7. Virtual SANs Reduce Infrastructure Costs
16 Port Switches
BEFORE VSANs
Virtual SANs allow dynamic provisioning and resizing of virtualized SAN islands
Virtual fabrics are built to meet initial port requirements
no stranded ports – need 40 ports? assign 40 ports
not bound by fabric switch port count configurations
Ports can be (re)assigned to VSANs non-disruptively
ISLs become Enhanced ISLs (EISLs) carrying hardware tagged traffic from multiple VSANs
ISL bandwidth is securely shared between VSANs – reduces cost of excessive ISLs
EISLs only carry permitted VSANs – can limit reach of individual VSANs
Ports Required: 70 Ports Deployed: 96 ISL Ports: 16 (7:1 fan-out) Ports Stranded: 10
Net: 96 ports for 70 used
32 Port Switches
Ports Required: 40 Ports Deployed: 64 ISL Ports: 0 Ports Stranded: 24
Net: 64 ports for 40 used
70 Port Fabric – Red_VSAN
40 Port Fabric – Blue_VSAN
WITH VSANs
Ports Required: Ports Deployed: 128 ISL Ports: 0 Ports Assignable: 18 (able to add more switching modules too!)
Net: 110 ports for 110 used

8. Virtual SANs Constrain Fault Impacts
Virtual SANs sectionalize fabric to increase availability
High availability - all fabric services are replicated and maintained per VSAN (name services, zoning services, etc)
Fabric events are isolated per VSAN – maintains isolation for HA
Misbehaving HBA or controller
Fabric rebuild event
Zone set change
etc.
Fabric recovery from a disruptive event is ‘per VSAN’ resulting in faster reconvergence – smaller scope
Future : Management segregation per VSAN
Fact: VSANs, DO contain RSCNs within the VSAN AND zone they are generated – filtering is done in hardware – no leaking
8-link bundled EISL trunk (VSAN-enabled)
!!Fabric event!!
HBA generates erroneous control frames
Faults are constrained to the extents of the VSAN and only affect devices within the VSAN

9. Differentiated Network Service – Per VSAN
Switch B
Domain 100 Domain 200
Virtual SANs enable resource allocation and preference per virtual fabric
Each VSAN runs a separate instance of FSPF routing – independent forwarding decisions per VSAN
EISL trunk links can be tuned for preferential routing per VSAN
Different recovery paths can be configured per VSANs
VSANs can be carried securely across metro and wide area networks via FCIP, SONET, DWDM, or CDWM
Quality of Service (QoS) per VSAN to give preferential treatment at points of congestion in network
QOS
Metric 50 - Red VSAN
16 Gbps EISL Trunk
Metric Blue VSAN
Metric 50 - Blue VSAN
8 Gbps EISL Trunk
Metric Red VSAN
QOS
QOS
Domain 104 Domain 204
Switch A
How do I get from switch A to switch B? It depends on which VSAN. Preferential routes can be configured per-VSAN to engineer traffic patterns.

10. VSANs as Migration Tool to SAN
Virtual SANs can be used to safely migrate applications to the SAN
Use any unused ports in the larger SAN fabric to assign to new application
Isolate new SAN-attached devices to new VSAN to help restrict fault domain
Using VSANs, control reach of traffic associated with new VSAN within larger fabric
Merge VSAN into greater SAN after proper testing and evaluation has been completed
Can simply use IVR as well to tie in new application to existing storage – no VSAN disruption
Green VSAN is used to deploy new application
Cisco MDS 9000 Family with VSAN Service
Green VSAN is used to deploy new application

11. VSANs Allow Sharing of DR Facilities
Switch A
Virtual SANs can be carried between data centers over various transport facilities
Allows consolidation ($$$) of DR facilities while still maintaining isolation
Can set preferential usage of DR transport per VSAN – routing metrics
Various wide and metro area facilities can be used securely: FCIP (PoS, ATM, Metro Ethernet), SONET, DWDM, or CDWM, etc.
Cisco MDS 9000 family provides traffic statistics per VSAN for chargeback possibility
Full fabric discovery per-VSAN through Cisco Fabric Manager
Domain 100 Domain 200
Metric 50 - Blue VSAN
1Gbps EISL Trunk
Metric Red VSAN
8 Gbps EISL Trunk
Metric 50 - Red VSAN
Metric Blue VSAN
DWDM or CWDM
SONET or SDH
IP Routed Network (FCIP)
Domain 104 Domain 204
Switch B
How do I get from switch A to switch B? Preferential routes can be configured per-VSAN helping utilize WAN/MAN transport facilities appropriately to meet the application needs. All this while still maintaining backup paths in case of path failures

12. iSCSI Adds to Connectivity Options
iSCSI offers additional connectivity options which help address mid-range server connection costs
iSCSI is implemented on the MDS 9000 to mimic Fibre Channel attachment
iSCSI hosts are discovered and displayed through the Cisco Fabric Manager
iSCSI hosts are bound to assigned WWNs creating a static relationship enabling :
Zoning of iSCSI initiators
Accounting against iSCSI initiators
Topology mapping of iSCSI initiators
iSCSI is offered through the IPS module in the MDS 9000 Family of switches
8 x 1Gb Ethernet connections capable of wire-rate iSCSI and/or FCIP
Configured through Cisco Fabric Manager
Offer HA features such as VRRP
Cisco IP Storage Switching Module
iQN2 is mapped to an allocated pWWN and registered in the fabric
IP-VRRP support for redundant IP gateway services
iSCSI
iQN2 = pWWN2
Cisco Catalyst 6500 Multilayer LAN Switches
iSCSI Login registering iQN
iSCSI qualified names are defined within iSCSI client
iSCSI
iSCSI
iQN1
iQN2
iSCSI-enabled hosts

13. iSCSI Leveraging VSANs
VSANs are extended to iSCSI transparently through address mapping
iSCSI is implemented on the MDS 9000 to mimic Fibre Channel attachment
iSCSI hosts are discovered and displayed through the Cisco Fabric Manager
iSCSI hosts are bound to assigned WWNs creating a static relationship enabling :
iSCSI host VSAN membership
Zoning of iSCSI initiators
Accounting against iSCSI initiators
Topology mapping of iSCSI initiators
iSCSI is offered through the IPS module in the MDS 9000 Family of switches
8 x 1Gb Ethernet connections capable of wire-rate iSCSI and/or FCIP
Configured through Cisco Fabric Manager
Offer HA features such as VRRP
Cisco IP Storage Switching Module
IPS
iSCSI
Cisco Catalyst 6500 Multilayer LAN Switches
iQN2 = pWWN2
iSCSI Login registering iQN
iSCSI
iSCSI
iQN1
iQN2
iSCSI-enabled hosts

14. FICON Intermixing Leveraging VSANs
Industry First!
FICON Intermixing Leveraging VSANs
Application / Department based SAN Islands
Z/OS (FICON) VSAN
Cisco MDS 9000 Family
FICON
Z-Series
Z/OS
Applicaitons
FICON Channel
Mainframe Storage
FICON
FICON
Common Storage Pool Shared Amongst VSANs
FICON FC
Z-Series Linux
FICON
FICON
LINUX
Applicaitons
Fibre Channel
Mainframe Storage FC
Linux
VSAN
Open Systems VSAN FC
Collapsed Fabric with VSANs
Open Systems
Servers and Applications
Fibre Channel
Open Systems Storage
Clean partitioning of different operating environments (FICON, Z-Series Linux-FCP, Open Systems FCP)
Significantly more stable and manageable than current zoning+best practices approach FC
Separate physical fabrics
Over-provisioning ports on each island
High number of switches to manage

15. Inter-VSAN Routing (IVR) : Sharing Resources Across VSANs
Industry First!
Inter-VSAN Routing (IVR) : Sharing Resources Across VSANs
Allows sharing of centralized storage services such as tape libraries and disks across VSANs – without merging separate fabrics (VSANs)
Provides high fabric resiliency and VSAN-based manageability
Works for all MDS 9000 switches with a software upgrade to SAN-OS 1.3(1)
Distributed, scaleable, and highly resilient architecture
Transparent to third-party switches
Enables blade-per-VSAN architecture for blade servers
VSAN-specifc Disk
Engineering VSAN_1
IVR
Marketing VSAN_2
Blade Server
IVR
IVR
Tape SAN_4 (access via IVR)
HR VSAN_3
Blade Server VSAN_1 (access via IVR)
Marketing VSAN_2
HR VSAN_3

16. Inter-VSAN Routing (IVR): Resilient SAN Extension Solutions
Industry First!
Inter-VSAN Routing (IVR): Resilient SAN Extension Solutions
Minimize the impact of change in fabric services across geographically dispersed sites
Limit fabric control traffic such as SW-RSCNs and Build/Reconfigure Fabric (BF/RCF) to local VSANs
Flexible connectivity with the highest availability
Works with any transport service (FC, SONET, DWDM/CWDM, FCIP)
Inter-VSAN Connection with Completely Isolated Fabrics
EISL#1 in Port Channel
Replication VSAN_1
Replication VSAN_4
Metro DWDM (or SONET/SDH or FCIP)
IVR
IVR
Transit VSAN_3 (IVR)
Local VSAN_2
Local VSAN_5
EISL#2 in Port Channel

17. Fabric-Based Virtualization Leveraging VSANs
Challenge: Optimize storage usage while supporting heterogeneous storage
Virtual Targets with Virtual LUNs are built from discovered physical storage
Virtual LUNs and targets can be zoned to destined host(s)
Separate VSAN used to isolate physical storage
Ability to virtualize across multiple vendors’ storage arrays
Cisco working with several partners to deliver solutions
Group ‘A’
Group ‘B’
TARG 1
TARG 2
TARG 3
TARG 1
TARG 2
50G
200G
100G
10G
200G
20G
300G
40G
240G
300G
50G
50G
125G
. . .
. . .
Virtual Enclosure
Storage VSAN
Backup VSAN
Shared Storage Pool
Main Data Center

18. VSANs and Zones - Complimentary
Virtual SANs and fabric zoning are very complimentary
Hierarchical relationship –
First assign physical ports to VSANs
Then configure independent zones per VSAN
VSANs divide the physical infrastructure
Zones provide added security and allow sharing of device ports
VSANs provide traffic statistics
VSANs only changed when ports needed per virtual fabric
Zones can change frequently (eg. backup)
Ports are added/removed non-disruptively to VSANs
Relationship of VSANs to Zones
Physical Topology
VSAN 2
Disk2
Disk3
ZoneA
Disk1
Host1
ZoneC
Disk4
Host2
ZoneB
VSAN 3
ZoneD
Host4
ZoneA
Host3
Disk5
Disk6

19. VSANs: Continual Innovation and Evolution
VSANs are key technology leveraged from large scale Ethernet datacenter expertise and technology (VLANs)
VSANs represent key innovation in SAN design, provisioning and management
Continual enhancements to further drive innovative cost-effective SAN solutions
Cisco has proposed VSAN technology to the ANSI T11 FS-SP group
SAN-OS v1.1
SAN-OS v1.3
SAN-OS v1.0
VSANs are extended to iSCSI and FCIP services
VSAN-assigned iSCSI hosts
VSAN-to-VLAN mapping
VSAN trunking over FCIP
VSANs enable cost-saving consolidation of SAN Islands
Accounting per VSAN
Separate services per VSAN zoneset, FSPF, RSCN, etc
Separate policies per VSAN
VSAN-based SPAN feature
VSAN-based accounting
SAN-OS v1.3
SAN-OS v1.2
FICON, fabric, and routing enhancements
VSAN-based FICON services (FICON VSANs)
FICON intermix with VSANs
Inter-VSAN routing
VSAN-based fabric timers
VSAN-based QoS services
SAN-OS v1.2
SAN-OS v1.1
VSAN-based management enhancements
VSAN-specific roles - RBAC role assigned to VSAN scope
- Admin only sees ports in assigned VSAN(s)
SAN-OS v1.0

20. How Do Virtual SANs Work?

21. Two Primary Functions of VSANs
The Virtual SANs feature consists of two primary functions:
Hardware-based frame tagging of traffic belonging to different VSANs – Hardware Isolation
No special drivers or configuration required for end nodes (hosts, disks, etc)
Traffic tagged at Fx_Port ingress and carried across EISL (enhanced ISL) links between switches
Create independent partition of Fibre Channel services for each newly created VSAN – services include:
Zone server, name server, management server, principle switch election, etc.
Each service runs independently and is managed/configured independently
Fibre Channel Services for Blue VSAN
VSAN header is removed at egress point
Fibre Channel Services for Red VSAN
Cisco MDS 9000 Family with VSAN Service
Trunking E_Port (TE_Port)
Enhanced ISL (EISL) Trunk carries tagged traffic from multiple VSANs
Trunking E_Port (TE_Port)
VSAN header is added at ingress point indicating membership
Fibre Channel Services for Blue VSAN
No special support required by end nodes
Fibre Channel Services for Red VSAN

22. EISLs and TE_Ports
The Virtual SANs feature introduces two new SAN elements:
The Trunking E_Port (TE_Port)
Negotiated between MDS 9000 switches – default
Carries tagged frames from multiple VSANs
Can be optionally disabled to yield E_Port
Only understood by Cisco MDS 9000 switches
Also has a native VSAN assignment (for E_Port)
Trunk all VSANs (1-4093) by default
Not to be confused with Brocade ISL aggregation (trunking)
The Enhanced ISL (EISL) link
The resultant link created by two connected TE_Ports
Superset of ISL functionality
Carry individual control protocol information per VSAN (eg. zoning updates)
Can be extended over distance (DWDM, FCIP, etc)
Cisco MDS 9500 Director with VSAN Service
Trunking E_Port (TE_Port)
Enhanced ISL (EISL) Trunk carries tagged traffic from multiple VSANs
Trunking E_Port (TE_Port)
Cisco MDS 9216 Fabric with VSAN Service
Trunking E_Port (TE_Port)
Enhanced ISL (EISL) Trunk carries tagged traffic from multiple VSANs
Notice: Blue VSAN doesn’t have to reside on switch for it to traverse switch

23. VSANs, EISLs, TE_ports and Port Channels
How do all these work together?
Hierarchical relationship –
Port Channels provide link aggregation to yield virtual ISL (E_Port)
Single-link ISL or Port Channel ISL can be configured to become EISL – (TE_Port)
VSANs can be selective grafted or pruned from EISL trunks
All member links of a Port Channel must have same configuration prior to creating channel (eg. TE_Port or E_Port, VSANs enabled, etc)
Port Channel technology provides high availability and fast recovery for VSAN trunk (EISL)
Multiple Port Channels yield multiple paths for custom traffic engineering
VSAN Metric
VSAN Metric
8 Gbps PortChannel
Trunking E_Port (TE_Port)
VSAN 20 VSAN 10 Backup
VSAN 10 Only
E_Port
Trunking E_Port (TE_Port)
E_Port
4 link (8 Gbps) PortChannel configured as EISL

24. VSANs – The Technical Details
How is an EISL link formed?
Negotiation details:
Two interconnected switch ports conduct an ELP (Exchange Link Parameters) exchange – forms two E_Ports and ISL link (standards based negotiation)
Two switches then conduct an ESC (Exchange Switch Capabilities) exchange – determines whether Cisco switch on other end or not capable of EISL (standards based negotiation)
If NO – then remain as ISL
If YES – then proceed to negotiate EISL/ISL
If Cisco switches, two switches then conduct an EPP (Exchange Peer Parameters) exchange – determines whether to stay as ISL, move to EISL (VSAN-enabled), or isolate in case of mismatched port VSANs E E
ELP Exchange
ISL Formed* E E
ELP Exchange (non-Cisco)
ISL Formed* E E E E
ESC Exchange
Two Cisco Switches E E E E
ESC Exchange (non-Cisco)
DONE E E E E
EPP Exchange
Normal ISL OR TE TE
EISL formed
These modes are negotiated based on the configuration of the switches and the parameters of the ports. Isolation can occur if VSANs are mismatched. OR
* Provided ELP parameters match such as timers and switches in interoperability mode if required
Isolated

25. VSANs – The Technical Details – cont.
EISL Header – What’s inside?
Several fields are added to the VSAN header that provide a variety of functions
Each frame on a VSAN trunk carries an extra 8 bytes of header which includes:
User priority – 3 bits – used for QoS functionality to designate priority of frame
VSAN ID – 12 bits – used to mark the frame as part of a particular VSAN – supports up to 4096 VSANs
MPLS flag – 1 bit – used to designate whether this frame is subject to Multi-Protocol Label Switching processing – future use
Time-to-live (TTL) – 8 bits – used to help avoid routing loops – standard part of an IP frame
Other misc. fields including version, frame type, and other reserved fields 4B
SOF (Start of Frame) 8B
EISL Header
FC Header
24B
Payload
0 ->
2112B 4B
CRC 4B
EOF (End of Frame)

26. VSAN Number Space VSAN numbering rules: VSAN 1 is the default VSAN
All ports are originally in VSAN1
VSAN 2 through 4093 can be assigned to ‘user’ VSANs – VSAN 0, 4094, 4095 are reserved
A maximum of 1000 VSANs can be created from the range of
Your mileage may vary based on other factors (eg. #zones, #zone sets, etc)
VSAN 4094 is a reserved ‘special’ VSAN
Called the ‘isolated VSAN’
Used to isolate ports who’s port-VSAN has been deleted
Not propagated across switches
Always present, can’t be deleted
Configured VSANs
VSAN 10
VSAN 20
VSAN 30
Cisco MDS 9000 Family with VSAN Service
Trunking E_Port (TE_Port)
Enhanced ISL (EISL) Trunk carries tagged traffic from multiple VSANs
VSAN 30 is not propagated across EISL due to nonexistence on remote switch
Trunking E_Port (TE_Port)
Port is in VSAN 4094 (Isolated VSAN)
VSAN 10
VSAN 20
Host is isolated from the fabric
VSAN 30
Configured VSANs

27. VSANs and Domain IDs
Recall: Each VSAN acts as a completely independent fabric
Each VSAN has its own principle switch and domain_ID allocation policy (static or dynamic)
Principle switches for different VSANs don’t have to reside on same physical switch
Each switch will have a separate domain_ID for each active VSAN
These domain_IDs can overlap between VSANs
All ports are originally in VSAN1
Each VSAN can have a separate FC_ID allocation policy (static or dynamic)
Domain 100 Domain 200
Domain 105 Domain 223
Domain 126
Domain 153 Domain 173
Domain 110 Domain 153
Domain 112 Domain 171
Domain 156 Domain 102
Domain 113 Domain 180
Domain 104 Domain 204
Domain 157
Domain 170 Domain 215
Domain 201 Domain 162
Each switch that has end ports in a particular VSAN will have a domain_ID assigned that that particular VSAN. Core switches that trunk these VSANs will also have assigned domain_IDs in these VSANs

28. VSANs and Non-Cisco Switches
The Virtual SANs feature involves a tagging mechanism which is not understood by 3rd party fabrics
Cisco MDS 9000 Family switches do support heterogeneous switch interoperability – non VSAN aware
Cisco “Interoperability Mode” (required) is configured per-VSAN – no loss of functionality in MDS 9000 switches
Cisco MDS 9000 switches negotiate an E_Port with non-Cisco switches
Cisco MDS 9000 E_Ports also have a port VSAN
Therefore, the entire non-Cisco switch, including all its ports, will reside in the port VSAN of the connecting E_Port
Enhanced ISL (EISL) trunks carrying numerous VSANs
Simple ISL links
E_Ports
Non-Cisco Fabric Switches
Each non-Cisco switch belongs to only one VSAN

29. Design Example Using VSANs
Challenge: Build a highly resilient, large port density and manageable SAN
Leverage VSANs in replacement for multiple separate physical fabrics
Use very high port-density platforms to minimize number of switches req’d
Use link bundling and multipathing within fabric to optimize resource usage
Use separate VSAN for backup traffic to isolate
Three platforms to optimize port density
Dept/App ‘A’
Dept/App ‘B’ FC FC FC FC FC FC FC FC FC FC FC FC FC FC
Cisco MDS 9216
Cisco MDS 9509
VSAN Trunk
Bundles
VSAN Trunks
Cisco MDS 9509
Shared Storage Pool
Backup VSAN FC FC FC FC FC FC
Main Data Center

30. Additional Information
Cisco Storage Networking
Cisco Metro Optical Product Information
Storage Network Industry Association (SNIA)
Internet Engineering Task Force – IP Storage
ANSI T11 – Fibre Channel