1. Investigating Serial Attached SCSI (SAS) over TCP (tSAS)
UCCS
Master’s Project
Deepti Reddy

2. Outline of the Talk Introduction to tSAS and related technology
Motivation and Challenges
tSAS Design
Performance Evaluation of a Mock Application
Lessons Learnt and Future Directions
Conclusion
11/3/2011
tSAS/sreddy

3. SCSI vs SAS SCSI (Small Computer System Interface)
The SCSI protocol is an application layer storage protocol.  
It's a standard for connecting peripherals to your computer via a standard hardware interface, which uses standard SCSI commands.
SAS (Serial Attached SCSI)
SAS is the successor of SCSI technology
Initially introduced at 3Gb/s in 2004
Currently, supports 6Gb/s
12Gbps by 2012
Significantly increased the available bandwidth offered by legacy SCSI storage systems.
Use of expanders increases the scalability over legacy SCSI
11/3/2011
tSAS/sreddy

4. SCSI vs SAS SCSI SAS Topology Parallel Bus Serial Bus Speed 3.2 Gbps
3 Gbps, 6Gbps, 12Gbps
Distance
1 to 12 meters
8 meters
Number of Targets
14 devices
128 expanders.
> 16,000 with cascaded expanders
Devices
SCSI only
SAS & SATA
Connectivity
Single Port
Dual Port
Drive Form Factor
3.5”
2.5”
Cost
Low
Medium
Need to have font size >=18 for easy viewing
11/3/2011
tSAS/sreddy

5. iSCSI
iSCSI protocol describes a means of transporting of the SCSI packets over TCP/IP
The iSCSI specification is revised and updated by the Internet Engineering task Force (IEFT).
Work by S. Aiken, D. Grunwald, A. Pleszkun and J. Willeke shows the performance of a commercial iSCSI software implementation compared quite favorably with fibrechannel [7]
Make diagram as big as possible .
11/3/2011
tSAS/sreddy
iSCSI protocol layering model

6. Typical SAS Topology X X X X SAS CONTROLLER (HOST BUS ADAPTER) ~8m
Driver
SAS CONTROLLER
(HOST BUS ADAPTER)
~8m X X X
- DISK DRIVES X
- EXPANDERS
11/3/2011
tSAS/sreddy

7. Motivation & Challenges
Overcome the distance and scalability limitations of traditional Serial Attached SCSI (SAS) by using tSAS.
Like iSCSI, tSAS takes advantage of existing internet infrastructure, internet management facilities as well as addresses distance limitations
Provide research results for future industry specification for tSAS and iSCSI.
11/3/2011
tSAS/sreddy

8. Related Work
Michael Ko’s patent on Serial Attached SCSI over Ethernet proposes a very similar solution to the tSAS solution provided in this project.
iSCSI specification (SCSI over TCP) itself is similar to a tSAS solution (SAS over TCP). The iSCSI solution can be heavily leveraged for a tSAS solution.
The Fibre Channel over TCP/IP specification also can be leveraged to design and implement a tSAS solution.
11/3/2011
tSAS/sreddy

9. tSAS Topology X - TCP link - SAS link X - DISK DRIVES - EXPANDERS tSAS
SCSI Driver
tSAS
HBA
SCSI Driver
tSAS CONTROLLER/HBA X
- TCP link
- DISK DRIVES
- SAS link X
- EXPANDERS
11/3/2011
tSAS/sreddy

10. tSAS Topology X - TCP link - SAS link X - DISK DRIVES - EXPANDERS tSAS
SCSI Driver
tSAS
HBA
SCSI Driver
tSAS CONTROLLER/HBA X
- TCP link
- DISK DRIVES
- SAS link X
- EXPANDERS
11/3/2011
tSAS/sreddy

11. Goals of the project Investigate how tSAS can be implemented
Design and develop a mock tSAS application
Write a client program using C/C++ that sends a set of tSAS commands to a server.
Write a server program using C/C++ that receives a set of tSAS commands, processes them and responds to the client with tSAS responses.
Evaluate the tSAS solution and compare with a mock iSCSI client/server application.
11/3/2011
tSAS/sreddy

12. Software and Hardware solutions for tSAS implementations
Software based tSAS solution
Cheaper than a hardware based solution
All tSAS processing is done by the processor and TCP/IP operations are also executed by the CPU
NIC is merely an interface to the network
TCP Offload Engine (TOE) solution
NICs with integrated TOEs have hardware built into the card that allows the TCP/IP processing to be done at the interface
Hardware based tSAS solution
In a hardware-based tSAS environment, the initiator and target machines contain a host bus adapter (HBA) that is responsible for both TCP/IP and tSAS processing
11/3/2011
tSAS/sreddy

13. tSAS Approach
The tSAS Request is initially sent by the tSAS Initiator to the tSAS Target over TCP.
The tSAS Target strips off the TCP header and sends the SAS frames using the SAS Initiator block on the tSAS Target to the SAS expander/drive.
The SAS expander/drive sends SAS frames to the tSAS Target.
Finally, the tSAS Target embeds the SAS frames received from the expander/drive over TCP and sends it to the tSAS Initiator.
11/3/2011
tSAS/sreddy

14. tSAS Message Format
Ethernet Header
IP Header
TCP
Header
eSAS Header
eSAS
Data
Ethernet Trailer
TCP Segment
IP Datagram
Ethernet Frame
The above shows how a legacy SAS header and data are embedded in an Ethernet frame.
iSCSI uses the same approach where the iSCSI header and data are encapsulated in an Ethernet frame.
11/3/2011
tSAS/sreddy

15. tSAS Approach SAS Expander tSAS Target tSAS Initiator
tSAS Request
Open Address Frame
Open Accept
SAS Request Frame
Data Frame
Data Frame .
Data Frame .
Data Frame
Response Frame
tSAS Response
The above diagram shows a typical tSAS (SSP Read ) Request & Response sequence diagram.
iSCSI uses a similar approach where the back-end between the iSCSI target and SCSI drives uses the legacy SCSI protocol.
11/3/2011
tSAS/sreddy

16. tSAS Approach Primitives Discovery
Primitives are special 8b/10b encoded characters that are used as frame delimiters, for out of band signaling, control sequencing in SAS
Most primitives can be conveniently ignored on the tSAS side except a few such as Broadcast primitives
SAS primitive can be encapsulated in an Ethernet frame
Discovery
Discovery in tSAS will be similar to SAS and will be accomplished by sending Serial management protocol (SMP) commands over TCP to the initiators and expanders downstream to learn the topology.
Will show a SAS Trace with READ commands to show the audience what primitives look like on the wire.
11/3/2011
tSAS/sreddy

17. tSAS Approach SMP Request & Response Sequence Diagram
SMP Initiator Port on Expander
SMP Target Port on Target Expander
Client
SMP Request
Open Address Frame
Open Accept
SMP Request
SMP Response
Close
SMP Response
Make the font bigger at least 18pt
SMP Request & Response Sequence Diagram
11/3/2011
tSAS/sreddy

18. Test benches used for performance calculations
tSAS and iSCSI Mock Application
The client application and server application run on two different Windows machines/servers connected using a NetGear ProSafe Gigabit Switch
Client makes Read/Write requests and the server processes and responds to requests made by the Client
Legacy SAS
A 6 Gbps SAS Host Bus Adapter in a PCIe slot of a Super Micro server
A 6 Gbps SAS target connected to the Host Bus Adapter
IOMeter in Windows and VDBench in RHEL used to generate Read/Write IOs and measure performance
A SAS analyzer placed between the HBA and the SAS Target
Legacy iSCSI
Two windows machines/servers were used
On one machine the StarWind iSCSi Initiator was used
On another machine, the KernSafe iSCSi target software was used to create a iSCSI target
IOMeter was used to send Read/Write requests from the iSCSI Initiator to the iSCSI Target and measure performance
Wireshark used to capture Network Traffic
11/3/2011
tSAS/sreddy

19. tSAS vs iSCSI Mock Application at 10 Mbps
11/3/2011
tSAS/sreddy

20. tSAS vs iSCSI Mock Application at 100 Mbps
11/3/2011
tSAS/sreddy

21. tSAS vs iSCSI Mock Application at 1 Gbps
11/3/2011
tSAS/sreddy

22. tSAS 512K Read at 10 Mbps, 100 Mbps and 1 Gbps
11/3/2011
tSAS/sreddy

23. Comparing tSAS mock application to legacy iSCSI and legacy SAS
tSAS results are extrapolated to 100 Gbps and so are the legacy iSCSI results to compare tSAS with legacy iSCSI
Assuming 100 Gbps connection rate performs times better than 1 Gbps connection rates based on the 1 Gbps and 10 Gbps server scalability paper by NetApp where it is concluded that 10 Gbps performs times better than 1 Gbps.
These comparison results can’t be considered to be a fair comparison as the tSAS mock application is not a full fledged tSAS implementation
tSAS results are compared to legacy SAS
Legacy SAS performance results between the HBA and the SAS drive observed using the SAS analyzer for various transfer sizes under 2 MB show that Read/Write completions are < 1 ms. tSAS performs much slower than this at 100 Gbps
tSAS is compared to legacy SAS without including the delay at drive
These comparison results can’t be considered to be a fair comparison as the legacy SAS application is HBA based while the tSAS application is S/W based and is not a full fledged application
11/3/2011
tSAS/sreddy

24. Comparing tSAS mock application to legacy iSCSI
11/3/2011
tSAS/sreddy

25. Comparing tSAS Mock application to legacy SAS X-Axis – Transfer Length (Bytes), Y-Axis – Time in Milliseconds
11/3/2011
tSAS/sreddy

26. Lessons Learnt
<describe what are difficulties you encountered in this project and how you solve them if you did>
11/3/2011
tSAS/sreddy

27. Future Work More data points
The tSAS mock application can be run using a faster switch with connection rates greater than 1 Gbps to get more data points
Piggybacking
Response frames are piggybacked with the last DATA frame or a DATA frame is piggy backed with a request frame
Jumbo frames
Can be used to increase the amount of DATA that is passed from the initiator and target per Ethernet packet improving the performance results.
The Storage Associations can be motivated with similar work to fuel the conception of a tSAS specification
tSAS code can be implemented in a SAS HBA and performance can be measured using this direct implementation
11/3/2011
tSAS/sreddy

28. Conclusion tSAS is a viable solution
tSAS will be faster than a similar iSCSI implementation
Frame sizes in tSAS smaller than in iSCSI
Back-end of tSAS will be legacy SAS
tSAS should be visibly faster at larger transfer sizes
tSAS hardware solution would be the fastest vs a software solution
tSAS would perform better at smaller transfer sizes
tSAS will overcome the distance limitation of legacy SAS
For the 2nd bullet item, are you listing all the reasons? Or they are independent facts?
11/3/2011
tSAS/sreddy

29. References
[1] T10/1760-D Information Technology – Serial Attached SCSI – 2 (SAS-2),
T10, 18 April 2009,
Available from
[2] Harry Mason, Serial attached SCSI Establishes its Position in the Enterprise, LSI Corporation, available from
[3] J Satran, K Meth, C. Sapuntzakis, M. Chadalapka, E. Zeidner,
RFC 3720 Internet Small Computer Systems Interface (iSCSI) Specification
IETF, April 2004,
available from
[4] Cai, Y.; Fang, L.; Ratemo, R.; Liu, J.; Gross, K.; Kozma, M.;
A test case for 3Gbps serial attached SCSI (SAS)
Test Conference, Proceedings. ITC IEEE International, February
2006,
available from
11/3/2011
tSAS/sreddy

30. References [6] Kalmath Meth, Julian Satran,
[6] Kalmath Meth, Julian Satran,
Design of the iSCSI Protocol, Mass Storage Systems and Technologies, (MSST 2003). Proceedings. 20th
IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies,
April 2003,
available from
[7] Stephen Aiken, Dirk Grunwald, Andrew R. Pleszkun, Jesse Willeke,
A Performance Analysis of the iSCSI Protocol
20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and
Technologies, 2003,
available from
[8] M. Rajagopal, E. Rodriguez, R. Weber,
RFC 3821 Fibre Channel over TCP/IP (FCIP) standard,
IETF, July 2004,
available from
[9] BLi Bigang, Shu Jiwu, Zheng Weimin,
SCSI Target Simulator Based on FC and IP Protocols in TH-MSNS*
Department of Computer Science and Technology, Tsinghua University, Beijing
China, 2005
11/3/2011
tSAS/sreddy

31. References
[10] S. Chaitanya, K. Butler, A. Sivasubramaniam, P. McDaniel, M. Vilayannur,
  Design, Implementation and Evaluation of Security in iSCSI-based Network
Storage Systems,
StorageSS '06 Proceedings of the second ACM workshop on Storage security
and survivability,
October 2006,
available from
[11] 1Gbps and 10Gbps Ethernet
Server Scalability, NetApp,
available fromhttp://partners.netapp.com/go/techontap/matl/downloads/redhat-
neterion_10g.pdf
[12] Michael A. Ko, LAYERING SERIAL ATTACHED SMALL COMPUTER SYSTEM
INTERFACE (SAS) vOVER ETHERNET, United States Patent Application
, 09/18/2008
available from
[13] SCSI Primary Commands - 4 (SPC-4), Revision 31, T10, 2011/06/13,
available from
11/3/2011
tSAS/sreddy