1. Using SCST to develop SCSI Storage Systems on Linux Richard Sharpe

2. Agenda ● Intro to Richard Sharpe ● Intro to SCST ● Using SCST ● Obtaining and building SCST ● Writing Device Handlers ● Writing Target Drivers ● Final Thoughts

3. Intro to Richard Sharpe ● Australian ● Drove here from Bay Area ● Wrote a bunch of Ethereal Dissectors ● Wrote some code in Samba ● Currently work for Data Robotics ● Developed a VTL Prototype using STGT – A Scalar 24 with 44 tape slots ● Wrote and maintain scst_local

4. STGT-based VTL Prototype NetBacku p mtx NetVault Initiator Target GbE iSCSI STGT SSC LTO3 One file per tape SMC 44 Slots

5. VTL Design FC SCSI Initiator (qla2xxx) FC Driver (qla_isp) FC Driver (qla_isp) SCST Management Module SMC Device Handler SSC Device Handler Some FileSyste m

6. scst_local ● FC hardware is expensive (cards, switches) ● Developers need ways to test device handlers ● scst_local is a Linux SCSI LLD and SCST target device driver all in one ● Started writing it in Santa Clara ● Continued in Bangalore on a trip to India to vet a VTL Outsourcing company ● It first started working in Hyderabad. ● Used by companies like FusionIO

7. Linux SCSI Target Frameworks ● STGT, the SCST Target Framework – Some in-kernel pieces, CDB handling in userspace – Few Target Mode drivers for adapters – http://stgt.berlios.de/

8. Linux SCSI Target Frameworks ● SCST – Mostly in-kernel, CDB handling in userspace possible – A number of Target Mode drivers ● QLogic FC controllers (qla_isp) ● Marvell and LSI SAS controllers ● Infiniband controllers ● ISCSI ● scst_local – http://scst.sourceforge.net/

9. Linux SCSI Target Frameworks ● iSCSI Enterprise Target – http://iscsitarget.sourceforge.net/

10. Relevant Standards ● T10 www.t10.orgwww.t10.org ● http://www.t10.org/drafts.htm http://www.t10.org/drafts.htm ● SCSI Architecture Model 5 – http://www.t10.org/cgi-bin/ac.pl?t=f&f=sam5r01.pdf http://www.t10.org/cgi-bin/ac.pl?t=f&f=sam5r01.pdf ● SCSI Primary Commands 4 – http://www.t10.org/cgi-bin/ac.pl?t=f&f=spc4r20a.pdf http://www.t10.org/cgi-bin/ac.pl?t=f&f=spc4r20a.pdf ● SCSI Block Commands 3 – http://www.t10.org/cgi-bin/ac.pl?t=f&f=sbc3r19.pdf http://www.t10.org/cgi-bin/ac.pl?t=f&f=sbc3r19.pdf ● SCSI Stream Commands 4 ● SCSI Medium Changer Command Set 3

11. Intro to SCST ● Linux in-kernel SCSI Target Framework ● Developed by Vladislav Bolkhovitin and others ● Target mode drivers ● SCST core services ● A Variety of device handlers ● http://scst.sourceforge.net/ ● http://scst.sourceforge.net/scst_pg.html http://scst.sourceforge.net/scst_pg.html ● http://scst.sourceforge.net/qla2x00t-howto.html http://scst.sourceforge.net/qla2x00t-howto.html ● http://scst.sourceforge.net/users.html

12. SCST, cont ● FiberChannel target drivers ● SAS target drivers ● Low overhead, fine-grained locking ● Built-in device handlers ● User-developed device handlers ● Advanced per-initiator device visibility management ● Reserve/Release functionality ● Generation of necessary UNIT Attentions

13. SCST, cont ● Generation of RESETS ● REPORT LUNs ● Task Management ● Persistent reservations ● UNMAP

14. SCST IO modes ● Pass-through – One-to-many – Disks, tapes, processors, CDROMs, changers, etc ● FILEIO mode, eg mapping a LUN to a file – Takes advantage of the Linux page cache ● BLOCKIO mode – Direct block io ● User-space mode ● Performance (NULLIO) handlers

15. SCST Overview SCSI Initiator Target Driver SCST SCSI ML In kernel Dev Handler SCSI ULDs (sd, st) SCSI LLD 1:0:0:0 Userland Dev Handler File System FC/SAS/iSCSI/ InfiniBand /Parallel

16. Obtaining SCST svn co https://scst.svn.sourceforge.net/svnroot/scsthttps://scst.svn.sourceforge.net/svnroot/scst

17. Building SCST ● cd scst/trunk/scst ● make – Need kernel sources or kernel-devel, kernel-headers ● make install ● Make any drivers – cd scst/trunk/qla_isp – make install – cd../vmsas_tgt – make; make install

18. Loading SCST SCSI Target Side ● modprobe scst ● modprobe isp_mod ● modprobe isp_scst ● Modprobe mvsas_tgt

19. Loading SCST, cont ● If you loaded SCST Target driver before loading a device handler, you will have to force the SCSI host to rescan on the initiator – echo '- - -' > /sys/class/scsi_host/hostX/scan

20. Configuring SCST ● cat /proc/scsi_tgt/help # to get help ● Finding what handlers are loaded – cat /proc/scsi_tgt/sessions ● Load the vdisk handler – modprobe scst_vdisk ● Configure a device handler – echo “open vm_disk0 /path/to/file” ● Make the device visible – echo “add vm_disk0 0” > \ – /proc/scst_tgt/groups/Default/devices ● lsscsi on initiator

21. Tools ● lsscsi [root@localhost]# lsscsi [0:0:0:0] disk ATA ST9320320AS 0303 /dev/sda [4:0:0:0] disk Kingston DataTraveler 2.0 PMAP /dev/sdb [5:0:0:0] cd/dvd TSSTcorp CD/DVDW TS-L632D TO04 /dev/sr0 ● sg3_utils [root@localhost]# sg_vpd -i /dev/sda Device Identification VPD page: Addressed logical unit: designator type: vendor specific [0x0], code_set: ASCII 00 20 20 20 20 20 20 20 20 20 20 20 20 35 53 58 30 5SX0 10 47 54 34 35 GT45 designator type: T10 vendor identification, code_set: ASCII vendor id: ATA vendor specific: ST9320320AS 5SX0GT45

22. SAS Demo Environment SCSI Initiator SCSI Target SuperMicro AOC-SASLP-MV8 Marvell 88SE6480 SuperMicro AOC-SASLP-MV8 Marvell 88SE6480 SFF-8088 SFF-8087

23. SAS Demo ● ssh to demo system ● Load up vdisk handler on target ● Load SCST and mvsas_tgt ● Fdisk/mke2fs etc... ● Show device visibility ● etc

24. LUN Masking and Visability ● Target oriented – Define for each target the LUNs visible to all initiators – Use Default_TARGET_NAME as group name for access control – Default group as a backstop ● Initiator oriented – Define for each initiator (or group of initiators) a group that controls which LUNs can be accessed ● Can use both schemes – Initiator mode has higher priority

25. LUN Masking and Visibility,cont ● /proc/scsi_tgt/scsi_tgt – Used for status, creating LUNS, creating groups, adding LUNs to groups, and adding initiators to groups ● /proc/scsi_tgt/groups/ /names – Initiator names ● /proc/scsi_tgt/groups/ /devices – LUNs ● Changing to /sys/... ● Described in.../scst/trunk/scst/README

26. scstadmin tool ● There is a perl-based admin tool ● You should use the admin tool, not manually create groups and so forth

27. Device Handlers & Target Drivers ● SCSI Concepts ● SCST Concepts ● Walkthrough scst_local ● Walkthrough SCSI Virtual Tape handler

28. SCSI Concepts ● Initiator – A Linux Host, an Linux Low Level Driver – scst_local.ko, qla2xxx.ko,... ● Target – A SCSI device, like a disk, tape, printer... ● Logical Units – Sub devices within a SCSI Target – lsscsi – cat /proc/scsi/scsi – 1:0:0:0, 1:0:0:1... ● Command Descriptor Blocks – CDBs

29. SCSI Concepts ● SCSI Transports – Parallel transports – FiberChannel – TCP/IP (iSCSI) – Serial Attached SCSI (SAS)

30. SCSI Concepts, cont DMA_TO_DEVICE (WRITE_XX, MODE_SELECT,...) Initiato r Target CDB Data Result & Sense Data DMA_FROM_DEVICE (READ_XX, MODE_SENSE,...) Initiato r Target CDB Data Result & Sense Data

31. SCSI Concepts, cont DMA_NONE (TEST_UNIT_READY,...) Initiato r Target CDB Result & Sense Data DMA_BIDIRECTIONA L Initiato r Target CDB Data Result & Sense Data

32. SCSI Commands ● Test Unit Ready ● Inquiry – Vital Product Data (VPD) ● READ CAPACITY – Allows devices up to 2TiB (32-bit LBA space) ● READ CAPACITY 16 – Allows much larger devices (64-bit LBA space) ● READ_6, READ_10, READ_16 ● WRITE_6, WRITE_16, WRITE_16 ● etc

33. SCSI CDBs 76535210 OpCode = 0x28 READ_10 LUN 0 1DPOFUAReserved Rel Adr MSB LBA LSB 2-5 7-8 1 Bits→ ↓Bytes Reserved6 MSB Transfer Len (LBAs) LSB Control

34. SCSI CDBs 76535210 OpCode = 0x88 RDPROTECT 0 1DPOFUARsvd MSB LBA LSB 2-9 10-13 Bits→ ↓Bytes 15Control MSB Transfer Len (LBAs) LSB Group Number READ_16 Rsvd FUA NV 14 MM C4 Res Reserved

35. SCSI CDBs 76535210 OpCode = 0x2A WRITE_10 LUN 0 1DPOFUA Rel Adr MSB LBA LSB 2-5 7-8 1 Bits→ ↓Bytes Reserved6 MSB Transfer Len (LBAs) LSB Control EBPRsvd

36. SCSI CDBs 76535210 OpCode = 0x8A RDPROTECT 0 1DPOFUA Obsl t MSB LBA LSB 2-9 10-13 Bits→ ↓Bytes 15Control MSB Transfer Len (LBAs) LSB Group Number WRITE_16 Rsvd FUA NV 14 Reserved

37. SCST Concepts SCSI Command Handling TARGET DRIVER SCST DEV HANDLER scst_rx_cmd scst_cmd_init_don e scst_rx_data scst_tgt_cmd_done Translate LUN to device and preprocesscommand Time→ Alloc space rdy_to_xfer parse Exec cmd or send to SCSI ML READ WRITE NO DATA WRITE READ exec xmit_response cmd finishe d... dev_done send respons e on_free_cm d free the cmd...

38. SCST Concepts, cont Initialization scst_register_template TARGET DRIVER SCST scst_registerscst_register_session alloc_session_done... Time→ !ATOMIC

39. SCST Concepts, cont Initialization scst_register_template TARGET DRIVER SCST scst_registerscst_register_session alloc_session_done... Time→ ATOMIC SCST thread... Send all deferred cmds or fail them

40. scst_local walkthrough ● Bring up terminal screen ● Walk through important parts of scst_local

41. scst_local overview SCST SCSI ML In kernel Dev Handler SCSI ULDs (sd, st) scst_local SCSI LLD File System

42. scst_local overview, cont SCSI UL and Mid-Layer SCST scst_local queuecommand scst_tgt_cmd_done done routine

43. mvsas_tgt walkthrough ● Bring up terminal ● Walk through the mvsas_tgt driver

44. Device Handler Concepts ● scst_register_dev_driver ● scst_register_virtual_device

45. Virtual Tape handler walkthrough ● Bring up terminal ● Walk through the in-kernel Virtual SCSI tape device

46. Target Drivers - mvsas

47. Remaining Issues ● User Space device handlers – Implemented ● Persistent Reservations – Recent implementation provided ● Kernel patches – scst_exec_req_fifo – io_context – readahead – readahead_context ● Kernel compatibility – As far back as 2.6.18 – Up to 2.6.30