1. draft-ietf-ips-iser-00 Mike Ko November 8, 2004
iSER Draft Status
draft-ietf-ips-iser-00
Mike Ko
November 8, 2004

2. iSER Flow Control for Control-Type PDU
As currently defined, the iSER protocol does not provide additional flow control beyond that provided by the iSCSI layer on control-type PDUs
From section 8.1: “The iSER Layer SHOULD provision enough Untagged buffers for handling incoming RDMAP Send Message Types to prevent a buffer underrun condition”
iWARP Verbs mechanisms such as the Shared Receive Queue mechanism can be used to effectively address the Send Message flow control question
Most iSCSI PDUs are paced by the CmdSN window mechanism in iSCSI
Question: Should some form of send side flow control be established for iSCSI control-type PDUs not regulated by the CmdSN window?
11/8/2004
M. Ko

3. Flow Control Alternatives for Unexpected PDUs
Require the receiver to replenish buffers faster than the arrival rate of incoming PDUs including the unexpected ones
Imposes a constraint on the implementation that may not be achievable in all circumstances
Require the RNIC to do graceful handling on lack of receiver buffer situations
Mandates the implementation of an optional feature
Declare a limit on the number of unexpected PDUs that can be sent
If architected correctly, can also accommodate options 1, 2, and 5
11/8/2004
M. Ko

4. Flow Control Alternatives for Unexpected PDUs
Define an explicit credit based flow control mechanism
Basically a receive side flow control
Extra overhead even when options 1, 2, and 5 are used
Require the RNIC to use Shared RQ
Mandates the implementation of an optional feature
Drop the link if there is buffer overrun
Default solution if no other flow control alternatives
Drastic solution compared with other alternatives
11/8/2004
M. Ko

5. Declared Limit on Sender Side on the Number of Unexpected PDUs
First discussed on the IPS reflector in July/August of 2003 between Mike Ko, Caitlin Bestler, David Black, etc.
Idea revived and refined in October 2004
A new key, MaxOutstandingUnexpectedPDU, can be used by both the initiator and the target to declare the maximum number of outstanding “unexpected” control-type PDUs it can receive
Key has session wide scope
Default is none
“None” works for options 1, 2, and 5
If not “none”, is there a good default which is not implementation dependent?
11/8/2004
M. Ko

6. iSCSI Control-Type PDUs from the Initiator Regulated by CmdSN Window
If the Initiator sends this PDU
It must provision a buffer for this PDU from the Target
SCSI Command
SCSI Response or Reject before the task is active
Task Management Function Request
Task Management Function Response
Text Request
Text Response
Login Request
Login Response
NOP-out (request)
ITT /= 0xffffffff
TTT = 0xffffffff
NOP-in (response)
TTT =0xffffffff
11/8/2004
M. Ko

7. iSCSI Control-Type PDUs from Initiator with Known Buffering Requirement at Target
SCSI Data-out PDU
For unsolicited data, the iSER layer at the target can use FirstBurstLength to determine the amount of buffering required
Solicited data is handled using RDMA Read Response
Data cannot be “solicited” since an R2T is always transformed into an RDMA Read Request and is never sent as is by the target
Section to be updated to remove references on sending “solicited” SCSI Data-out PDUs
NOP-out PDU with ITT = 0xffffffff and TTT /= 0xffffffff (ping echo) which is sent as a response to a NOP-in PDU from the target
11/8/2004
M. Ko

8. iSCSI Control-Type PDUs from the Initiator Subject to Declared Limit
Any PDU with the immediate flag set
For the PDUs listed on slide 6, the PDU is outstanding until the target responds with the corresponding response PDU
For NOP-out PDU with ITT = TTT = 0xffffffff, the PDU is outstanding until the target responds with a control-type PDU with ExpCmdSN set to at least x + 1 where x is the CmdSN of the PDU with the immediate flag set
NOP-out PDU with ITT = 0xffffffff and TTT /= 0xffffffff (ping echo) is not subject to the declared limit since it is sent as a response to a NOP-in PDU from the target
SNACK PDU
Not needed in iSER
If sent, PDU is outstanding until the target responds with a SCSI Response PDU for the referenced command
11/8/2004
M. Ko

9. Potential Problem with Overuse of Unidirectional NOP-out
When the target sends a PDU with ExpCmdSN set to x+1, up to MaxOutstandingUnexpectedPDU unidirectional NOP-out PDUs with ITT = TTT = 0xffffffff with CmdSN = x could be in flight from the initiator
Upon receiving the PDU from the target, the initiator can send MaxOutstandingUnexpectedPDU unidirectional NOP-out PDUs with CmdSN = x+1
To guard against this pathetic case, the target needs to provision the number of buffers equal to twice MaxOutstandingUnexpectedPDU in this situation for the unexpected PDUs
The number of buffers for unexpected PDUs can be reduced to MaxOutstandingUnexpectedPDU when the initiator sends a non-immediate PDU with CmdSN = x+1
Recommend that we add a cautionary note instead against the overuse of the unidirectional NOP-out
11/8/2004
M. Ko

10. iSCSI Control-Type PDUs from the Target Subject to Declared Limit
The following PDUs are outstanding until the initiator sends a control-type PDU with ExpStatSN set to at least x + 1 where x is the StatSN of the “unexpected” PDU
Asynchronous Message PDU
Reject PDU sent after the task is active
NOP-in PDU initiated by the target with ITT = TTT = 0xffffffff
A NOP-in PDU sent as a ping request by the target with ITT = 0xffffffff and TTT /= 0xffffffff is outstanding until the initiator responds with a NOP-out PDU (ping echo)
11/8/2004
M. Ko

11. Potential Problem with Overuse of Unidirectional NOP-in
For a NOP-in PDU from the target with ITT = TTT = 0xffffffff, StatSN is not advanced
When the initiator sends a PDU with ExpStatSN set to x+1 where x is the StatSN of this NOP-in PDU, up to MaxOutstandingUnexpectedPDU unidirectional NOP-in PDUs with ITT = TTT = 0xffffffff with StatSN = x could be in flight from the target
Upon receiving the PDU from the initiator, the target can send MaxOutstandingUnexpectedPDU unidirectional NOP-in PDUs with CmdSN = x+1
To guard against this pathetic case, the initiator needs to provision the number of buffers equal to twice MaxOutstandingUnexpectedPDU in this situation for the unexpected PDUs
The number of buffers for unexpected PDUs can be reduced to MaxOutstandingUnexpectedPDU when the target sends a PDU with StatSN = x+1
Recommend that we add a cautionary note instead against the overuse of the unidirectional NOP-in
11/8/2004
M. Ko