1. Beyond 100G OTN Interface Standardization OFC Th1I
Beyond 100G OTN Interface Standardization OFC Th1I.1 – March Steve Gorshe, Ph.D., FIEEE

2. Outline Introduction OTN evolution for rates beyond 100Gbit/s
Motivations and considerations
OTN protocol modifications for the new rates
New clients and client mappings
Including FlexE and 25GbE
Flexible OTN (FlexO) PHY (G.709.1)
Signal format
FlexO 100G interface (FOIC)
Reference material

3. Introduction

4. Converged transport over OTN
All client types carried over OTN
CBR and packet clients

5. Information flow for OTN
OTN TDM allows multiple client signals to share the same wavelength (OCh)
WDM allows multiple signals to share the same fiber, each using a separate wavelength

6. Information containment relationships (electrical portion)

7. Evolution of OTN for rates beyond 100Gbit/s (B100G)

8. Motivations and Considerations behind G.709 OTN B100G (2016)
The Shannon channel capacity limits are catching up to optical transport network capabilities.
The current standard 50 GHz channel spacing used for dense wavelength division multiplexing (DWDM) imposes limits on transporting signals over reasonable distances when they have rates much over 100Gbit/s.
200Gbit/s is the practical limit per wavelength for distances of interest in telecom networks
The old paradigm of adding new discrete rates for OTN had largely reached its practical limits, making a modular rate and frame structure approach more attractive.
There should not be a new switching layer in OTN associated with B100G.

9. Motivations and Considerations behind G
Motivations and Considerations behind G.709 OTN B100G (2016) (continued)
The higher bit rates and increased use of multi-lane interfaces poses additional considerations regarding Forward Error Correction (FEC) and performance monitoring.
Different B100G interface types have different FEC performance capability requirements. Consequently, while the legacy OTUk frame format had fixed dedicated FEC overhead, it was more appropriate to specify the FEC on a per-interface basis and not make the FEC overhead in integral part of the frame structure.
The distinction between Inter-Domain interfaces (IrDI) between carriers or carrier domains, and Intra-Domain interfaces (IaDI) used between equipment within a carrier network domain, was recognized as artificial at this point, leading to that terminology being removed from G.709

10. Motivations and Considerations behind G
Motivations and Considerations behind G.709 OTN B100G (2016) (continued)
Continuing to use 1.25Gbit/s Tributary Slot (TS) sizes in OTN would be impractical for B100G rates, so a larger Tributary Slot size was desirable.
The high data rates and the introduction of new data client signals such as the OIF’s Flexible Ethernet (FlexE) have made the legacy byte-oriented mapping approach for data clients impractical. This motivated the desire for a wide-word type of mapping.
In order to optimize the use of each wavelength, including for transmission reach, there was a desire to transmit the OTN signals at the rate required for the client payload being carried rather than at the full discrete rate of the OTUCk signal.

11. Motivations and Considerations behind G
Motivations and Considerations behind G.709 OTN B100G (2016) (continued)
B100G interfaces should re-use as much IP from the 100Gbit/s OTN interfaces as practical.
The IEEE 802.3bs Task Force working on 400Gbit/s Ethernet was examining several new approaches that were different than its previous interfaces. The new OTN format needed to not only carry 400GbE, but also re-use its technology and PHY components whenever possible in order to benefit from the Ethernet component cost curves.

12. Foundational Agreements
The signal format is a modular concatenation of n 100Gbit/s base frames
The base signal format reuses the current OTN frame and overhead formats
Maximize IP re-use, but with minor changes as needed
Hence the naming convention OTUCn/OTUCn/OPUCn, where “C” corresponds to the Roman numeral for 100

13. Foundational Agreements (continued)
The ODUCn is a point-to-point signal that is not switched in the electrical domain
This would appear to imply that TCM (Tandem Connection Monitoring) is not required. However, the TCM overhead was preserved in order to allow for sectionalized monitoring of spans between 3R repeaters, and/or optical amplifiers
All the components of the OTUCn interface signal go through the same fiber and optical switches (i.e., the same Optical Multiplex Section trails) such that the OTUCn signal can be managed as a single entity.
Consequently, very limited deskew is required if multiple wavelengths are used

14. OTN B100G Frame
The interleaving of OTUC1 slices to form the OTUCn is not specified in G.709, since it is interface-dependent.

15. OTN B100G Frame Details – OTUCn Overhead
BEI = Backward Error Indication
BDI = Backward Defect Indication
IAE = Incoming Alignment Error
TTI = Trail Trace Identifier
MFAS = Multiframe Alignment Signal
GCC = Generic (OAM) Communications Channel

16. OTN B100G Frame Details – ODUCn Overhead
DM = Delay Measurement

17. Rate Considerations
The rate of the base OTUC signal was chosen in order to meet the following criteria:
An OPUC1 must be capable of carrying an ODU4 client
An OPUC4 must be capable of carrying a 400GbE client.
The resulting signal rate should be reasonable efficient within the constraints of the first two criteria.
For example, when the signal is divided down to the nominally 25Gbit/s rate for electrical interface lanes, the resulting electrical lane rate should be compatible with the OIF CEI-28G specification.

18. OTN B100G Rates 1.163 μs OTUCn/ODUCn signal rate
OPUCn payload area rate
OTUCn/OPUCn frame period
n × (239/226) × Gbit/s
= n × Gbit/s
n × (238/226) × Gbit/s
= n × Gbit/s
1.163 μs
Note: All rates are ±20 ppm.

19. Foundational Agreements (continued)
The OPUCn is a single contiguous payload area rather than n separate ones
As a compromise between complexity and mapping efficiency, a 5Gbit/s Tributary Slot (TS) was chosen, with a maximum of 10n tributaries in the OPUCn
Allows efficient mapping and placement of 25GbE and 16GFC clients
Most OPUCn clients will be >10Gbit/s
No client signals are directly mapped into the OPUCn.
They must first be mapped into an ODUk (including ODUflex), which is then mapped or multiplexed into the OPUCn.

20. OTUCn Multiplexing Hierarchy
All OTUCn clients are first mapped into their own ODUk before multiplexing into the OPUCn
2-stage multiplexing allows clients to be multiplexed into a legacy OPUk first

21. OPUC Payload Area Structure
Each TS occupies 16 consecutive bytes per appearance in the OPUC
i.e., the TS uses 16-byte (128-bit) words

22. OPUCn Tributary Slot Order
The TS numbering TSa.b goes by OPUC slice (“a”, a = 1 to n) and then by the order of appearance of the sequential groups of 16-byte blocks with the OPUC slice (“b”, b = 1 to 20)

23. Example: Mapping a Client using 3 TS into an OPUC2
Using TS2.3, 1.4 and 1.15
Different colors are used here for each of the 3 x 16-byte words shown for that client.
The sequence number of the 16-byte segment of the 3 x 16-byte word is shown in italics.

24. OTN B100G Frame Details – OPUCn Overhead
GMP = Generic Mapping Procedure
PT = Payload Type

25. GMP Overhead for OPUCn
Since each TS has a maximum of 952 words per OPUCn multiframe, it uses a 10-bit count field (C1-C10)
GMP for legacy OTN uses a 14-bit count field
D1-D18 provide remainder (phase) information for finer resolution

26. GMP Count Inversion Patterns for Increment or DecrementII DI Δ U I 1 +1 –1 +2 –2
Binary value
>±2
NOTE – I indicates inverted Ci bit – U indicates unchanged Ci bit
Inverting the Ci count field bits allows immediate, robust signaling of the count incrementing or decrementing by ±1 or ±2
The combination of the using the same inversion pattern in both JC1 and JC2, with the CRC in JC3, guarantees correct operation with any single byte burst error corruption.

27. Finer Grain GMP Resolution
The number of bits in the finer-grained phase field was increased to allow the same level of resolution as legacy OTN with the much higher B100G signal rates
Fine grain phase (D1-D18) is protected by a CRC and filtering
Byte granularity example:
In each multiframe, the mapper receives on average K words of data plus J additional bytes (i.e., K×M + J bytes)
The Cm (JC1-JC3) tells how may data words are being transmitted
The ΣCnD indicates the number of bytes (<M) remaining in the transmitter’s virtual FIFO that were not sent
Since ΣCnD represents phase information, the receiver PLL can easily filter out the effect from receiving an errored ΣCnD value.

28. ODUflex(IMP) – New for OTN B100G
IMP = Idle Mapping Procedure
Motivation
GFP for packet mapping is byte-oriented, and not as convenient with wide data paths in high-speed devices
A wide-word (64-bit) version of GFP was considered
Having all packet data clients go through the Ethernet data path was less complex than having a separate or subsequent GFP data path
It was possible to decrease some complexity by reducing the degree of packet-aware processing
All Ethernet and most other data client streams with rates ≥10Gbit/s use 64B/66B block coding on the medium independent interface (MII)
Creates a common basis for handling high-speed data clients

29. ODUflex Concept – Review
ODUflex introduced in 2nd Generation OTN
Allow complete flexibility to define mappings for any new client signal into OTN
Generates an ODU frame to carry the client signal, and the ODU is then multiplexed into Tributary Slot(s) of a higher rate OPU
Three types of ODUflex
ODUflex(CBR)
Simply (239/238)×(client rate) client wrapper mapping
ODUflex(GFP)
ODUflex generated with local source clock and OPUflex filled with GFP data and Idle frames
Allows PW or VLAN switching within the OTN domain via ODU switching
New ODUflex(IMP) added for B100G
Replaces ODUflex(GFP) for packet clients mapped into OPUCn

30. ODUflex(IMP) Concept and Approach
ODUflex(IMP) carries a stream of 64B/66B characters rather than packets like ODUflex(GFP)
The character stream aspect is somewhat more like ODUflex(CBR)
Ethernet performs rate adjustment by inserting or removing 64B/66B Idle characters between packets (i.e., adjusting the size of the inter-packet gap)
The rate adjustment between the client 64B/66B stream and the chosen OPUflex rate is done similarly by inserting 64B/66B Idle characters
The rate chosen for the ODUflex is enough higher than the maximum client rate that only Idle insertion is required at the OTN mapper
Hence the name IMP = Idle-insertion Mapping Procedure

31. ODUflex(IMP) Concept and Approach (continued)
The ODUflex(IMP) mapper and demapper are aware of the packet boundaries (i.e., the IPG), but don’t process the packets
Non-Ethernet clients (e.g., MPLS) are first encapsulated into Ethernet streams and are then mapped the same way.

32. FlexE Client Signal
In parallel to the ITU-T OTUCn standardization, OIF defined “Flexible Ethernet” (FlexE) in order to decouple the Ethernet MAC and Physical Medium Dependent (PMD) sublayers, especially in terms of rates.
Allows multiple MAC flows that are each less than the PMD rate to share a PMD.
Also allows MAC flows that are each more than the PMD rate, to share a set of PMDs.
Primary motivation was efficient data center interconnections
FlexE defines a CBR signal that is constructed as a FlexE Group carrying one or more FlexE client signals.
The FlexE Group consists of between 1 and GBASE-R Ethernet PHYs, all sharing a common PHY clock source.

33. FlexE Client Signal (continued)
The FlexE signal carried on each PHY consists of a round-robin repeating set of “calendar slots” for 64B/66B characters.
The frame format for each PHY consists of a slot carrying the FlexE signal overhead followed by 1023 sets of 20 “Calendar” slots for carrying FlexE client data.
The calendar associated with each PHY is called a sub-calendar of the overall calendar for the FlexE Group.
The characters in the overhead slot provide for frame alignment on each PHY and alignment across all the PHYs, in addition to carrying other required overhead for the FlexE signal.
The nominal bandwidth of each calendar slot is: (100 Gbit/s/PHY) / (20 calendar slots/PHY) = 5 Gbit/s.

34. FlexE Client Signal (continued)
Ethernet inter-frame Idle insertion/deletion is used to adapt the FlexE client rate to the exact rate of the calendar slot set.
A FlexE client is a stream of 64B/66B characters associated with an Ethernet MAC packet flow, and occupies one or more of the repeating calendar slots.
Sub-rate (i.e., <100 Gbit/s) clients can thus be time division multiplexed onto the same PHY by assigning them the set of calendar slots needed for that client’s bandwidth.
Note that the calendar slots for a sub-rate client can be spread across multiple PHYs.
Clients with bandwidth greater than an individual PHY are accommodated by bonding multiple PHYs to carry the client (i.e., spreading the clients calendar slots across multiple PHYs as needed).

35. FlexE Client Signal (continued)
Calendar slots that will not be used can be marked as Unavailable and filled with Ethernet Error control blocks.
A PHY can discard the Unavailable calendar slots in order to use a lower bit rate (e.g., in order to meet an optical reach objective or reduce the power for that PHY)
Unavailable calendar slots are always located at the end of the sub- calendars

36. FlexE Data Flow (Transmit Direction)

37. FlexE Client Signal Mapping
There are 3 options for mapping FlexE into an OPUCn:
Terminate the FlexE and carrying the FlexE clients as Ethernet packet clients over OTN.
I.e., carry the FlexE client’s packets rather than the FlexE signal
Simply treat each FlexE Group PHY as a 100GBASE-R signal and transport it accordingly.
Referred to as “FlexE Unaware” mapping
Exploit calendar fill information within the FlexE stream in order to allow carrying a lower rate signal over OTN.
Referred to as “FlexE Aware” mapping

38. FlexE Client Signal Mapping (continued)
The FlexE Aware mapping discards the Unavailable FlexE calendar slots in one or more PHYs in order to construct a lower rate (“crunched”) CBR client signal
The resulting signal is mapped into an OPUflex using the special bit-synchronous mode of GMP (BGMP)
BGMP generates the Cm values deterministically rather than deriving them from the input client rate and buffer fill

39. 2-bit Aligned BMP Mapping
This mapping is used for carrying other CBR clients that are 64B/66B streams
25GbE was the first client defined to use this mapping
Conceptually it is the same as ordinary BMP, except that a 64B/66B block will always start on an odd- numbered bit within a payload byte

40. Sub-rate OTUCn (OTUCn-M)
Some carrier applications need optimization for power and/or transmission distance.
E.g., interconnections between two routers where the packet flow peak rate is less than the N×OTUC rate, or the interconnections between two OTN crossconnects where the required capacity is less than the N×OTUC rate.
G.709 includes the option of transmitting a signal that has the full set of OTUCn/ODUCn overhead, but has an OPUCn consisting of only the active Tributary Slots.
Specifically, an OTUCn-M signal consists of n copies of the OTUC, ODUC and OPUC overhead, and M of the 5Gbit/s TS.
For vendor-specific implementations. Hence, the frame format details are not specified.
Conceptually somewhat similar to the FlexE crunching of unavailable calendar slots

41. FlexO

42. Motivations Behind FlexO
The FlexO approach was defined by the ITU-T in order to provide a flexible, modular PHY mechanism to support different interface rates for B100G signals.
FlexO has conceptual similarities to the OIF FlexE, which inspired FlexO.
Like FlexE, FlexO is a modular interface consisting of a set of 100Gbit/s optical PHY streams.
Allows using any value of “n” for an OTUCn interface rather than defining only certain discrete values of n (e.g., just n = 4 and 10).

43. Motivations Behind FlexO (continued)
FlexO allows using 100GbE/OTU4 optical modules for the individual FlexO PHYs, thus benefitting from the lower cost of these optical modules.
As future higher rate Ethernet modules become available (e.g., 200Gbit/s or 400Gbit/s PHYs), FlexO can be extended to also make use of them.
FlexO makes partial reuse of the lane architecture and FEC structure from 100GbE and 400GbE Ethernet in order to leverage Ethernet IP.

44. Additional Concepts Behind FlexO
For FlexO, a set of n 100Gbit/s PHYs are bonded together to carry an OTUCn, with each 100Gbit/s PHY carrying an OTUC slice.
If a set of m 100Gbit/s PHYs are available, a subset of n PHYs can be chosen to carry an OTUCn (n<m).
For example, this would allow choosing the subset of PHYs with the best optical channel characteristics, or carrying multiple OTUCn signals over a set of PHYs.
The Ethernet reuse includes:
“KP4” RS(544,514,10) FEC,
400GbE Lane Alignment Markers for framing,
400GbE method for distributing the data across logical lanes

45. FlexO Signal and Frame Format
The signal is a constant-rate stream of RS(544,514) codewords containing the OTUCn data as its payload
While the ODUC rate is higher than ODU4, the RS(544,514) overhead is less than GFEC
The resulting bit rate is <5ppm of OTU4 rate
Allows direct reuse of IEEE KP4 IP/engine
128 FEC codewords / FlexO frame, and 8 frames / multiframe
Each frame begins with Lane Alignment Markers (AM) and FlexO overhead
One 120-bit AM per 25Gbit/s logical lane for that interface
320 bits reserved for overhead
Fixed stuff inserted to achieve a bit rate near the OTU4 rate
OTUCn data begins on 128-bit block boundary
Boundary slides between codewords, but integer number per frame

46. FlexO Frame and Multiframe Format Illustration

47. FlexO Overhead Overhead associated with managing the interface
Information for the receiver to properly reassemble the OTUCn
Group and PHY IDs, and map of which PHYs are used by that group
FlexO Communications Channel (FCC) for PHY configuration negotiation
STAT indicates remote PHY failure

48. FlexO Overhead (continued)
OSMC = OTN Synchronization Message Channel
IEEE 1588 PTP messages for carrying frequency and time-of-day information to synchronize OTN nodes
Located in FlexO overhead rather than OTUCn since the FlexO is the PHY layer
Carried in only the first member of the FlexO group

49. FlexO Transmit and Receive Process Data Flows

50. FlexO Interface (FOIC)
The data is round-robin striped onto the 25Gbit/s logical lanes on a 10-bit (i.e., RS10 symbol) basis
Identical to the format used by IEEE with the KP4 FEC
The FlexO frame begins on the first lane
The AMs are inserted into the beginning of the FlexO frame such that the lane distribution will place the correct AM sequence onto each lane.
Hence the name “lane” alignment marker
Currently only 25Gbit/s electrical lanes are used, creating a 1-to-1 logical-to-electrical PHY mapping.
Future 50Gbit/s electrical lanes would carry the two AM sets associated with the two logical lanes carried over the electrical PHY lane

51. FlexO Lane AMs, and Mapping/Striping
Field
CM0
CM1
CM2
UP0
CM3
CM4
CM5
UP1
UM0
UM1
UM2
UP2
UM3
UM4
UM5
Eth AM0 9a 4a 26 b6 65 b5 d9 01 71 f3 fe 8e 0c
FlexO AM0 59 52 64 6d a6 ad 9b 80 cf 7f 30
Eth AM1 04 67 5a de 7e 98 a5 21 81
FlexO AM1 20 e6 7b 19 84
Eth AM2 46 3e 56 c1 a9
FlexO AM2 62 7c 6a 83 95
Eth AM3 86 d0 79 2f
FlexO AM3 61 0b 9e f4
AM bits
Lane bit symbol of AM0
Lane bit symbol of AM1
Lane bit symbol of AM2
Lane bit symbol of AM3
1 – 40
41 – 80
81 – 120
121 – 160 .
441 – 480
NOTE – Transmission order of each 10-bit word is left-to-right (MSB first). The transmission order within the FlexO frame is left-to-right across the row, and down the table. The transmission order for each lane is per-word and down the table.

52. OTUCn with FlexO Example

53. © 2016 Microsemi Corporation. CONFIDENTIAL
Q&A
Questions?
© 2016 Microsemi Corporation. CONFIDENTIAL

54. References and For Further Reading

55. Important Related Standards
ITU-T Recommendation G.709 (2016), Interfaces for the Optical Transport Network (OTN)
ITU-T Recommendation G.798 (2010), Characteristics of optical network hierarchy equipment functional blocks
ITU-T Recommendation G (2016) Flexible OTN short- reach interface
IEEE 802.3:2015, IEEE Standard for Ethernet
ITU-T Supplement 58 (2016) Optical transport network (OTN) module frame interfaces (MFIs)
ITU-T Recommendation G.8251 (2010), The Control of Jitter and Wander within the Optical Transport Network
ITU-T Recommendation G (2012) Spectral grids for WDM applications: DWDM frequency grid

56. White Papers
ESC , “A Tutorial on ITU-T G.709 Optical Transport Networks (OTN),” Microsemi white paper by Steve Gorshe
Microsemi white papers are available at:
OIF FlexE Application Note (

57. Background Slides

58. Optical Transport Platform (OTP) concept
Amp
Add Side
Coupler
WSS
Mesh Out
Mesh In
Mux
Drop Side
TDM
Packet
PoS
Hybrid Fabric
OC-192
GbE
OC-12/48
VCAT
10 GbE
I/O
East Direction
West Direction
OC-192 
10 GbE 
Mixture of optical and electronic crossconnect within the same equipment
Hybrid TDM/packet electronic fabric
Also referred to as a “Packet-OTP” (P-OTP)
Figure from the Verizon Interoperability Forum

59. Tandem Connection Monitoring (TCM) Illustration
Support s up to 6 layers of TCM

60. Glossary Term Definition AM Alignment Marker (Ethernet and FlexO) CBR
Constant Bit Rate
CRC-n
n-bit Cyclic Redundancy Check error detection code
DWDM
Dense Wavelength Division Multiplexing
FCC
FlexO Communications Channel
FEC
Forward Error Correction
FlexE
Flexible Ethernet (from OIF)
FlexO
Flexible OTN (G.709.1)
FOIC
FlexO Interface
GFP
Generic Framing Procedure (ITU-T Rec. G.7041)
GID
FlexO Group ID
GMP
Generic Mapping Procedure
IaDI
Intra-Domain Interface (obsolete term)
IMP
Idle Mapping Procedure
IrDI
Inter-Domain Interface (obsolete term) JC
Justification Control
KP4
Reed-Solomon RS(544,514,10) FEC from IEEE 802.3
MAP
FlexO overhead field for the OTUC to FlexO PHY mapping
MFAS
MultiFrame Alignment Signal
MSI
Multiplex Structure Identifier
OCh
Optical channel with full functionality

61. Glossary (continued) Term Definition ODUC
100Gbit/s element (slice) of an ODUCn
ODUCn
n × 100Gbit/s Optical Channel Data Unit
ODUflex(CBR)
Flexible rate ODU for carrying CBR client signals
ODUflex(GFP)
Flexible rate ODU for carrying packet client signals that use a GFP-F mapping into the OPUflex
ODUflex(IMP)
Flexible rate ODU for carrying packet client signals with Ethernet Idle characters used for rate adaptation when mapping into the OPUflex OH
Overhead
OPUC
100Gbit/s element (slice) of an OPUCn
OPUCn
n × 100Gbit/s Optical Channel Payload Unit
OSMC
OTN Synchronization Message Channel
OTUC
100Gbit/s element (slice) of an OTUCn
OTUCn
n × 100Gbit/s Optical Channel Transport Unit
PID
FlexO PHY ID
PMD
(Ethernet) Physical Medium Dependent sub-layer
PSI
Payload Structure Identifier PT
Payload Type
ROADM
Reconfigurable Optical Add / Drop Multiplexer
TCM
Tandem Connection Monitoring
WDM
Wavelength Division Multiplexing

62. Biography
Steven Scott Gorshe received his B.S.E.E. from the University of Idaho (1979) and M.S.E.E. (1982) and Ph.D. (2002) from Oregon State University. His work includes a variety of hardware design, system architecture, and applied research for GTE, NEC America, PMC- Sierra, and Microsemi where he is a Distinguished Engineer. He is ITU-T Q11/15 Associate Rapporteur. His standards activity there and in other bodies includes >400 contributions, and multiple technical editorships. He is an IEEE Fellow, has 38 patents granted/pending, is co-author of two books, three chapters and many papers. His IEEE ComSoc activities include Communications Magazine EiC and Board-of-Governors MAL.