0. OpenSSL on QorIQ Communications Platform and C29x Crypto Coprocessor Family
AMF-SNT-T1047
Jim Bridgwater | Product Line Manager
MAR.2015
OpenSSL is a very popular crypto toolkit and protocol stack, but it isn’t very good at exploiting modern crypto acceleration, like that found in Freescale’s QorIQ communications platform and C29x crypto coprocessor family. This session describes Freescale’s SEC driver integration with standard OpenSSL and optimizations made for RSA and record layer offload. Included is a performance demo of OpenSSL speed test with and without optimizations and HTTPS connection rate on T4240+C29x illustrating splitting offloads between the T4240’s local security engine and an external C29x.

1. Agenda OpenSSL Overview QorIQ Processors with Crypto Accelerator
Enable OpenSSL with QorIQ SEC Engine
Performance Benefits with SEC Engine
Summary

2. Secure Socket Layer (SSL) Overview
The Secure Socket Layer (SSL) protocol is the most widely deployed application protocol to protect data during transmission by:
Encrypting the data using popular cipher algorithms such as AES and 3DES
Message authentication using popular hash/digest algorithms such as SHA1 and MD5
SSL is widely used in application web servers (HTTP) and other applications such as cloud storage, webmail (POP3 and IMAP), Proxy servers and many more, where protection of data in transit is essential for data integrity
There are various version of SSL protocol such as TLSv1, SSLv3 and SSLv2 that are commonly used
Other newer versions are available, such as TLSv2, TLSv3 and DTLS (Datagram TLS)
Of all the SSL protocol versions, TLSv1 and SSLv3 are in common use

3. SSL Handshakes Between Client and Server
1. Client Hello
Public
Private
2. Server Hello
Public
Private
3. Certificate (Optional)
4. Certificate Request (Optional)
5.Server key exchange (Optional)
6. Server Hello Done
7. Certificate (Optional)
Server
Client
8. Client key exchange
9. Certificate Verify (Optional)
10. Change Cipher Spec
ClientHello: The client sends the information such as SSL protocol, cipher suites information such crypto algos supported etc to server.
ServerHello: Server communicates back the selected cipher suite to client.
Certificate: This message is optional and is used when server authentication is required. Server sends its certificate to client which contains server's public key.
Certificate Request: This message is sent only if the server requires the client to authenticate itself.
Server Key Exchange: This message is sent if the certificate, which contains the server's public key, is not sufficient for key exchange.
ServerHelloDone: This message informs the client that the server finished the initial negotiation process.
Certificate: This message is sent only if the server requested the client to authenticate itself.
Client Key Exchange: The client generates a secret key to be shared between the client and server. If the RSA encryption algorithm is used, the client encrypts the key using the server's public key and sends it to the server. The server uses its private or secret key to decrypt the message and retrieves the shared secret key. Now, client and server share a secret key that has been distributed securely.
Certificate Verify: If the server requested to authenticate the client, this message allows the server to complete the authentication process.
Change Cipher Spec: The client asks the server to change to encrypted mode.
Finished: The client tells the server it is ready for secure communication.
Change Cipher Spec: The server asks the client to change to encrypted mode.
Finished: The server tells the client it is ready for secure communication. This marks the end of the SSL handshake.
Encrypted Data: The client and server can now start exchanging encrypted messages over a secure communication channel.
11. Finished
12. Change Cipher Spec
13. Finished
Encrypted Data

4. OpenSSL Overview
OpenSSL is the de-facto high-level open standard library for Linux security user space applications
Sample applications: Apache, PGP, SSL-based apps
Documents are available from 
OpenSSL allows the selection of an “ENGINE' to replace the default implementation during the operation of the command.
OpenSSL can easily extended to use QorIQ SEC accelerator
The ENGINE interface provides callback hooks to integrate with hardware accelerators with the crypto library
The custom callback hooks implement the glue logic (code) to interface with various hardware accelerators
The OpenSSL library has several sub-components:
SSL protocol library: libssl
Crypto library (Symmetric and Asymmetric Crypto support):  libcrypto
Digest Support
Certificate Management: CA.pl

5. OpenSSL Layered Architecture with Linux Kernel
Engine

6. Freescale Solution for OpenSSL Hardware Offloading
Freescale Layer solution for OpenSSL hardware offloading:
User Space
OpenSSL- implements the SSL protocol
Cryptodev-engine- implements the OpenSSL ENGINE interface; talks to cryptodev-linux (/dev/crypto) via ioctls, offloading cryptographic operations in kernel
Kernel Space
Cryptodev-linux- Linux module that translates ioctl requests from cryptodev-engine into calls to Linux Crypto API
Linux Crypto API- Linux kernel crypto abstraction layer
CAAM driver- Linux device driver for the QorIQ crypto engine
crypto_register_alg() to register CAAM driver's algorithm interface function pointers to the crypto layer.
The following are offloaded in hardware in current SDK:
Protocols: TLS v1.0
Cipher modes:
Two passes (two ioctls - one for encryption, the other for authentication): all other combinations of AES with SHA, all combinations of DES and 3DES with SHA
Single pass (a single ioctl for both encryption and authentication): AES128-SHA
Reduce the amount of memory copies and API calls for authentication, encryption, and protocol specific operations
TLS code block algorithms there is a need to perform padding in order to prepare data for encryption, padding can be done with a SEC combo descriptor

7. QorIQ Processors with Crypto Accelerator

8. QorIQ Processors with Crypto Accelerator
Frees CPU from draining repetitive RSA, VPN and HTTPs traffic
Supports protocol processing for the following:
IPSec
802.1ae (MACSEC)
SSL/TLS
3GPP RLC
LTE PDCP
SRTP
802.11i (WiFi)
802.16e (WiMax)
Data Encryption Standard Accelerators (DESA)
DES, 3DES (2K, 3K)
ECB, CBC, OFB modes
AES Accelerators (AESA)
Key lengths of 128-, 192-, and 256-bit
ECB, CBC, CTR, CCM, GCM, CMAC,
OFB, CFB, and XTS
Message Digest Hardware Accelerators (MDHA)
SHA-1, SHA-2 256, 384, 512-bit digests
MD5 128-bit digest
HMAC with all algorithms
Random Number Generator

9. C29x Public Key Accelerator
Coherent System Bus
JTAG
Real Time Debug
32-bit
DDR3/3L
Memory Controller
512KB
Platform Cache
4 Lane 5GHz SERDES
Power Architecture™ e500-v2 Core
32KB
D-Cache
I-Cache
PCIe
DMA
veTSEC
Security Fuse Processor
Security Monitor
IFC
Power Management
SD/MMC+
2x DUART
2x I2C
SPI, GPIO
SEC 0
Platform SRAM
SEC 1
SEC 2
Processor
1x e500 v2, 32b, up to 1.2GHz
Memory SubSystem
1MB Frontside L2 cache/SRAM w/ECC
32-bit DDR3/3L, 1200MHz data rate w/ECC
Up to 64GB addressability (36-bit physical addressing)
ECM Coherent System Bus High Speed Serial IO
1 PCIe 2.0 Controller (5GHz)
x1, x2, x4 options
Network IO
2 x 10/100/1000 Ethernet Controllers
RGMII /SGMII
Lossless Flow Control, IEEE 1588
Misc IO
Integrated Flash Controller supporting NOR, SLC and MLC based NAND devices
Dual UARTs, 2x I2Cs
Acceleration
Secure Boot
Battery Backed Secret Key
Anti-Tamper
Side channel attack resistance
6Gbps AES-HMAC-SHA-1
Asymmetric Ops
1024b Private Key (CRT) 115,400
1024 Public Key (17b exp) 1.6M
2048b Private Key (CRT) 31,700
2048b Public Key (17b exp) 571,000
Device
45 SOI process
29x29 1.0mm package
Power
~15W at 1.2GHz
0C to 105C Tj
Private Key operations are harder, they use equal sized modulus and exponent (1024, 2048, 4096) .
Public key operations typical have a full sized modulus (1024, 2048, 4096) but a small exponent (often 65,537, a 17b value)
Hash + Private key encrypt = Digital signature creation
Public key decrypt + hash = Digital signature verification
Private key operations allow for a short-cut known as Chinese Remainder Theorem (CRT). Because you only have 1 private key (or a very small number), you can pre-compute some values and save them off, for use in faster CRT based calculations. Everyone quotes private key operations with CRT.

10. OpenSSL with QorIQ SEC Accelerator
OpenSSL cryptodev “ENGINE” interface enable software to offload crypto operation to the SEC accelerator by:
Provides Hook Function glue code for interfacing with SEC engine
Provides customized crypto driver and plug in code for SEC engine access
Supports Cipher, Digest, PKI and RNG
Utilize the SEC capabilities to the fullest possible manner
Symmetric cryptography (AES, DES, 3DES)
Digest Calculation (MD5, SHA1, SHA2)
Random Number Generation
Asymmetric Crypto Support (Public Key Crypto)
SSL Record creation (TLSv1, SSLv3)
Two Approaches:
Direct Access to SEC (via USDPAA)
Indirect access to SEC (via a kernel driver)

11. OpenSSL Features Included
SSL protocol offloading
TLSv1 and SSLv3
Cipher offloading
AES (128/192/256), 3DES and DES
Digest offloading
MD5, SHA1, SHA224, SHA256, SHA384 and SHA512
Diffie-Hellman: First published public-key (asymmetric) crypto algororithm
Key generation: openssl genpkey -genparam -algorithm DH …
RSA: the second publicly announced public key (asymmetric) cryptography method
> openssl genpkey -algorithm RSA -out ${RSAKEYFILE} …
Encrypt with public key
Decrypt with private key
Digital Signature Algorithm: creation and verification of cryptographically secure signatures
Key generation e.g. openssl dsa -in ${DSAKEYFILE} -text -noout
Sign, Verify
Modified C Modules
openssl-1.0.1c/ssl/ssl_lib.c
openssl-1.0.1c/ssl/s3_clnt.c
openssl-1.0.1c/ssl/s3_srvr.c
openssl-1.0.1c/crypto/*
openssl-1.0.1c/crypto/pkc.c
include/openssl/ssl.h

12. Encryption Data Path with SSL offloading
With SSL offloading the same data packet can be encrypted with MAC appended in a SINGLE PASS
Plain Data/Contr ol Packet
If DATA Packet? No
Encrytp Packet
Encrypted Data/Control Packet
Yes
Offload SSL Data Record to SEC via ENGINE
Encrypt Data Record with MAC appended
Any error? No
Yes
Return ERROR

13. Decryption Data Path with SSL offloading
With SSL offloading the same data packet can be decrypted with MAC verification in a SINGLE PASS
Encrypted Data/Control Packet
If DATA Packet? No
Decrypt Packet
Plain Data/Control Packet
Yes
Offload SSL Data Record to SEC via ENGINE
Decrypt Data Record with MAC Verification
Any error? No
Yes
Return ERROR

14. OpenSSL Layered Architecture
OpenSSL application
SSL Library
SSLV3 handshake state machine
Crypto APIs
EVP
BIO
User
space
Engine Layer
Cryptodev engine
Kernel
Space
Crypto dev framework APIs
Kernel socket layer
Offload Driver (CAAM)
Hardware
Accelerator
T4240 SEC Engine
C290 Key Mgmt Accel

15. Enable OpenSSL with QorIQ SEC Engine

16. US-DPAA Flow Chart for ENGINE operations
A generic flow of data path in ENGINE glue code, used by all cryptographic offloading operations such as cipher, digest, RNG, PKI and SSL offloading.
Input Data
Crypto Library
Data Offloading
Engine Data Offloading
Copy Data to USDPAA DMA memory
Output Data
Copy Data to User Space Mem
Create:
Job Descriptor
& Frame Desc.
Return Error
to Engine
DEQ output from FQ
Queue Manager
ENQ FD to SEC Frame Queue
DEQ
ENQ
SEC Engine
Pull QMAN for Output Data
Error? No
Yes

17. SEC Queue Interface (QI) Interface
When the SEC’s Queue Interface has room for more jobs, it issues dequeue requests to the Queue Manager.
The QI uses a mechanism called Subportals to request FDs from different FQs.
Each dequeue request specifies one of N subportal IDs.
The QI is configured to request 1 or 3 Frame Descriptors.
In response, the Qman provides 1-3 Frame Descriptors and Frame Queue Summary Information.
Aside from debug scenarios, the SEC uses the following from the FQ Summary Info:
Number of Frames dequeued
Context A : Pointer to PreHeader
Context B : Frame Queue ID to enqueue results
Note:
T4240 SEC use dedicated channel 840h, WQ 4200h to 4207h.
P4080 SEC use dedicated channel 80h, WQ 400h to 407h.
channel
WQ0
WQ1
WQ2
WQ3
WQ4
WQ5
WQ6
WQ7
WQ0
WQ1
WQ2
WQ3
WQ4
WQ5
WQ6
WQ7
Channel 2112
HW Portal (DCP)
SP1
SP2
SP3
SP4
SP5
Queue Interface
SEC

18. SEC5 IPSec Protocol Example
preheader:
[00] BA8F0221 shrhdr: stidx=15 share=serial len=33
[15] A jump: jsl1 all-match[shrd] offset=17 local->[32]
PDB: IPSEC ESP ENCAP (CBC) PDB
[16] key: class2-md-split len=40 imm
[01] D Options:NextHdr=0x09 NHOffset=0 ChainedIV IPHeaderInPDB
[17] C8C1D7BF key=[c8c1d7bfa4e3ee84b ac9f
PrependOptIPHdr tunnel
[18] A4E3EE84
[02] rsv(ESN)
[19] B
[03] Seq Num = 1
[20] 3897AC9F
[04] 92CD6CE9 IV[92cd6ce9ab7c728c153a85cefd12ab79]
[21] 53A90ACA 53a90acac4c5e59e65940e330fa54d3d
[22] C4C5E59E
[05] AB7C728C
[23] 65940E33
[06] 153A85CE
[24] 0FA54D3D
[07] FD12AB79
[25] C71EFE3E c71efe3e2f205908]
[08] SPI=0x
[26] 2F205908
[09] OptIPHdrLength=20
[27] key: class1-keyreg len=16 imm
[10] 45A60014 Opt IP Header
[28] 86EB545E key=[86eb545eec5347b851a0539e2ff69a8d]
[11] 58335CB7
[12] 55C809F8
[29] EC5347B8
[13] B44767BB
[30] 51A0539E
[14] 79890A98
[31] 2FF69A8D
[32] 87010C07 operation: encap ipsec aes-cbc hmac-sha1-160

19. Enable OpenSSL with QorIQ Processors
Supported SDK
SDK1.5
Support boards:
P4080DS, P5040DS 32b / 64b, B4860QDS, T4240QDS
Building OpenSSL with hardware offloading support
Cross Compile Machine
$ ./scripts/host-prepare.sh
$ source ./poky/fsl-setup-poky -m t4240qds -t 16 -j 16 -l
$ bitbake fsl-image-core
Development System
modprobe cryptodev
cryptodev: driver 1.6 loaded.
openssl engine

20. Building a Custom OpenSSL and a Web Server
OpenSSL with TLS support can be used directly from Freescale SDK image or can be manually built with support for cryptodev:
$ ./config –DHAVE_CRYPTODEV –DUSE_CRYPTODEV_DIGESTS
$ make && make install
These two options are required to enable support for cryptodev into OpenSSL. If nginx is used as a web server, it can be built with support for SSL/TLS with the commands:
$ ./configure –-with-http_ssl_module –-with_openssl=<openssl_dir>
OpenSSL directory <openssl_dir> is the one where openssl tarball is extracted. Nginx build scripts will dive in there and build openssl before building itself.
If support for HW acceleration is required, nginx configuration command will be slightly different. For testing TLS acceleration, nginx must be linked with the OpenSSL version from Freescale.
$ ./configure –-with-http_ssl_module –-with_openssl=<openssl_dir> -- with_openss_opt=”-DHAVE_CRYPTODEV –DUSE_CRYPTODEV_DIGESTS”
Cryptodev engine will then have to be enabled in nginx configuration file: ssl_engine cryptodev;

21. Inserting Crypto-dev Module and Basic Checks
Test Scenario
A Freescale board is used as an HTTPS server responding to HTTPS requests from various SSL clients (e.g. web browsers).
TLS record offload shows performance improvement when the HTTPS is used to transfer large amount of data between server and client.
$ modprobe cryptodev
cryptodev: driver 1.6 loaded.
$ openssl engine
(cryptodev) BSD cryptodev engine
(dynamic) Dynamic engine loading support
If cryptodev driver is not loaded, OpenSSL will report only dynamic engine support and all operations will be done in user-space without HW acceleration
If crypto testing module was built into the kernel, it can be used to check if TLS support is available:
$ modprobe tcrypt
$ grep tls /proc/crypto
name : tls10(hmac(sha1),cbc(aes))
driver : tls10-hmac-sha1-cbc-aes-caam

22. OpenSSL Demo Https server(nginx+100M.html) ---------------------- DUT
OpenSSL s_client command will be used on Freescale board to make the connection with the server:
$ openssl s_client
The command can be scripted and run without further intervention:
$ echo GET /index.html | openssl s_client –connect <server_ip>:443 –cipher AES128-SHA –tls1 –quiet
The option “–quiet” can be removed to see more details about the TLS session.
OpenSSL will use automatically the HW acceleration if cryptodev module is loaded in the kernel.

23. OpenSSL Demo Configuration
A100M file from https server to DUT:
time echo GET /100M.html | openssl s_client -connect : tls1 -cipher AES128-SHA -pause -quiet > /dev/null 2>&1
collect cpu utilization by mpstate during the https get.
mpstate -P ALL 1
Example:
if got real time 5
throughput = 100M / 5 = 20Mbps

24. Performance Benefits with SEC Engine

25. T4240QDS System SEC 5.0 Board OS Core Sec Engine Frequency Cache
T4240QDS (Rev 2.0 silicon) OS
SMP Linux bit / 32bit user space
Core
12 x e6500 cores * 2 threads @1600MHz
Sec Engine
Sec 5 : 8 x DECO
Frequency
Core/CCB/DDR: 1667/733/933
Cache
L1 : 32 Kbytes Dcache and Icache with 64 byte line size
L2 : 2MB shared L2 cache
L3 : 512KB CPC per DDRC, totally 1.5MB CPC for 3 DDRC
Memory
CPC : 512K per DDR controller
6GB single rank DIMM with 1x 64b
U-Boot
U-boot
Filesystem
Ramdisk file system
Compiler
gcc-4.7.3, eglibc-2.15, binutils

26. OpenSSL Offload Result for P4080DS
Benefits of SEC Accelerator
4x Performance improvement

27. Key Management CPU Utilization Without Accelerator
T4240 has 24 cores, reach maximum performance with a thread per core
With no offload, OpenSSL uses up all CPU resources

28. Key Management CPU Utilization With Accelerator (C293)
With C293 accelerator, CPU utilization is fairly low
Core 0-3 show higher CPU utilization as they are dedicated for C290 IO task

29. Summary

30. Summary
Enabling the OpenSSL offload through SEC means most Linux apps will benefit through performance improvement and reduced CPU utilization
Freescale offers user space application the flexibility of direct access via USDPAA environment, or indirect access to SEC via kernel driver

32. Kernel Driver Based Access to SEC Engine
AF_ALG Interface
CryptoDev Interface
AF_ALG Interface
CryptoDev Interface
Supported by Linux Community – Upstreaming possible
Asynchronous support is not available
Do not support PKI and SSL
Supported by Linux Community - Upstreaming possible
Supports Asynchronous mode
openssl-1.0.1c/ssl/ssl_lib.c
SSL_set_async ( )
SSL_Async_cb()
SSL_set_eng_handle ( )
Throughput better than AF_ALG
No support for PKI and SSL