1. Building and Porting embedded operating systems
Difference between booting a PC or an Embedded System

2. The Booting Process Power On PC Embedded Power On BIOS X-Loader
Get Bootloader
Get Bootloader
GRUB
U-Boot
Get Kernel
Get Kernel
Kernel
Kernel
So what exactly is the difference between an embedded system and a PC when booting? [CLICK] When you take it from a top level abstracted point of view, we can see it as such. [CLICK] Taking the PC and an IGEP board for an example, the board first powers on, loads the BIOS and X-loader binaries respectively These test the system and bring up various interfaces, for example disk access, memory and networking. Using this, they go off and get their respective bootloaders, which then go and boot the kernel.

3. Embedded System vs PC
PCs are a highly modular platform. Most components are in sockets or slots that permits easy replacement.
Embedded Systems tend to solder their components directly to the PCB as they don’t need to be replaced.
What is the difference between an embedded system and your common desktop PC? Simply put, the PC’s modularity and vast array of interchangeable components are both a boon and at the same time a hindrance. Modularity means upgrades and replacements are easy, so you can upgrade your old machine to breathe new life into it, or replace failed components when all life has seemingly been put out. [CLICK]
Embedded Systems commonly do not have modular components. Indeed, the core things like CPUs, RAM, I/O are all soldered directly to the board. [CLICK] When it comes to embedded boards, you also have a large gap between peripherals on them and what you’d find on a desktop PC. All this comes into play when you have to consider bootloaders on the two. [NEXT SLIDE]

4. Embedded System vs PC i7-920 i7-930 i7-940 i7-950 i7-965X i7-960
To illustrate my point, consider your computer motherboard. Here is mine, for example. [CLICK]. The motherboard itself is already a pretty busy place, and you can quite clearly see all the slots and connectors scattered over the surface that permit different components to be inserted.
Let’s start with the CPU socket itself. This is an LGA1366 socket, [CLICK, lots happen] and these are an example of all the different CPUs that could possibly fit into this socket that the BIOS has to be aware of and know what to deal with. The Intel Nehalem platform is quite a small platform too, if you look back at the Pentium 4/Core 2 LGA775 socket, there are literally hundreds of CPUs that fit that socket. [NEXT SLIDE]
i7-980X

5. Embedded System vs PC Voltage DDR3-2133 1.2-1.5v DDR3-1866 DDR3-1600
Latency
CS-RAS-CAS-WE
Now for the memory. Six slots, all DDR3. [CLICK] But the variety of memory modules that could possibly be inserted into the board is astonishing. Different manufacturers of different types of memory. Infact, this exact board has fallen foul of not supporting a specific combination of memory, leading to many “interesting” BSODs and Kernel Panics due to the board not recognising the correct timings.
[CLICK] DDR3 itself comes in many different speeds, [CLICK] Different voltages and [CLICK] different timings.

6. Different PCI-E standards Also Legacy PCI support
Embedded System vs PC
Different PCI-E standards
V1.0/v2.0/v3.0
Different lane counts
x1/x2/x4/x8/x16
Also Legacy PCI support
Then we come to the more “general purpose” slots, that can house many different sorts of add in cards. PCI-Express has pretty much entirely supplanted the older AGP and PCI slots, indeed some of the newer boards that have been released within the last 2 years have come without any PCI slots at all.
[CLICK] PCI-Express itself was originally designed as a slot for graphics cards, with bandwidth far surpassing the existing AGP standard. [CLICK] But then the standard also found itself home to other cards, for example Sound Cards, Ethernet NICs, WiFi, even solid state storage devices. And with ATX boards supporting up to 7 of these slots with a capability of 7 different devices being present is a nightmare for initialising the board and its connected hardware.
[CLICK] And there are differing standards across devices and boards that need to be negotiated on discovering what devices they are. [NEXT SLIDE]

7. Embedded System vs PC USB 2.0 USB 3.0 Intel PCH Gigabit Ethernet
7.1 Audio Codec
S-ATA
3 & 6Gbps
And then we have some of the on board peripherals, Including USB [CLICK] and its newer sibling [CLICK], Ethernet [CLICK], Audio [CLICK], Disk access [CLICK] and even legacy IDE and floppy support for those that really don’t want to let go from their archaic 128kbps WAV files on their 80GB IDE drives are are too stingy to buy a cheap new SATA drive. [CLICK]. Even the chipset itself is scanned in the BIOS boot process. [NEXT SLIDE]
IDE & Floppy

8. Embedded System vs pc TI OMAP 3730 Processor 512MB DDR@200MHz
PowerVR SGX530 Graphics
All soldered directly to the
board. Not intended for
replacements.
And so, what is the difference on an embedded board such as the IGEP? Well it still has a lot of the peripherals to worry about that the PC does, like USB [CLICK], Audio [CLICK], Storage Devices [CLICK], Bluetooth [CLICK], WiFi [CLICK], Ethernet [CLICK], and Video output [CLICK]. But crucially, what matters most to the boot process is that the CPU, memory and other vital components are always going to be there. [CLICK]. This allows you to write a very customised bootloader for it, as it doesn’t need to scan for different processors or different memory, you can just tell it that this is a 720MHz ARM core running this version of the instruction set, this is what memory it has at this speed and what else is directly on the board and is never going away. [CLICK] [NEXT SLIDE]

9. Embedded System vs PC
PC booting is a much longer process than for embedded systems.
Desktop PCs are by far the worst due to their massive compatibility issue
Laptops are faster because their components are much more controlled.
[CLICK] So as we can see from the previous slides, the desktop machine has so much more hardware to deal with that the boot process is much much longer than it is for an embedded System [CLICK] The desktop PC has so many components both legacy and current that it has to deal with that it gets simply bogged down.
To showcase just how slow it is, I did a bit of testing on four different platforms with some data I have from my placement last year. [CLICK] This graph shows the comparative boot times for the system to respond, i.e. getting to the POST stage; time to the bootloader being run from the BIOS, and then the time until a Linux 2.6 kernel prompt. (3.2 on x86). We can see that the IGEP and Raspberry Pi boards are very fast bootingthrough the BIOS and bootloader stages. The Raspberry Pi almost doesn’t count here as the BIOS and bootloader are one and the same in Raspberry Pi architecture. The BIOS reads a text file contained at the root of the SD card and that details the CPU operating speed, memory configuration and where the kernel image is contained along with any options to be passed to the kernel. It is a very stripped down and not particularly extensible interface because the Pi is a production device, NOT a development board. (That and Broadcom don’t want people faffing around in the bootloaders)
The IGEP takes longer to bring up because u-boot is far more fully featured in its bootloader design.
The BCM97425 is a set top box platform I worked with last year that I have some data from doing a boot test for a customer I was working with. It’s an interesting design because it’s not your average embedded system as it contains a lot of the features common to PCs. It has video encoders and decoders, transport busses, graphics, as well as peripheral support including SATA, PCI-E, Bluetooth, WiFi, USB. So you can see that the sheer amount of things it has to initialise means the POST stage of CFE takes around 5 seconds.
Laptops and Desktops are interesting because Laptops are a more confined set of components, so their BIOS is much smaller.
Desktop PC: Gigabyte GA-X58A-UD3R, Intel Core 4.2GHz, 12GB DDR3-1600, GeForce GTX580, Crucial M4 256GB SSD
Laptop PC: Dell XPS 15 L502x, Intel Core 2.2GHz, 6GB DDR3-1600, Intel HD3000 & GeForce GT540M, Western Digital Caviar Black 750GB.

10. Building and Porting embedded operating systems
BIOS and Bootloaders

11. BIOS BIOS stands for Basic Input Output System
First performs Power on self test and initializes Memory, PCI bus, Video, storage, Network(PXE), HID and any other systems a bootloader may need to load the OS.
Allows user to choose hardware settings within a simple UI
X86 always starts CPU in real mode.
20 bit segmented memory address space
Only 1 MiB of addressable memory.

12. BIOS

13. BIOS
Post POST search for a bootable drive and copy 512byte MBR to RAM.
This 512bytes is the MBR
Partition table 64b
Two byte boot signature
Optional Disk timestamps and signatures
First stage Bootstrap code takes up the rest
The BIOS yields control to the bootstrap who's only task is to locate and execute a more complex second stage bootloader (grub,BOOTMGR,NTLDR)

14. Bootloader What is a bootloader?
Program used to load an operating system or another boot loader.
Boot loaders run in Real mode and use BIOS provided code
Operating systems replace BIOS code with their own.
What is a second stage bootloader?
Programs like GRUB, LILO or NTLDR
allow multi-boot systems
They are aware of file systems
Allow creation of ram discs in memory to boot the linux kernel
Once the kernel has retrieved all the information it requires in Real mode it will change the cpu to protected mode providing support for virtual memory, memory protection, multitasking and code privilege levels.

15. UEFI – The Future? What is it? Unified Extensible Firmware Interface
Replacement for BIOS
Improvements
Not a single piece of firmware but a programmable software interface.
It is hardware independent
It provides a standard method of interfacing with firmware during the boot process
It's faster due to using Block I/O over interrupts

16. UEFI – The Future?
64bit version can address all system memory while booting.
Can boot from disks larger than 2TB
Pre-boot networking support
Mouse usable in boot options
You can mount partitions and read file systems
Doesn't solve the problem of requiring two drivers. One for the firmware and one for the operating system.
Secure Boot(arguably)

17. Building and Porting embedded operating systems
IGEPv2 framework

18. IGEP boot process
ROM Code
X-Loader
U-Boot
Kernel

19. ROM Code What does the IGEP ROM code do?
Code is executed in a known location in ROM
Minimal Configuration is performed
Some clocks set
Some memories and peripherals initialised
Searches boot devices for suitable boot image
Supports serial, SD card, NAND flash and USB amongst others.
Boot order is defined by a set of GPIO configuration pins referred to as SYSBOOT
On finding the first x-loader image it is copied to SRAM and then executed

20. ROM Code
Serial Boot
ID is written out to the serial port and a response is waited for.
If response received within time limit ROM will transfer data received from serial port to SRAM and execute it.
SD Boot
The ROM code looks for a suitably formatted SD card on the MMC controllers. If a readable partition is found directory is scanned for a specially signed file called an "MLO" . Assuming the file is as expected, it is transferred into the internal SRAM and control is passed to it.
NAND\eMMC
If NAND is in the boot list the ROM code will attempt to load the first sector, skipping it if its bad, corrupt or blank. If it reaches the 4th sector and no good sector has been found it will move on. If a good sector is found then it will load the contents to ram and then start executing it.

21. X-loader What is x-loader?
X-loader is a first stage boot-loader that implements a subset of the features of u-boot.
It's small enough to fit in on-chip memory but still provides the ability to configure the pin muxing, clocks, serial console and DDR access to load the fully featured u-boot into the second stage boot loader.
In IGEP a copy of the x-loader is part of the NAND
Not limited to 512bytes
MLO files
Through MLO files the need for a second stage boot loader can be removed.
An MLO file is a an x-loader file that has been given a header with the size of the image and deployment memory location.
You can optionally define clock frequency amongst other things
MLO files allow zImages to be booted without the need for a second stage boot loader
Serial and SD
X-loader supports transfering an image to the board using kermit over a serial connection and will also load a uboot.bin file stored on the SD.

22. Boot-loaders are doing configuration that the kernel is not!
Watch out!
Boot-loaders are doing configuration that the kernel is not!

23. JTAG
Initially devised for testing printed circuit boards with its boundary scanning functionality JTAG is now used extensively for debugging, programming CPLDS and initialising flash memory.
Can be useful to recover bricked devices or write new firmware to NAND on restricted devices. But the IGEP is un-brickable
Tools like OpenOCD and GDB ARM have successfully been used on OMAP530 devices like the beagleboard.

24. Building and Porting embedded operating systems
Das U-Boot

25. Building and Porting embedded operating systems
OS/Bootloader Coupling in:

26. Qnx? QNX is a Canadian microkernel-based RTOS.
Device drivers are all run as userspace applications outside the kernel.
Used in many applications, mainly automotive systems.
HAM (High Availability Manager).
Adaptive Partitioning Scheduler.
Safety critical certification.
QNX owned by RIM.
Tightly coupled bootloader and kernel.
Transparent distributed networking.
Adaptive partitioning is a weird way of scheduling real-time processes which introduces not just priorities, but puts processes into groups with a pre-defined % of the CPU time, and determines priorities within those groups and allowing the highest of those to run. This can be used to circumvent issues of priority inversion tidily, allowing the programmer not to have to care, as it’s a problem of system configuration.
Called QNX “Neutrino” technically.
Started in the 80s as “QUNIX” told to stop by UNIX name owners.
Transparent networking, easy programming interfaces, automatically detects other QNX network nodes and makes them accessible under /net/*
Can open net paths with fopen, and the kernel queries the I/O manager what driver to pass it to. If it’s a network address, I/O manager sends you to network driver, if it’s local, it sends you to the relevant filesystem driver. Programmer doesn’t have to care! 

27. Bootloader/OS Interaction
Most general purpose operating systems (such as linux) avoid becoming too tightly coupled with their bootloaders.
This has led to bootloaders which support a multitude of functions.
An RTOS developer can take advantage of this by using the generic bootloader (and it’s more advanced capabilities (NFS, ping) to ensure hardware works, and easily bring it to life.
During development this general purpose bootloader can be effectively used as a “bootloader loader”.
The high-functionality (and easy to hack!) bootloader (i.e. U-Boot) can then be removed from a shipped product, leaving behind only a basic tightly-coupled version.
Introduce advantages of configurable bootloaders (flexibility during development, works across many hardware configurations etc.)
Discuss disadvantages (many industries paranoid about hardware hacking, custom firmware, services for free)
QNX Kernel doesn’t have to care about what bootloader is at bottom level, it can always build on known environment provided by IPL.
That’s useful for Dev and Release.

28. QNX Booting
U-Boot
On-Chip boot ROM code generally causes the CPU to perform minimal initialisation of peripheral such as NAND Flash and instructs it to begin reading code from there into memory and executing it.
This code can be a standard embedded bootloader such as U-Boot, or it can be a QNX IPL.
U-Boot loads the QNX “IPL” image into memory, and begins executing it.
The “IPL” is an “Initial Program Loader” which is responsible for initialising basic hardware and passing control to “Startup” code, and subsequently the QNX Kernel.
IPL
Startup
QNX Kernel
Explain that the board can be booted purely from IPL if desired.
Many CPUs include an on-chip boot ROM which is very capable. (SPI NOR Flash booting, Ethernet booting.)
Often the problem is not how to boot the board, but which of the many methods suits you best.

29. QNX IPL Begins in assembly, performs initialisation for HLL.
Initialises CPU/(some) Peripheral Clocks.
Initialises basic I/O (serial).
Minimal pin multiplexing for required peripherals (i.e. SDHC hardware).
Reads in and decompresses “IFS” image (ramdisk + kernel).
Includes basic (FAT) filesystem drivers for SDHC reading.
Passes control to “Startup”
Can start “minidrivers” for device interaction before OS/Kernel even begins booting.
CODE
Initialises more advanced chip features in assembly language.
Not running in SMP mode yet, system is only aware of one thread of execution.
Talk quickly about pinmuxing, multiple functions per pin, not enough physical pins to map every possible peripheral combination of SoC.
Discuss IFS (normally compressed binary blob of kernel and rootfs.)
Coolest thing is minidrivers. Tight coupling, will discuss more detail later.
CODE

30. QNX Startup
Startup begins in C language, initialises most peripherals, and sets up important kernel structures.
The QNX kernel expects a “syspage” structure to exist at a pre-defined location in memory. This structures provides important information about the host system.
Enables CPU SMP operation (multiple-cores).
Often re-does initialisation done by IPL (such as serial I/O) to enable more advanced functionality.
Informs minidrivers of new environment before passing control to kernel.
The syspage is the important thing here. It contains some information determined at compile time.
Other information is determined by the startup and IPL, and then passed to kernel to prevent need for re-discovery.
The minidrivers must be informed that they’re going to be switching to a managed scheduled environment. Up till now, they’ve been running outside the OS, but they’re about to be brought in.
CODE

31. syspage
Indicates CPU type (e.g. ARM) and vital information (e.g. number of cores), and other supported features such as NEON extensions.
Provides access to hardware-specific function callouts made available to the system before the Kernel was running.
Provides information about the memory environment in which the kernel is running.
Information about bus devices, IRQs.
Information about connected peripherals and device trees for /dev population.
Syspage prevents need for re-discovery of connected hardware and features.
Although hardware is often re-initialised, details about it are still known which speeds up the process. -

32. Minidrivers
Prevents need for auxillary processors for instant-on peripheral ineraction.
Linked against bootloader code.
Called periodically during startup, initially with polling and then interrupts when they become available.
Full drivers are allowed access to minidriver memory once the kernel has loaded so they can take over without data loss.
Make use of I/O hardware within 10ms of power-on.
Minidrivers are a little bit unique.
E.g. You need to respond to messages on a CAN bus in a car within 10ms of powering on, because other devices might be that fast.
The minidriver allows you to respond and be a part of the whole system long before the kernel has even thought about booting.
Vital information and messages are not lost while waiting for the system to boot – the minidriver memory is passed on to high level device drivers which should have been programmed to read it.

33. Summary QNX Implements its own FS-aware bootloader (IPL).
This bootloader is tightly coupled with its own system initialisation and kernel.
This bootloader is only configurable via source code, unlike U-Boot or Grub.
Source code for bootloader and startup is freely available for a variety of hardware (kernel is not):
Questions?
Any Q?