Building and Porting embedded operating systems Difference between booting a PC or an Embedded System

The Booting Process Power On PC Embedded Power On BIOS X-Loader Get Bootloader Get Bootloader GRUB U-Boot Get Kernel Get Kernel Kernel Kernel So what exactly is the difference between an embedded system and a PC when booting? [CLICK] When you take it from a top level abstracted point of view, we can see it as such. [CLICK] Taking the PC and an IGEP board for an example, the board first powers on, loads the BIOS and X-loader binaries respectively These test the system and bring up various interfaces, for example disk access, memory and networking. Using this, they go off and get their respective bootloaders, which then go and boot the kernel.

Embedded System vs PC PCs are a highly modular platform. Most components are in sockets or slots that permits easy replacement. Embedded Systems tend to solder their components directly to the PCB as they don’t need to be replaced. What is the difference between an embedded system and your common desktop PC? Simply put, the PC’s modularity and vast array of interchangeable components are both a boon and at the same time a hindrance. Modularity means upgrades and replacements are easy, so you can upgrade your old machine to breathe new life into it, or replace failed components when all life has seemingly been put out. [CLICK] Embedded Systems commonly do not have modular components. Indeed, the core things like CPUs, RAM, I/O are all soldered directly to the board. [CLICK] When it comes to embedded boards, you also have a large gap between peripherals on them and what you’d find on a desktop PC. All this comes into play when you have to consider bootloaders on the two. [NEXT SLIDE]

Embedded System vs PC i7-920 i7-930 i7-940 i7-950 i7-965X i7-960 To illustrate my point, consider your computer motherboard. Here is mine, for example. [CLICK]. The motherboard itself is already a pretty busy place, and you can quite clearly see all the slots and connectors scattered over the surface that permit different components to be inserted. Let’s start with the CPU socket itself. This is an LGA1366 socket, [CLICK, lots happen] and these are an example of all the different CPUs that could possibly fit into this socket that the BIOS has to be aware of and know what to deal with. The Intel Nehalem platform is quite a small platform too, if you look back at the Pentium 4/Core 2 LGA775 socket, there are literally hundreds of CPUs that fit that socket. [NEXT SLIDE] i7-980X

Embedded System vs PC Voltage DDR3-2133 1.2-1.5v DDR3-1866 DDR3-1600 Latency CS-RAS-CAS-WE Now for the memory. Six slots, all DDR3. [CLICK] But the variety of memory modules that could possibly be inserted into the board is astonishing. Different manufacturers of different types of memory. Infact, this exact board has fallen foul of not supporting a specific combination of memory, leading to many “interesting” BSODs and Kernel Panics due to the board not recognising the correct timings. [CLICK] DDR3 itself comes in many different speeds, [CLICK] Different voltages and [CLICK] different timings.

Different PCI-E standards Also Legacy PCI support Embedded System vs PC Different PCI-E standards V1.0/v2.0/v3.0 Different lane counts x1/x2/x4/x8/x16 Also Legacy PCI support Then we come to the more “general purpose” slots, that can house many different sorts of add in cards. PCI-Express has pretty much entirely supplanted the older AGP and PCI slots, indeed some of the newer boards that have been released within the last 2 years have come without any PCI slots at all. [CLICK] PCI-Express itself was originally designed as a slot for graphics cards, with bandwidth far surpassing the existing AGP standard. [CLICK] But then the standard also found itself home to other cards, for example Sound Cards, Ethernet NICs, WiFi, even solid state storage devices. And with ATX boards supporting up to 7 of these slots with a capability of 7 different devices being present is a nightmare for initialising the board and its connected hardware. [CLICK] And there are differing standards across devices and boards that need to be negotiated on discovering what devices they are. [NEXT SLIDE]

Embedded System vs PC USB 2.0 USB 3.0 Intel PCH Gigabit Ethernet 7.1 Audio Codec S-ATA 3 & 6Gbps And then we have some of the on board peripherals, Including USB [CLICK] and its newer sibling [CLICK], Ethernet [CLICK], Audio [CLICK], Disk access [CLICK] and even legacy IDE and floppy support for those that really don’t want to let go from their archaic 128kbps WAV files on their 80GB IDE drives are are too stingy to buy a cheap new SATA drive. [CLICK]. Even the chipset itself is scanned in the BIOS boot process. [NEXT SLIDE] IDE & Floppy

Embedded System vs pc TI OMAP 3730 Processor 512MB DDR@200MHz PowerVR SGX530 Graphics All soldered directly to the board. Not intended for replacements. And so, what is the difference on an embedded board such as the IGEP? Well it still has a lot of the peripherals to worry about that the PC does, like USB [CLICK], Audio [CLICK], Storage Devices [CLICK], Bluetooth [CLICK], WiFi [CLICK], Ethernet [CLICK], and Video output [CLICK]. But crucially, what matters most to the boot process is that the CPU, memory and other vital components are always going to be there. [CLICK]. This allows you to write a very customised bootloader for it, as it doesn’t need to scan for different processors or different memory, you can just tell it that this is a 720MHz ARM core running this version of the instruction set, this is what memory it has at this speed and what else is directly on the board and is never going away. [CLICK] [NEXT SLIDE]

Embedded System vs PC PC booting is a much longer process than for embedded systems. Desktop PCs are by far the worst due to their massive compatibility issue Laptops are faster because their components are much more controlled. [CLICK] So as we can see from the previous slides, the desktop machine has so much more hardware to deal with that the boot process is much much longer than it is for an embedded System [CLICK] The desktop PC has so many components both legacy and current that it has to deal with that it gets simply bogged down. To showcase just how slow it is, I did a bit of testing on four different platforms with some data I have from my placement last year. [CLICK] This graph shows the comparative boot times for the system to respond, i.e. getting to the POST stage; time to the bootloader being run from the BIOS, and then the time until a Linux 2.6 kernel prompt. (3.2 on x86). We can see that the IGEP and Raspberry Pi boards are very fast bootingthrough the BIOS and bootloader stages. The Raspberry Pi almost doesn’t count here as the BIOS and bootloader are one and the same in Raspberry Pi architecture. The BIOS reads a text file contained at the root of the SD card and that details the CPU operating speed, memory configuration and where the kernel image is contained along with any options to be passed to the kernel. It is a very stripped down and not particularly extensible interface because the Pi is a production device, NOT a development board. (That and Broadcom don’t want people faffing around in the bootloaders) The IGEP takes longer to bring up because u-boot is far more fully featured in its bootloader design. The BCM97425 is a set top box platform I worked with last year that I have some data from doing a boot test for a customer I was working with. It’s an interesting design because it’s not your average embedded system as it contains a lot of the features common to PCs. It has video encoders and decoders, transport busses, graphics, as well as peripheral support including SATA, PCI-E, Bluetooth, WiFi, USB. So you can see that the sheer amount of things it has to initialise means the POST stage of CFE takes around 5 seconds. Laptops and Desktops are interesting because Laptops are a more confined set of components, so their BIOS is much smaller. Desktop PC: Gigabyte GA-X58A-UD3R, Intel Core i7-950 @ 4.2GHz, 12GB DDR3-1600, GeForce GTX580, Crucial M4 256GB SSD Laptop PC: Dell XPS 15 L502x, Intel Core i7-2670QM @ 2.2GHz, 6GB DDR3-1600, Intel HD3000 & GeForce GT540M, Western Digital Caviar Black 750GB.

Building and Porting embedded operating systems BIOS and Bootloaders

BIOS BIOS stands for Basic Input Output System First performs Power on self test and initializes Memory, PCI bus, Video, storage, Network(PXE), HID and any other systems a bootloader may need to load the OS. Allows user to choose hardware settings within a simple UI X86 always starts CPU in real mode. 20 bit segmented memory address space Only 1 MiB of addressable memory.

BIOS

BIOS Post POST search for a bootable drive and copy 512byte MBR to RAM. This 512bytes is the MBR Partition table 64b Two byte boot signature Optional Disk timestamps and signatures First stage Bootstrap code takes up the rest The BIOS yields control to the bootstrap who's only task is to locate and execute a more complex second stage bootloader (grub,BOOTMGR,NTLDR)

Bootloader What is a bootloader? Program used to load an operating system or another boot loader. Boot loaders run in Real mode and use BIOS provided code Operating systems replace BIOS code with their own. What is a second stage bootloader? Programs like GRUB, LILO or NTLDR allow multi-boot systems They are aware of file systems Allow creation of ram discs in memory to boot the linux kernel Once the kernel has retrieved all the information it requires in Real mode it will change the cpu to protected mode providing support for virtual memory, memory protection, multitasking and code privilege levels.

UEFI – The Future? What is it? Unified Extensible Firmware Interface Replacement for BIOS Improvements Not a single piece of firmware but a programmable software interface. It is hardware independent It provides a standard method of interfacing with firmware during the boot process It's faster due to using Block I/O over interrupts

UEFI – The Future? 64bit version can address all system memory while booting. Can boot from disks larger than 2TB Pre-boot networking support Mouse usable in boot options You can mount partitions and read file systems Doesn't solve the problem of requiring two drivers. One for the firmware and one for the operating system. Secure Boot(arguably)

Building and Porting embedded operating systems IGEPv2 framework

IGEP boot process ROM Code X-Loader U-Boot Kernel

ROM Code What does the IGEP ROM code do? Code is executed in a known location in ROM Minimal Configuration is performed Some clocks set Some memories and peripherals initialised Searches boot devices for suitable boot image Supports serial, SD card, NAND flash and USB amongst others. Boot order is defined by a set of GPIO configuration pins referred to as SYSBOOT On finding the first x-loader image it is copied to SRAM and then executed

ROM Code Serial Boot ID is written out to the serial port and a response is waited for. If response received within time limit ROM will transfer data received from serial port to SRAM and execute it. SD Boot The ROM code looks for a suitably formatted SD card on the MMC controllers. If a readable partition is found directory is scanned for a specially signed file called an "MLO" . Assuming the file is as expected, it is transferred into the internal SRAM and control is passed to it. NAND\eMMC If NAND is in the boot list the ROM code will attempt to load the first sector, skipping it if its bad, corrupt or blank. If it reaches the 4th sector and no good sector has been found it will move on. If a good sector is found then it will load the contents to ram and then start executing it.

X-loader What is x-loader? X-loader is a first stage boot-loader that implements a subset of the features of u-boot. It's small enough to fit in on-chip memory but still provides the ability to configure the pin muxing, clocks, serial console and DDR access to load the fully featured u-boot into the second stage boot loader. In IGEP a copy of the x-loader is part of the NAND Not limited to 512bytes MLO files Through MLO files the need for a second stage boot loader can be removed. An MLO file is a an x-loader file that has been given a header with the size of the image and deployment memory location. You can optionally define clock frequency amongst other things MLO files allow zImages to be booted without the need for a second stage boot loader Serial and SD X-loader supports transfering an image to the board using kermit over a serial connection and will also load a uboot.bin file stored on the SD.

Boot-loaders are doing configuration that the kernel is not! Watch out! Boot-loaders are doing configuration that the kernel is not!

JTAG Initially devised for testing printed circuit boards with its boundary scanning functionality JTAG is now used extensively for debugging, programming CPLDS and initialising flash memory. Can be useful to recover bricked devices or write new firmware to NAND on restricted devices. But the IGEP is un-brickable Tools like OpenOCD and GDB ARM have successfully been used on OMAP530 devices like the beagleboard.

Building and Porting embedded operating systems Das U-Boot

Building and Porting embedded operating systems OS/Bootloader Coupling in:

Qnx? QNX is a Canadian microkernel-based RTOS. Device drivers are all run as userspace applications outside the kernel. Used in many applications, mainly automotive systems. HAM (High Availability Manager). Adaptive Partitioning Scheduler. Safety critical certification. QNX owned by RIM. Tightly coupled bootloader and kernel. Transparent distributed networking. Adaptive partitioning is a weird way of scheduling real-time processes which introduces not just priorities, but puts processes into groups with a pre-defined % of the CPU time, and determines priorities within those groups and allowing the highest of those to run. This can be used to circumvent issues of priority inversion tidily, allowing the programmer not to have to care, as it’s a problem of system configuration. Called QNX “Neutrino” technically. Started in the 80s as “QUNIX” told to stop by UNIX name owners. Transparent networking, easy programming interfaces, automatically detects other QNX network nodes and makes them accessible under /net/* Can open net paths with fopen, and the kernel queries the I/O manager what driver to pass it to. If it’s a network address, I/O manager sends you to network driver, if it’s local, it sends you to the relevant filesystem driver. Programmer doesn’t have to care! 

Bootloader/OS Interaction Most general purpose operating systems (such as linux) avoid becoming too tightly coupled with their bootloaders. This has led to bootloaders which support a multitude of functions. An RTOS developer can take advantage of this by using the generic bootloader (and it’s more advanced capabilities (NFS, ping) to ensure hardware works, and easily bring it to life. During development this general purpose bootloader can be effectively used as a “bootloader loader”. The high-functionality (and easy to hack!) bootloader (i.e. U-Boot) can then be removed from a shipped product, leaving behind only a basic tightly-coupled version. Introduce advantages of configurable bootloaders (flexibility during development, works across many hardware configurations etc.) Discuss disadvantages (many industries paranoid about hardware hacking, custom firmware, services for free) QNX Kernel doesn’t have to care about what bootloader is at bottom level, it can always build on known environment provided by IPL. That’s useful for Dev and Release.

QNX Booting U-Boot On-Chip boot ROM code generally causes the CPU to perform minimal initialisation of peripheral such as NAND Flash and instructs it to begin reading code from there into memory and executing it. This code can be a standard embedded bootloader such as U-Boot, or it can be a QNX IPL. U-Boot loads the QNX “IPL” image into memory, and begins executing it. The “IPL” is an “Initial Program Loader” which is responsible for initialising basic hardware and passing control to “Startup” code, and subsequently the QNX Kernel. IPL Startup QNX Kernel Explain that the board can be booted purely from IPL if desired. Many CPUs include an on-chip boot ROM which is very capable. (SPI NOR Flash booting, Ethernet booting.) Often the problem is not how to boot the board, but which of the many methods suits you best.

QNX IPL Begins in assembly, performs initialisation for HLL. Initialises CPU/(some) Peripheral Clocks. Initialises basic I/O (serial). Minimal pin multiplexing for required peripherals (i.e. SDHC hardware). Reads in and decompresses “IFS” image (ramdisk + kernel). Includes basic (FAT) filesystem drivers for SDHC reading. Passes control to “Startup” Can start “minidrivers” for device interaction before OS/Kernel even begins booting. CODE Initialises more advanced chip features in assembly language. Not running in SMP mode yet, system is only aware of one thread of execution. Talk quickly about pinmuxing, multiple functions per pin, not enough physical pins to map every possible peripheral combination of SoC. Discuss IFS (normally compressed binary blob of kernel and rootfs.) Coolest thing is minidrivers. Tight coupling, will discuss more detail later. CODE

QNX Startup Startup begins in C language, initialises most peripherals, and sets up important kernel structures. The QNX kernel expects a “syspage” structure to exist at a pre-defined location in memory. This structures provides important information about the host system. Enables CPU SMP operation (multiple-cores). Often re-does initialisation done by IPL (such as serial I/O) to enable more advanced functionality. Informs minidrivers of new environment before passing control to kernel. The syspage is the important thing here. It contains some information determined at compile time. Other information is determined by the startup and IPL, and then passed to kernel to prevent need for re-discovery. The minidrivers must be informed that they’re going to be switching to a managed scheduled environment. Up till now, they’ve been running outside the OS, but they’re about to be brought in. CODE

syspage Indicates CPU type (e.g. ARM) and vital information (e.g. number of cores), and other supported features such as NEON extensions. Provides access to hardware-specific function callouts made available to the system before the Kernel was running. Provides information about the memory environment in which the kernel is running. Information about bus devices, IRQs. Information about connected peripherals and device trees for /dev population. Syspage prevents need for re-discovery of connected hardware and features. Although hardware is often re-initialised, details about it are still known which speeds up the process. -

Minidrivers Prevents need for auxillary processors for instant-on peripheral ineraction. Linked against bootloader code. Called periodically during startup, initially with polling and then interrupts when they become available. Full drivers are allowed access to minidriver memory once the kernel has loaded so they can take over without data loss. Make use of I/O hardware within 10ms of power-on. Minidrivers are a little bit unique. E.g. You need to respond to messages on a CAN bus in a car within 10ms of powering on, because other devices might be that fast. The minidriver allows you to respond and be a part of the whole system long before the kernel has even thought about booting. Vital information and messages are not lost while waiting for the system to boot – the minidriver memory is passed on to high level device drivers which should have been programmed to read it.

Summary QNX Implements its own FS-aware bootloader (IPL). This bootloader is tightly coupled with its own system initialisation and kernel. This bootloader is only configurable via source code, unlike U-Boot or Grub. Source code for bootloader and startup is freely available for a variety of hardware (kernel is not): http://community.qnx.com/sf/wiki/do/viewPage/projects.bsp/wiki/BSPAndDrivers Questions? Any Q?