Column 3: Two levels of bare-metal programming (2011-07-13)

To understand the essence of Coreboot, it is important to understand that there are two levels of bare metal programming on the PC. I will illustrate this with the RS232 serial port, under DOS known as COM1. There are generally two ways to use the serial port.

• Using the operating system (or BIOS) functions. Under MS-DOS you could open the COM1 device and send data to it. At a slightly lower level it is possible to use the BIOS INT 0x14 interrupt. Windows has its own API and under Linux you can use the ttyS0 device (with the "termios" API to change many parameters). Whatever mode you use you will never use I/O instructions directly to access hardware I/O registers.
• Using I/O instructions directly to access the I/O registers of the UART (at I/O address 0x3f8 for COM1). Most serious communications programs under DOS used this method, as BIOS and DOS serial access was inefficient because it was not interrupt-driven. Most people consider this programming at the bare metal. Operating systems such as Linux contain device drivers that access the UART at the bare metal (so applications do not have to).

Coreboot is split into two levels:

• The low level initialization and startup code. This part is very dependent on the exact motherboard model that you have. Apart from diagnostic output to the serial port, it does nothing recognizable. If you do not enable diagnostic output you end up with a blank screen when this code is finished, exactly like when you started. Of course this code has done a lot of useful things, you just do not see them yet. For one thing, this code takes care that the PC's main memory is operating like you would expect from RAM, a non-trivial task. Because it works at the lower level of bare metal, the very machine-dependent level, this is considered the hard part.
• The payload code. This is the part that does the "real work" as far as the user is concerned. A payload does its job at the easy higher level of bare metal. The same payload will usually work in nearly every standard PC. The most frequently used payload is SeaBIOS, an open source replacement of the traditional PC BIOS. This way almost all standard PC boot disks can be used and will work. But a Coreboot payload can also be a boot loader, a diagnostic tool, a small application or a small Linux system.

The easy part

When you boot up the Linux kernel (or any modern OS), it disables the BIOS services (effectively by changing to protected mode and reloading the Interrupt Descriptor Table, so the BIOS interrupts cannot work the way they did in real mode). All further accesses to devices by the kernel are at the bare metal, i.e. direct access to I/O registers. But this is the easy high level of bare metal. The same Linux kernel (including a limited set of device drivers) can run on nearly every standard PC compatible machine in the world.

At the higher level of bare metal, RAM just works and all familiar I/O registers (8259 programmable interrupt controller, 8254 timer) are available at their standard I/O addresses. The PCI bus is configured, the VGA hardware is in a sane state and lots of other tiny but important things.

Originally Coreboot was called LinuxBIOS. The original intended payload was a small Linux system: a kernel plus a small root file system. As many BIOS chips have a capacity of only 512kB, they do not have sufficient capacity for a usable Linux system. Hence it never became popular. But if you do have enough capacity, you can use Linux as a boot loader. Linux can load and run a different kernel if this is desired, so the kernel in flash can obtain the kernel for the production system and run that instead of itself. The great advantage of a small Linux system over a traditional boot loader is that it has any device drivers that you may want, whether for storage media or for network interfaces and that these drivers tend to be efficient and reliable. Linux has a full-fledged USB subsystem, it has a full-fledged TCP/IP stack.

Instead of a Linux system, several other Coreboot payloads have been developed.

• Bootloaders such as FILO and GRUB2. These can run from ROM and then boot an operating system from the hard disk. These do away with traditional BIOS and they include their own low-level device drivers. Just about the only OS they can actually boot is Linux.
• SeaBIOS, the most popular Coreboot payload today. The advantages over a traditional BIOS are that you can customize it and that you can fix bugs or add features. A few years after you buy a motherboard, the manufacturer will usually stop supporting it and there will be no more BIOS updates. SeaBIOS will hopefully be maintained forever. Almost any traditional PC operating system can be booted under SeaBIOS. Although it does not provide network booting by itself, it can be used in conjunction with gPXE, an open source network boot loader.
• TianoCore is an open source implementation of the UEFI specification. This is a potential Coreboot payload, but no usable Coreboot port is available yet.
• Small stand-alone programs, such as TINT (a falling blocks game), Memtest86, CoreInfo etc.

Nearly all those payloads will work unmodified on almost any PC and of course they will run under a PC emulator such as QEMU.

The hard part

When you power up a PC, the chips that make up the chipset (northbridge, southbridge and SuperIO) are not yet initialized properly. Even though the BIOS ROM is as far removed from the CPU as it could be, this is accessible by the CPU, because it has to be, otherwise the CPU would have no instructions to execute. This does not mean that BIOS ROM is completely mapped, usually not. But just enough is mapped to get the boot process going. Any other devices, just forget it.

When you run Coreboot under QEMU, you can experiment with the higher layers of Coreboot and with payloads, but QEMU offers little opportunity to experiment with the low level startup code. For one thing, RAM just works right from the start.

There are four reasons why the startup code is harder than the payload code.

• It is more machine-dependent. Every northbridge, southbridge and SuperIO chip is different. CPUs have different ways to configure their caches.
• Accurate documentation is lacking for some hardware.
• It is tricky code. For one thing RAM is not available at startup.
• It has to be tried first on the real hardware without any debugging aids. There is usually no accurate simulator at this level of the hardware, so you cannot simulate the code. Before you have at least some I/O ports working, there is usually no way to trace what your code is doing. If you have a logic analyzer (which a typical hobbyist doesn't have), you can trace the ROM accesses during the early stages. If you do not enable the instruction cache, this gives a reasonably good overview of what code the CPU is executing.

Coreboot does two things to make thing a bit easier for developers

• It runs nearly entirely in 32-bit protected mode.
• It is nearly completely written in C.

Getting into 32-bit protected mode is comparatively easy. It can be done in about 16 instructions from reset. It involves loading the GDT (Global Descriptor Table) to a table stored in ROM that contains one code segment descriptor and one data segment descriptor. Then a status bit has to be set. Next a far jump (to the newly defined code segment) has to be carried out. Finally the segment registers (DS, SS and possibly ES, FS and GS) have to be reloaded.

Running C code is a bit more difficult. The problem is that the code generated by any standard C compiler for the x86 CPU will heavily depend on RAM. System RAM is not enabled yet and the code to enable it is so complex that you really want to do it in C. Two solutions have been devised and both of these are in use (one or the other depending on the hardware).

• Use a special C compiler (romcc) that does not make use of RAM, but keeps all data in registers. As the register set of the x86 is quite small (only 8 general purpose 32-bit registers), this severely limits the things that your C program can do. As the CALL and RET instructions cannot be used (they always use the stack in RAM), all C functions have to be inlined.
• Use the CPU cache as a data RAM. This requires some special tricks to pretend that all cache lines contain valid data and to prevent them from being evicted. Which tricks are exactly required, depends on the exact model of CPU, but it can be done. The Cache As RAM trick (CAR) yields at least 16kB of usable RAM, sufficient as stack space for a simple C program. All recent hardware ports use this trick.

When Coreboot starts up, it runs through the following stages

• The boot block, written in assembly language and using no RAM at all. It enters protected more and enables Cache As RAM.
• The ROM stage, written in C and using the CPU cache as its data RAM. It executes from ROM (using the instruction cache to compensate for the really slow ROM access). The main task of this stage is to enable the system RAM.
• The RAM stage, written in C and running from system RAM.It initializes PCI buses and composes the device tree. Finally it loads and executes the payload.
• The payload