1.1. Who needs to read this ?

This guide describes a work in progress, to port Linux to a custom PowerPC-based board. This means making the operating system work on unfamiliar hardware. Anyone who is on the same track might benefit from reading this paper, as it highlights the pitfalls and problematic points along the way.

1.2. What do I need to know (why so much) ?

Before attempting to port Linux, know at least the following: (whenever possible, a link to a proper information source is attached)

• Hardware: know what hardware you've got, how it works (if it works), and how is it initialized. Get all the hardware manuals you can - you'll probably need them. Also, never assume the hardware works the way it supposed to ! Hardware people do the darnest things :-(

• Basic understanding of drivers and how they work in Linux. Programming knowledge of simple drivers is an advantage - but not a must. http://www.tldp.org/HOWTO/Module-HOWTO/index.html

• How to work with Vision-ICE, how configure it and use it to load a binary kernel into the target RAM. Also, at the beginning, you'll need to know how to use ICE to debug in assembly.

• How to compile and configure a Linux kernel. http://www.tldp.org/HOWTO/Kernel-HOWTO.html

• The Linux boot process. http://www.tldp.org/HOWTO/BootPrompt-HOWTO.html

• Working knowledge of C programming is a must. Some assembly is sure to help. Also, it is best to get to know Makefiles. They tend to raise their ugly head once in a while.

• The Internet is your friend. All the information you need is probably on the net. You just have to know how to find it. Google is a good way to start; mailing lists and news groups usually keep the real gold.

• How to install Linux, configure it, administrate it and basically take care of everything it needs. This guide does not cover anything regarding system administration, setting up a server etc.

1.3. The tools

This section describes the tools we used during the process. Most are trivial to install and use. When neccesary, consult the appropriate url or manual.

• HardHat Linux: First and foremost, HHLinux, now known as MontaVista Linux, is the distribution we started with. The distribution contains LSPs (same as BSPs) for PowerPC in a number of board configurations. For porting to our board, we took the LSP which is closest in hardware to our Artysyn PMPPC board, and started from there.

• LXR: This is THE killer tool, which allowed us to port Linux in a very short time. LXR is a cross referencer, which means it reads a piece of code (the Linux kernel, for example), and then allows browsing the code, searching through it and much more. I cannot emphasize enough how important this tool is. To see what the end result looks like, look at http://lxr.linux.no/source. LXR itself can be downloaded at http://lxr.sf.net

• VisionICE: A hardware debugger, which has the ability to stop, run and add breakpoint straight in the CPU. VisionIce is very usuful when no operating system is running, and allows to step in the kernel during boot process. The application can also be used to take a binary image of a kernel, load it into the target's RAM memory and run it - useful when you've got no boot loader.

• CVS: A version control system, allows you to keep multiple versions of the code. Other than backing up the code, it allows diffing between different version, and reverting to older version, when needed.

• A terminal program, like HyperTerminal or ProCOMM for Windows™, or minicom for Linux.

1.4. The hardware

The board is based on PPC750 (PowerPC) processor. It is 6U VME64 standard. The board is designed to host two PCI Mezzanine cards (CCPMC) - Mezzanine cards that comply with Std CCPMC1386 can be attached.

• COP connector.

• 1 MB of L2 cache.

• CPC700 system controller.

• 128 MB SDRAM with ECC.

• Flash memory, divided to boot flash and user flash.

• NVRAM memory.

• I/O discretes.

• RS232 channels.

• General purpose registers.

• PCI 2.1 local bus.

• 10/100 BaseT ethernet channel.

• VME64 system bus.

1.5. Copyright & License

Copyright (c) 2002 Shie Elrich

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in Appendix A.

2.1. Creating a development environment

The minimum requirments are obviously a development station and a target. However, the recommended way of working is having a third host which acts as a server. The server runs several services such as ftp, telnet, NFS, tftp (if needed) and CVS. The main role of the server is to run CVS and track version control, however once you can boot the target from network, the server will also hold the target images, and filesystem, which makes development much easier.

Regardless, the first step is to install a tool-chain (compiler, linker etc.) for your target. The HardHat Linux cdrom includes all the needed files, and the installation sequence is documented in the HardHat Linux documentation. During the installation, you must select your LSP (basic software for the selected board), and HardHat will install a set of tools and a kernel source tree matching your LSP.

We had a board that had vxWorks running on it, so we setup the target to boot using the standard vxWorks loader. Once the loader initiated, we used visionICE to take-over the target (so that vxWorks won't load an image file) and load a Linux image into the target. What you need to do at this point is get an ICE, connect it to the network and to the target - through a JTAG connection - and install the ICE software on your host.

What should have been done so far:

• A Linux host installed, and the HardHat tool-chain.

• A working target (hardware should be functional)

• ICE is connected to the target and the network and its software usable.

• Optionally, a server running CVS, telnetd, NFS and FTP.

2.2. Compiling the first kernel

If you've installed the Linux kernel that comes with HardHat, then cross-compiling should already be enabled in the kernel Makefile. If your kernel is not from the HardHat CD, you should enable cross-compiling in the Makefile by defining a CROSS_COMPILE entry in the following manner: (a code segment from the main Makefile)

CROSS_COMPILE 	= /opt/hardhat/devkit/ppc/7xx/bin/ppc_7xx-
AS		= $(CROSS_COMPILE)as
LD		= $(CROSS_COMPILE)ld
CC		= $(CROSS_COMPILE)gcc

The Linux kernel is modular, and allows you to configure it and choose which "blocks" should be compiled with the kernel. In order to do this, first cd /usr/src/linux (assuming your kernel source code is installed at /usr/src/linux). Once there, type make xconfig.After saving your options, you should make vmlinux to create a kernel image suitably for using with VisionICE.

We will not go into more details here, as it's outside the scope of this document. For more information, try http://www.tldp.org/HOWTO/Kernel-HOWTO.html

2.3. Booting the machine

First, configure the terminal program, in our case minicom, the following way: 9600 bps, 8 bits, no parity, 1 stop bit and no flow control of any kind. The serial port in Linux should be /dev/ttyS0 for COM1, /dev/ttyS1 for COM2 etc.

Start the target. You should see the vxWorks bootloader on your terminal screen, and should be able to stop the boot sequence by pressing the space bar.

We cannot use the vxWorks bootloader to load a Linux kernel since it looks in the ELF header and loads the image to the address written there. However, the Linux kernel, which uses virtual memory, is linked to a high-memory address, and vxWorks can't handle that.

Once the target is stopped, run the VisionICE software and perform the following steps:

• Initialize the target by pressing Target|Initialize

• Press File|Load Executable. A dialog box will open, asking you to choose a file. Please choose your kernel image (vmlinux). Before pressing Load, don't forget to enter a value in the +/- Bias field.

  The bias field makes it possible to tell ICE to load a certain image in a different address than what's stated in the ELF binary. We wanted to load the kernel into address 0x300000, and since the binary was linked to 0xC0000000, we entered -0xBFD00000.

• Once the image is loaded succesfully, you can press Run or Step to start executing your kernel.

VisionICE is not the correct tool to use when debugging Linux. ICE doesn't know about virtual memory and protected mode (at least the version we had), and since the Linux kernel turns on virtual memory very early, ICE is only useful for debugging the first assembler statements. After VM is turned on, ICE starts crashing and giving wierd results.

3.1. Debugging with print_str()

As stated in the previous chapter, the machine starts to boot, but nothing happens. At least, nothing that we can see. The screen is blank and no kernel messages appear. At this point, you have to ask yourself, is it really booting?

Since the console wouldn't start, and ICE died real fast, we had no choice. We had to debug somehow, and the oldest way is good here - printing to the screen. Obviously, we couldn't use printk(), so we wrote a short function which pushes characters straight into the serial port. We used the boot process "map" shown in the previous section, and inserted some prints along the way. This helped us to know at what stage we are completing and where we're dying. The following piece of code prints a single character to the serial port, by polling it and waiting for it to be free.

/* tx holding reg empty or tx */ #define LSR_THREMPTY 0x20 /* fifo is empty (in fifo mode ) */ #define THR_REG 0x00 /* Transmit holding reg */ #define LSR_REG 0x05 /* Line status reg */ #define COM1_ADDRESS 0xFF600300 /* == replace with your UART address */ void print_char (char ch) { volatile unsigned char status = 0; /* wait until txempty */ while ((status & LSR_THREMPTY ) == 0) status = *((volatile unsigned char *)(COM1_ADDRESS + LSR_REG)); *((volatile unsigned char *)(COM1_ADDRESS + THR_REG)) = ch; }

There's a better code for printing directly to the serial port, however, it's a bit more complicated. You can find it in arch/ppc/boot/common/misc-common.c, using puts() or putc().

3.2. Modifying code using compiler flags

Although it is not a porting issue, the way you modify your code matters. It's easier if you do it right the first time. The Linux kernel uses standard configuration flags CONFIG_XXXX (like CONFIG_PPC, CONFIG_ISA etc), which are used to mark a certain machine, architecture or device. We defined ourselves a new flag (let's call it CONFIG_TESTMACH), and surrounded our new/modified code with these flags:

....original code.... #ifdef CONFIG_TESTMACH ....modified code.... #else ....original code.... #endif /* CONFIG_TESTMACH */

To "activate" our code, we added the new flag to the kernel configuration file - .config - by adding CONFIG_TESTMACH=y to it. In the first stage, this solution allows you a quick way to find the code you changed, but later the flag you chose will allow you to add your code into the kernel tree and into the configuration program (make xconfig).

3.3. Getting the console to work

4.1. Memory probing, RTC and decrementors

Now that the console was working, we could see the real problems. The system wasn't booting yet. Since we were working with C code, we traced the code, and found that a function called sdram_size() wasn't completing correctly. The function probed a register for the size of the RAM, a register our board doesn't have. We made the function return a given value of 128MB, it's an ugly hack, but our board doesn't have a way of knowing the amount of RAM.

Wwe had the same problems with a bunch of functions called todc_XXXX, mainly todc_get_rtc_time(), todc_set_rtc_time(), and time_init() since we don't have a RTC (real-time clock) chip on our board, and those functions were using it. For the time being, we made the todc_XXX function only set and get a constant date and time, since our board doesn't have a bios battery and so cannot keep time when powered off.

Once all this was done, we found todc_calibrate_descr(), which again uses the RTC chip. We had to replace that function with our own:

void calibrate_decr() { int freq, divisor; freq = bus_freq(); divisor = 4; tb_ticks_per_jiffy = freq / HZ / divisor; tb_to_us = mulhwu_scale_factor(freq / divisor, 1000000); }

4.2. Big-little endian (we should have known)

4.3. Ethernet: our first PCI device

Our board uses an Intel ethernet chip, called i82559er, which has a module called eepro100. After compiling the module and booting, we discovered that the module isn't working, although an ethernet device was found. We guessed that it was an irq problem, and that the devices don't get the IRQs they need. We modified a function called pmppc_map_irq() to map our ethernet devices:

XXXX_map_irq(struct pci_dev *dev, unsigned char idsel, unsigned char pin) { static char pci_irq_table[][4] = /* * PCI IDSEL/INTPIN->INTLINE * A B C D */ { {22, 0, 0, 0},/* IDSEL 3 - Ethernet */ {0, 0, 0, 0},/* IDSEL 4 - unused */ {0, 0, 0, 0},/* IDSEL 5 - unused */ {0, 0, 0, 0},/* IDSEL 6 - ???? */ {0, 0, 0, 0},/* IDSEL 7 - unused */ {0, 0, 0, 0},/* IDSEL 8 - unused */ {0, 0, 0, 0},/* IDSEL 9 - unused */ }; const long min_idsel = 3, max_idsel = 9, irqs_per_slot = 4; return PCI_IRQ_TABLE_LOOKUP; }

The function maps IRQs according to IDselects, which means in the order on the PCI bus by which the devices are set. This structure is a bit tricky: min_idsel denotes the topleft corner of the array, and max_idsel is the bottomleft corner. irqs_per_slot is the number of IRQs per line. The structure is as follows:

each cell contains (IDSEL, SLOT#, IRQ) +----------------------------------------+ | (3,0,22) | (3,1,0) | (3,2,0) | (3,3,0) | +----------------------------------------+ | (4,0,0) | (4,1,0) | (4,2,0) | (4,3,0) | +----------------------------------------+ .......... .......... +----------------------------------------+ | (9,0,0) | (9,1,0) | (9,2,0) | (9,3,0) | +----------------------------------------+

As you can see, our i8559er needs IRQ 22, and is seated in IDselect 3. Of course, we didn't know that at the start, so we wrote a small piece of code that read all the vendor IDs in all the IDselects. Once done we compiled, but the ethernet device still didn't work.

The next problem was that the module couldn't decide on a MAC address for the device. The MAC address should be written on an EEPROM chip (connected to the device), but we discovered that the hardware guys decided that i82559 doesn't need the EEPROM, so they removed it. After hardcoding a MAC address inside eepro100.c, the ethernet device finally worked. The final solution was to make the module read the MAC address from NVRAM memory, and if no other choice was available, to fall back to a default MAC address.

The next step was to mount a NFS root filesystem. For details see the documentation in Documentation/nfsroot.txt

4.4. Some Miscellaneous Issues

We had new problems, some would say good problems. We didn't have a bootloader yet, however we needed to pass a command line to the kernel at boot time. We hard-coded the command line into the kernel inside the parse_options(). After that was finished, we made console_init() and serial_console_setup() work the way they should. They no longer ignored the command line, but still RTS and DTR stay low.

Another important issue was memory mapping. The file arch/ppc/mm/init.c contains a function called MMU_init(). This function is actually a big switch statment, divided by the machine type. Each machine maps its memory using the setbat() and ioremap() functions. The BAT mechanism is a way of translating virtual addresses into physical ones. Thus, setbat() is used by specifying a virtual address, a physical address and a page size. Not every size can be used here; you should use one of the finite set of sizes, ranging from 128KB to 256MB. We mapped our IO memory so that virtual equalled physical.

As mentioned, there is another way of mapping memory - ioremap(). ioremap() is used to map physical addresses into virtual ones, making them available to the kernel. The function does not allocate any memory, simply returns a virtual address by which one can access the memory region. The following is a snippet from MMU_init():

case _MACH_mymachine: setbat(0, LOW_IO_VIRT_BASE, LOW_IO_PHYS_BASE, LOW_IO_SIZE, IO_PAGE); ioremap(UNIVERSE_BASE,UNIVERSE_SIZE); /* Universe VME */ ioremap(EEPRO100_BASE,EEPRO100_SIZE); /* Ethernet EEPRO100 */ break;

As you can see, we don't take the return value of ioremap(). We don't need it, since at this stage the kernel maps the addresses so that virtual address == physical address.

5.1. The 64 bit barrier

The CPC700 has a "feature" which is supposed to make some memory access use 64 bit wide. This is a problem since some test-and-set registers on our board might get set unintentionally, because we were trying to read something 16 bits lower. In order to solve this situation, we set the memory controller to 64 bit wide intervals. If you try to access those areas in another manner (8 or 16 bit access), the CPC700 simply throws them away. We had to be able to read/write those areas, since important "discretes" (controlled by an Altera device) were mapped there.

In order to access those areas, we needed a function that does a 64 bit write. As far as I know, doing a 64 bit write on a PowerPC is possible in two ways: using cache lines and using a floating point register. The floating point register is a 64 bit sized register, so when we write it, the whole 64 bit get written. The problem is that you can't do floating point in the kernel. Since the kernel doesn't save the floating point registers during context switch, it doesn't allow FP, and will throw an exception if done in the kernel.

After messing with cache lines, we decided to go the FP way, and added the following function:

void out64(__u32 addr, long long *pVal) { __u32 flags, tmp_msr; save_flags(flags); cli(); tmp_msr = __get_MSR(); tmp_msr |= MSR_FP; tmp_msr &= ~(MSR_FE0 | MSR_FE1); __put_MSR(tmp_msr); sysOut64(addr, pVal); __put_MSR(flags & ~(MSR_EE)); restore_flags(flags); }

The function adds a floating point to the PowerPC MSR register, and makes sure that no exceptions will be generated as a result of doing FP. Once done, it uses an assembly code, described below in the sysOut64() to do the actual floating-point operation. Note that the function turns off interrupts, but this is acceptable here, since we use the function on rare occasion.

_GLOBAL(sysOut64) stwu r1, -DEPTH(r1) mflr r0 stw r31, FP_LOC(r1) stw r0, LR_LOC(r1) mr r31, r1 stfd fr0, FPR_SAVE(r31) /* save floating point reg contents */ lfd fr0,0(r4) stfd fr0,0(r3) eieio lfd fr0, FPR_SAVE(r31) /* restore floating point value */ lwz r4, 0(r1) /* now restore the stack frame */ lwz r0, 4(r4) mtlr r0 lwz r31, -4(r4) mr r1, r4 blr

5.2. Booting from flash

While Linux was booting using an NFS filesystem, this was not enough. For an actual field product, we needed Linux to boot from an independent device, without the need for a network at all. We decided to create a special kind of image, called initrd, which is basically a Linux kernel with a compressed file. The compressed file includes a Linux filesystem. The filesystem is unpacked to a ramdisk on boot, and mounted as the root filesystem.

During the boot process, the bootloader relocated the kernel image to address zero - which was fine, and the initrd part to a higher address. The area to which initrd was relocated was not mapped in our kernel's memory, and all we got was a kernel error (access to bad area). After modifying the bootloader to relocate initrd to a different address, all was fine and Linux booted succesfully.

If your board has some NVRAM memory, it would be a good idea to use it for bootloader purposes. After writing a module for our NVRAM memory (out of scope for this paper), we modified the bootloader, so that the kernel command-line, and MAC address were saved in NVRAM. When the bootloader starts, it checks NVRAM and if it is initialized (by a certain magic number), the bootloader uses the command line written there. Otherwise, the bootloader reverts to a default command line, allowing the user to edit it.