Booting Linux off of a USB drive.

Last update: January 23, 2005.

Author: Simon Ilyushchenko (simonf@simonf.com)

Contents:

(With the 2.6 kernel and Fedora 2, make sure you either run 2.6.8-1 or later kernel or disable USB 2.0 support in the kernel configuration, as some 2.6 kernels were reported freezing intermittently.)

Warning: I am not a USB expert. Feel free to email me if you do not understand the instructions or have corrections, but if you understood and followed them, but the drive still would not boot, I WILL NOT be able to help you. Try the usb-devel mailing list instead.

Here is how to use a small USB pen drive as the primary hard drive for Linux. This information floats on the net in bits and pieces, so I figured I'd put it together. You can also do this with a kernel patch listed here (look for boot-usb.patch). However, the initrd way, while longer to set up initially, is easier to reuse when you upgrade the kernel.

You have to first install Linux (I used Redhat 9.0, with which you might need to go into the "linux expert" mode) on your hard drive. Then, when it won't boot, use the rescue CD. Make sure your motherboard supports booting off of USB. You might need to enable USB in the BIOS. Try all USB slots (one out of two did not work for me).

Instructions for Fedora Core 3

You still may need to use the longer instructions below if this simple way does not work.

1. Boot off of a rescue CD. If you use Redhat/Fedora, type

chroot /mnt/sysimage

If your rescue CD would not mount the partitions (for some reason, mine did not), you have to mount them manually and then change root to the hard disk's filesystem. In my case,

mount /dev/sda3 /mnt/sysimage
mount /dev/sda1 /mnt/sysimage/boot
chroot /mnt/sysimage

2. You'll need to create a custom initrd image. Fortunately, FC3 can do it for you (thanks to David Chalmers for pointing this out and to Mike Marmar for coming up with the idea):

mkinitrd --preload=ehci-hcd --preload=usb-storage --preload=scsi_mod --preload=sd_mod /boot/usbinitrd.img 2.6.9-1.667smp


Warning: this did not work for me on FC3, probably because the sleep call (see below) was not there, and I was not
able to create a custom initrd using the steps below. But my hardware is pretty flaky. Other people reported success with this.

Stanislav Umansky made some corrections to the default mkinitrd script. He writes:

The mkinitrd command wasn't okay, because it lacked sleeps and some other commands, so we've changed it a little. The main thing is having sleeps and running sfdisk on /dev/sdb, /dev/sdc and /dev/sdd. When you boot on some systems with the primary SATA drive, /dev/sda is that SATA drive. Our USB drives become /dev/sdb, /dev/sdc...

As we installed FC3, it used volume labels to find a disk when  booting (ROOT in our case), but when we tried to boot on a system where Linux had already been installed on the primary hard drive, the system found two ROOT-labeled partitions and didn't boot successfully, so we had to change the labels to ROOT_USB, etc and modify grub.conf.

3. Update your lilo.conf or grub.conf to use the new usbinitrd.gz with the same kernel that you grabbed the modules from. Run lilo or grub-install.


Instructions for both FC3 and older distributions

If you are doing this on Fedora Core 3, note a couple of differences mentioned below.

1. If your rescue CD would not mount the partitions (for some reason, mine did not), you have to mount them manually and then change root to the hard disk's filesystem. In my case,

mount /dev/sda3 /mnt/source
mount /dev/sda1 /mnt/source/boot
chroot /mnt/source

2. You'll need to create a custom initrd image. Take your current image from /boot (see /etc/lilo.conf or /etc/grub.conf for the exact name). I'll call it initrd.img. (Use mkinitrd to create an initrd image if you don't have one.)

cp /boot/initrd.img /tmp/initrd.gz

3.

gunzip /tmp/initrd.gz

4.

mkdir /tmp/a

5.

mount -o loop /tmp/initrd /tmp/a

If this step fails, there is no loopback device support in your kernel. You might be able to load the module with "modprobe loop". If it does not work, you have to compile it in.
On Fedora Core 3, you should do

cd /tmp/a
cpio -i < /tmp/initrd

6. The file /tmp/a/linuxrc (/tmp/b/init on Fedora Core 3) is the script that runs before the root partition is mounted. We need to make sure it can mount a USB partition. If you use Redhat, replace /tmp/a/linuxrc with this (edit your script otherwise) :

#!/bin/nash

echo "Loading jbd.o module"
insmod /lib/jbd.o
echo "Loading ext3.o module"
insmod /lib/ext3.o
echo Mounting /proc filesystem
mount -t proc /proc /proc
echo "Loading usb modules"
insmod /lib/usbcore.o
insmod /lib/scsi_mod.o
insmod /lib/sd_mod.o
insmod /lib/usb-storage.o
insmod /lib/usb-uhci.o
/bin/sleep 5
echo Creating block devices
mkdevices /dev
echo Creating root device
mkrootdev /dev/root
/bin/sfdisk -R /dev/sda
echo 0x0100 > /proc/sys/kernel/real-root-dev
echo Mounting root filesystem
mount -o defaults --ro -t ext3 /dev/root /sysroot
pivot_root /sysroot /sysroot/initrd
umount /initrd/proc

The lines before "Loading usb modules" and after "sfdisk", as well as mkdevices an mkrootdev are probably already there, so you just need to insert the extra stuff. First the script loads all the necessary kernel modules, then waits five seconds in order for the USB code to recognize the disk, then runs sfdisk to reread the partition table.

Note: Alan Deehr reported that usb-uhci.o did not see his USB 2.0 external disk. He used usb/host/ehci-hcd.o instead.

7. Copy kernel modules into the initrd image. For example, if you are running kernel 2.4.22 (uname -r to find out),

cd /lib/modules/2.4.22/kernel/drivers
cp usb/usbcore.o /tmp/a/lib
cp usb/usb-uhci.o /tmp/a/lib
cp usb/storage/usb-storage.o /tmp/a/lib
cp scsi/scsi_mod.o /tmp/a/lib
cp scsi/sd_mod.o /tmp/a/lib

If you have a monolithic kernel without the modules, you have to prepare a kernel that has them first.

8. Find out what libraries sleep and sfdisk use from the output of ldd:

ldd /bin/sleep
ldd /sbin/sfdisk

Copy the libraries into /tmp/a/lib. In my case it was:

mkdir /tmp/a/lib/tls
cp /lib/tls/libm.so.6 /tmp/a/lib/tls
cp /lib/tls/libpthread.so.0 /tmp/a/lib/tls
cp /lib/tls/libc.so.6 /tmp/a/lib/tls
cp /lib/librt.so.1 /tmp/a/lib
cp /lib/ld-linux.so.2 /tmp/a/lib

(Instead of copying sleep and the libraries, Konrad Ludwikowski suggests using Busybox, which contains smaller versions of common utilities in one executable.)

9. Copy the executables over:

cp /bin/sleep /tmp/a/bin
cp /sbin/sfdisk /tmp/a/bin

10.

umount /tmp/a

On Fedora Core 3, do

cd /tmp
mkcramfs /tmp/a initrd

11.

gzip /tmp/initrd

Alternatively, on Fedora Core 3, instead of steps 10 and 11 you can do: (thanks to Angela Jean)

cd /tmp
find . | cpio -c -o | gzip -9 > /boot/usbinird.img

12.

cp /tmp/initrd.gz /boot/usbinird.gz

13. Update your lilo.conf or grub.conf to use the new usbinitrd.gz with the same kernel that you grabbed the modules from. Run lilo or grub-install.

14. Tom Haux reported that he successfully booted a RH 9 box in this way, but then X would not start due to some hardware errors. He had to disable kudzu to get it working.

/sbin/chkconfig --level 5 kudzu off

P.S. Presumably you can also patch the kernel to add a delay before a USB device is mounted at boot time.

Using USB boot and Systemimager.

If you want to use Systemimager to auto-install your nodes (which you should - it's a great tool), beware of some differences. Systemimager works by taking an image of a working node, booting into its own temporary kernel on another node and copying all the files over. Obviously, we want to make sure that the temporary kernel can see the USB drive, or there will not be a place to copy files to.

The differences with the process above:

  1. The initrd file is included into the RPM package systemimager-i386boot-standard. On my system its files are in /usr/share/systemimager/boot/i386/standard.
  2. The initrd file is cramfs, not loopback filesystem. This means that you can't edit a mounted file, you have to

    cp -r

    the whole mounted directory (let's call the new directory mydir), change its contents and then

    mkcramfs mydir initrd
    gzip -f initrd
    cp -f initrd.gz /tftpboot/initrd.img

  3. As you can see from the previous step, initrd.img is usually placed into /tftpboot. That's where the boot files for the systemimager kernel live.
  4. The same RPM package systemimager-i386boot-standard includes the necessary USB kernel modules. Untar boel_binaries.tar.gz and copy the USB modules into mydir/lib.
  5. As luck would have it, one of the modules (usb-storage.o) was missing. I had to install the systemimager source RPM, change the kernel configuration file (CONFIG_USB_STORAGE=m) and "make modules".
  6. The script that is run by initrd is /etc/init.d/rcS. It's more complicated than the usual initrd scripts. Put the section that loads modules in the beginning of the control flow (before switch_root_to_tmpfs) and the sfdisk call after get_boel_binaries_tarball.
  7. Note that the temporary systemimager kernel uses devfs, not the regular dev filesystem. If everything goes well, you don't need to know that, but it can make debugging interesting.